Smart Interaction (SI) Gesture Recognition for Samsung SmartTV Applications - dummies

Smart Interaction (SI) Gesture Recognition for Samsung SmartTV Applications

By Handstudio Co., Ltd

Before looking at Smart Interaction (SI) gesture recognition in detail, it will be helpful to start with an example application with the feature. The following is RoyPoy, a Handstudio- developed educational art application.


The Samsung SmartTV’s mouse event handling is similar to a common JavaScript mouse event handling. However, it has a major potential problem of losing the focus through a mouse onClick event. This process needs to be even more carefully handled while using multiple input devices.

Configuring the gesture recognition feature

Set the mouse element to “y” in the config.xml file to enable the application of the gesture recognition feature.

<mouse itemtype="string">y</mouse>

Registering and unregistering the gesture recognition event

Use the Web API SubscribeExEvent() function to register the gesture recognition. Use the UnsubscribeExEvent() function to unregister an event that was previously registered with the SubscribeExEvent() function.

Event handling

Samsung SmartTV runs the registered callback function when voice recognition is started. The function receives the event object as its parameter, which returns event type, result, and a few other properties.

Gesture Recognition Event Types
Event type Description
EVENT_GESTURE_BEGIN_MONITOR Primary hand’s movement is detected
EVENT_GESTURE_SECONDARY_LOST Secondary hand’s movement is lost
EVENT_GESTURE_SECONDARY_DETECT Secondary hand’s movement is detected
EVENT_GESTURE_2HAND_ZOOM User made the zoom gesture
EVENT_GESTURE_2HAND_ROTATE User made the rotate gesture
EVENT_GESTURE_LIKE User made the like gesture

The result value depends on each event type.

Configuring the Gesture HelpBar

The gesture helpbar shows available gestures or status information on the bottom of the Samsung SmartTV screen.


The Gesture HelpBar shown is from the 2012 Model, and Areas selectable by a pointer are from the 2013 Model.


Like the voice recognition, the gesture recognition process depends on the helpbar mode. Use the SetGestureHelpbarInfo() function to control the gesture helpbar.

The SetGestureHelpbarInfo() function
Function SetGestureHelpbarInfo
Version Support from RECOGNITION-0001
Security Type RECOG
Usage SetGestureHelpbarInfo(helpbarInfo)

[string] Information of the helpbar to be registered
Example var helpbarInfo = ‘{ helpbarType :
“HELPBAR_TYPE_GESTURE_CUSTOMIZE”, helpbarItemsList : [{ itemType :
“HELPBAR_GESTURE_ITEM_RETURN” , itemText : “Go back” },{ itemType :
“HELPBAR_GESTURE_ITEM_ZOOM_OUT_IN” , itemText : “Zoom in / out” },
{ itemType : “HELPBAR_GESTURE_ITEM_ROTATION” , itemText : “Rotate
photo” }]}’;

The parameter helpbarInfo object has the following structure. While the chart treats it as if it is an object data type to help understanding, it is actually a String data type.

Structure of the Gesture Recognition HelpBar
Structure {
helpbarType: HelpBar Type;
helpbarItemList: [ { itemType: Item Type, itemText: “Text” }
helpbarType HelpBar Type
ItemsList Item list for the HelpBar (includes commands list)
itemType ItemTypes is the gesture commands to be displayed in the
HelpBar. Only events for gestures registered in here are

itemText ItemText is text to be shown on the HelpBar

Example gesturehelpbarInfo = ‘{ “helpbarType :

Non-standard mouse events — Slap or Thumbs up — need to be registered in the helpbarItemList to be recognized by the TV. As in the voice recognition, the helpbarItemList has a maximum number of items — four.