No announcement yet.

Kinect Plug-in - User Guide

This topic is closed.
This is a sticky topic.
  • Filter
  • Time
  • Show
Clear All
new posts

    Kinect Plug-in - User Guide

    Installation / Configuration

    System Requirements:

    This plugin is for HS3 or HS3PRO.
    It supports both the Kinect for Windows sensor and the Kinect for Xbox 360 sensor. Kinect v2 (i.e. Kinect for XBox One) is not yet supported.
    The Kinect needs Windows 7 or later

    You need to have the Kinect drivers installed, if you don't have them installed yet, you can download them from the folowing links:
    Kinect for Windows drivers (Runtime)
    Kinect for Xbox 360 drivers (SDK)

    for Speech Recognition features you can install additional language packs to enable speech recognition for different languages or accents:

    Plugin Installation:
    Download and install from the updater, or download package from the Beta Testing thread and follow instructions to install.
    Last edited by spud; August 16, 2015, 06:52 PM.


    Voice Recognition

    Go to PLUG-INS > Kinect > Config, and click on the Voice Recognition tab
    • set the Speech Recognizer to the language you want to use, you can install some additional languages as explained above in the prerequisites section
    • if you get too much false positive try to increase the Confidence Threshold, if the system miss too much commands try to decrease it. It must be a value between 0 and 1.
    • the Attention Phrase is the phrase that will prefix all your commands (except for sub grammars see below): "Kinect, turn the lights on". You can set this to anything you want.

    Click image for larger version

Name:	vr_settings.jpg
Views:	1
Size:	57.2 KB
ID:	1174531


    You can use the Kinect Trigger "A Phrase Was Recognized" to trigger any event using a voice command. Different variations of a same command can be configured by using a special syntax (see syntax guide for a full description) For example the phrase diagram on the screenshot below will recognize 4 different phrases: "play some music", "play music", "put some music" and "put music". Do not forget to say your Attention Phrase before each command. For example if your command is "Play some music" and your Attention Phrase is "Kinect", you need to say "Kinect, play some music"

    Click image for larger version

Name:	vrevent2.jpg
Views:	1
Size:	43.6 KB
ID:	1170460


    You can also add voice command to any HS device without using any event or script.

    To do so, from the "Device management" page, click on the device you want to control then click on the Kinect tab. Tick the "Voice Command Enabled" checkbox. Then for each control enter a custom command of your choice. The only requirement is for controls that are ranges, for those type of controls the command need to include the "(value)" string. This string will be replaced by all possible values by the plugin.

    Click image for larger version

Name:	device_cmd2.jpg
Views:	2
Size:	48.5 KB
ID:	1174532
    So, if you use more than one kinect instance, this feature will only work with the first one (usually the instance without a name)

    (2) For range controls the plugin replaces the (value) string with all possible values in words, for example something like <one|two|three|...|hundred>, so right now only english is supported.


    Sub grammars are grammars (i.e set of voice commands) that can be loaded temporarily. This allow to create context specific commands without having to say the attention phrase before each command. See this thread for more details
    Last edited by spud; August 7, 2014, 12:39 PM.



      People Detector

      The Kinect is capable of detecting people (up to 6) and tell where they are. From the config page you can define zones and the plugin will automatically create a corresponding device which will report how many people have been detected in this zone. An overall zone is a global zone which detect people no matter where they are. Zones can overlap or not.

      Click image for larger version

Name:	zonessettings.jpg
Views:	1
Size:	38.0 KB
ID:	1174528

      Click image for larger version

Name:	zonedevices.jpg
Views:	1
Size:	46.9 KB
ID:	1174529

      Posture And Gesture Recognition

      The plugin is capable of detecting some basic gestures and postures that can be used as triggers for events.

      Click image for larger version

Name:	posture.jpg
Views:	1
Size:	60.2 KB
ID:	1174530

      A posture is a static position that needs to be maintained for 1 or 2 seconds to be detected. It can't detect multiple postures at the same time.
      Current list of Postures are:
      • HandsJoined
      • LeftHandOverHead
      • RightHandOverHead
      • LeftHello (= Left hand near head)
      • RightHello (= Right hand near head)

      Current list of Gestures are:
      • RightHandSwipeLeft (= Right hand moving from right to left)
      • RightHandSwipeRight (= Right hand moving from left to right)
      • RightHandSwipeDown(= Right hand moving top down)
      • RightHandSwipeUp (= Right hand moving bottom up)
      • LeftHandSwipeLeft (= Left hand moving from right to left)
      • LeftHandSwipeRight (= Left hand moving from left to right)
      • LeftHandSwipeDown(= Left hand moving top down)
      • LeftHandSwipeUp (= Left hand moving bottom up)


      To use the kinect as an IP Camera, go to the config page, Camera tab and tick the checkbox called "Use Kinect Camera as NetCam"
      This will start a snapshot server on the machine where the plugin is running, then you can use the snapshot url (http://<plugin_ip>:8078/kinectsnapshot/) in a browser, in HSTouch, or any other software that takes a snapshot url as input.

      Warning you need to run the plugin as Administrator for this to work.

      Click image for larger version

Name:	netcam.jpg
Views:	2
Size:	38.5 KB
ID:	1176606

      You can use the same snapshot url with the tilt=up or tilt=down parameter to use the motor to move the kinect sensor up or down
      For example in HSTouch configuration set the "Pan Up Start" and "Pan Down Start" as in the screenshot below. It will allow you to tilt the kinect sensor up or down when you tap the up or down arrows from the HSTouch interface.
      Click image for larger version

Name:	tilturl.jpg
Views:	2
Size:	64.7 KB
ID:	1176607

      The step parameter is optional and let you specify the number of degrees you want the sensor to move for one tap.
      Last edited by spud; March 3, 2015, 03:23 PM.


        Event Actions

        Event Actions

        Here is the current list of kinect actions that can be done from events:
        • "Load Grammar"
        • "Unload Grammar"
        • "Rebuild Grammars"
        • "Enable Voice Recognition"
        • "Disable Voice Recognition"
        • "Enable Visual Recognition"
        • "Disable Visual Recognition"
        • "Enable Camera"
        • "Disable Camera"
        • "Take Picture"
        • "Tilt Sensor Down"
        • "Tilt Sensor Up"

        and here is an example of how to take a picture using the kinect camera:

        Click image for larger version

Name:	take_picture_event.jpg
Views:	1
Size:	40.9 KB
ID:	1174533
        Last edited by spud; May 25, 2015, 04:27 PM.


          Multiple instances and Remote instances

          Multiple Instances

          You can control multiple kinect by starting new instances of the plugin. If the kinect is plugged to a remote computer, see the Remote instance section below, if you have multiple kinects plugged into your HS machine, start a new instance of the kinect plugin by clicking the + button from the PLUG-INS > Manage page.

          Click image for larger version

Name:	multiple_instances.jpg
Views:	1
Size:	45.3 KB
ID:	1174527

          Enter a name for your instance, submit, and then enable it.
          This will create a new Config link in the PLUG-INS > Kinect menu, allowing you to choose which sensor this instance is supposed to control and to change the settings for this instance.

          Click image for larger version

Name:	multiple_instances2.jpg
Views:	1
Size:	84.9 KB
ID:	1174526

          Remote Instances

          To run the plugin remotely, you need first to install it on your HS machine, even if you don't have a Kinect connected to your HS machine you need to have it installed there (but you don't have to enable it) so that HS doesn't complain about licensing issue when you start your remote instances.
          • copy HSPI_KINECT.exe, bin/Kinect/Kinect.Toolbox.dll to a directory on your remote computer
          • copy HomeSeerAPI.dll, HSCF.dll and Scheduler.dll from your HS3 directory to the same remote directory
          • open a command prompt on your remote computer
          • cd to the directory, then type HSPI_KINECT.exe server=<ip_of_your_hs_machine> instance=<instance_name> autoreconnect=<seconds_to_wait> console=<false|true>

          where <instance_name> can be anything you want but only alpha numeric characters.

          autoreconnect is an optional parameter which makes the plugin automatically reconnect when the connection with HS is lost. It can be useful for example when you want to restart HS without having to restart all the remote plugins manually.
          <seconds_to_wait> is the number of seconds to wait before the plugin tries to reconnect.

          console is an optional parameter that hides the console window when set to false
          Last edited by spud; January 28, 2015, 11:10 AM.