Announcement

Collapse
No announcement yet.

Plugin: Google Home DialogFlow natural language processing -- BETA

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #46
    Thank you for the reply. So this does not make use of the HS3 JSON API to receive and send the text? If that is the case, I would need to also recreate all standard HS voice triggers as well, and not just the Alexa helper ones, correct?

    I'm really looking for something that plugs into existing voice processing, like Jon00's Alexa helper, so I can keep things simple when possible and use regular HS triggers when I can, in addition to keeping voice interaction uniform across all devices.

    Sounds to me this plugin would be sort of separate from all of that, correct?

    My little Automate Flow (Tasker-like) for my phone is quite simple:

    1) Transcribe voice to text
    2) Send text to HS3 JSON API
    3) Parse JSON response from HS3 which includes text to speak
    4) Speak response

    I tried accomplishing this with IFTTT and got close. I just couldn't figure out how to have multiple actions or send text to my GH to be spoken. I may spend a little more time tinkering w/ that before delving into this plugin, unless perhaps some of my assumptions above are incorrect.

    Have a great day!

    Comment


    • #47
      For reference, this is what the JSON speech call looks like in my implementation:

      /JSON?request=voicecommand&phrase={urlEncodedTextHere}

      Then, HS will do its thing as if you've spoken to a speaker client, HSTouch, my Automate Flow, or used the "Tell HomeSeer to..." Alexa skill. Jon00's helper taps into this flow and gives me my custom, and dynamic, responses.

      Is this how your method interacts with HS, or are you acting on events and devices directly?

      Comment


      • #48
        I also heavily rely on Tasker, but I don't really understand your flow. Why do you use the JSON interface to send the voice command instead of using e.g. the Google Home app ? I use the voice recognition of Google Home in my car to e.g. open the front gate and garage a few hundred meters before I get home.

        This plugin does not use the HomeSeer 'speech recognition', but it uses the one from DialogFlow (which is part of Google). As far as I am aware, it is not possible to parse the raw voice command from GH within HomeSeer, that's why I'm using DialogFlow. It has a downside: setup if more complex, but also a lot of upside: more supported languages, natural language processing (you don't have to say the phrase exactly as you specified) and it can parse the entitiies out of the command (and ask additional questions). But then the input has to originate from Google Assistant ...

        Comment


        • #49
          This looks very interesting and could be very useful. We have been an Alexa household for a few years now and the wife still has trouble saying the correct syntax. This sounds perfect for her. I just started playing with a trial license on an RPi 2 days ago. After I get a bunch of stuff moved over and have figured my main events I'll give this a try with the free Google Mini I got from Spotify.
          HS3 SEL running Pro Edition 3.0.0.531 on Ubuntu 18.04, mono 5.20, 656 devices, 209 events.
          Plug-Ins: Chromecast, Device History, EasyTrigger, Ecobee, JowiHue, LutronCaseta, MeiUnifi, PHLocation2, Pushover 3P, SDJ-Health, Sonos, WeatherFlow, weatherXML
          Scripts: SparkMan's Lock Event, 5 of Jon00 scripts.

          Comment


          • #50
            Originally posted by kriz83 View Post
            I also heavily rely on Tasker, but I don't really understand your flow. Why do you use the JSON interface to send the voice command instead of using e.g. the Google Home app ? I use the voice recognition of Google Home in my car to e.g. open the front gate and garage a few hundred meters before I get home.
            Why? Because in some cases I want to get a custom and variable response, or use a command that doesn't include On/Off/Dim/Set/etc. I also don't want to have to setup separate trigger commands in HS and the Google Home app for special commands. I'd like all of my voice endpoints to work the same way (accept the same commands) and have to set it up in only 1 location. My Automate flow can also be launched via a shortcut on my home page and starts listening immediately which makes it a better option than HSTouch on my phone since HSTouch takes a while to launch. It would likely be equivalent to using Google Home/Assistant on my phone if I wasn't using a special command or needed a custom/variable response.

            Originally posted by kriz83 View Post
            This plugin does not use the HomeSeer 'speech recognition', but it uses the one from DialogFlow (which is part of Google). As far as I am aware, it is not possible to parse the raw voice command from GH within HomeSeer, that's why I'm using DialogFlow. It has a downside: setup if more complex, but also a lot of upside: more supported languages, natural language processing (you don't have to say the phrase exactly as you specified) and it can parse the entitiies out of the command (and ask additional questions). But then the input has to originate from Google Assistant ...
            Yes, it is possible to have HomeSeer parse a voice command from GH. I have done it using IFTTT using both the HS Action and a Webhooks https request action. The problem is I can't get the response back to GH and have it be spoken using just IFTTT. From my research, this appears to be possible with a Google Action built using API.ai. Unfortunately it appears you need to jump through hoops to keep the action private and usable for more than 30 min.

            I definitely see the value in your method and DialogFlow. Seems very cool and powerful. But like I mentioned earlier, I really would like all of my different voice endpoints to behave the same way, and right now they all use the built in HS voice processing along with Jon00s Alexa helper (even though I don't have any Echos).

            Does DialogFlow allow you to make an https request (maybe via myHS), parse the response, and make GH say something based on that data? Are you utilizing API.ai at all? I'm wondering how easy it would be for you to add what I am talking about as an 'option' in your plugin for those that have requirements like I do, or for those that want to keep things a little more simple.

            Thanks!

            Comment


            • #51
              Originally posted by mrceolla View Post
              Does DialogFlow allow you to make an https request (maybe via myHS), parse the response, and make GH say something based on that data? Are you utilizing API.ai at all? I'm wondering how easy it would be for you to add what I am talking about as an 'option' in your plugin for those that have requirements like I do, or for those that want to keep things a little more simple.
              Api.ai has been renamed to DialogFlow once Google acquired it. I'm trying to make it work with myHS in the next version (hope to release it next week), but that would mean that you will have to store the myHS credentials in the DialogFlow website, which I personally don't find very appealing.

              Originally posted by mrceolla View Post
              I definitely see the value in your method and DialogFlow. Seems very cool and powerful. But like I mentioned earlier, I really would like all of my different voice endpoints to behave the same way, and right now they all use the built in HS voice processing along with Jon00s Alexa helper (even though I don't have any Echos).
              In theory you could trigger the vb script from Jon00 (I believe that is were the logic is, right ?). I could make it that if the string it is searching for is empty, it will match on every voice input. Would that help ?

              Originally posted by mrceolla View Post
              The problem is I can't get the response back to GH and have it be spoken using just IFTTT. From my research, this appears to be possible with a Google Action built using API.ai. Unfortunately it appears you need to jump through hoops to keep the action private and usable for more than 30 min.
              The plugin does just that. You can have multiple responses, a random one is choosen. You can also return the text from a script to make it even more dynamic. Where do you get that 30 min thing from ?

              Comment


              • #52
                I read about the 30 minute thing in a few places, here is one:

                https://stackoverflow.com/questions/...or-linked-acco

                ...but that one is about 2 years old. Maybe that's not the case anymore.

                I'm not sure I understand your suggestion. I'm not sure how Jon00's helper works but I have a trigger in there setup to fire on multiple versions of the same phrase, then it runs a vb script with logic to set a global variable, then Jon00's helper puts the contents of that global variable into the JSON response. I'd like GH to read that response.

                Yes I realize your plugin can also have custom responses, but it appears to be a separate 'system' which means my GH would work differently than my phone, HSTouch, etc., and/or I would have to set things up twice to keep them in sync.

                I'm basically looking to get the functionality of HomeSeer's old Google Action/Skill before they were limited to offering just the one that they have now. So I guess that technically shouldn't require a HS plugin at all, it could just be a special Google Action leveraging the existing JSON API. I think I'm going to have to look into building one.

                What happens on the HS side with your plugin?

                Comment


                • #53
                  As far as I am aware, that 30 day restriction is no longer in place.

                  It is not a separate system per se, it hooks into the Google Assistant. So you can use the same voice commands from your phone (using the google home app) as your Google Home. It won't work from HSTouch, that is true. But I'm not using that anyways :-)

                  Unlike the Alexa skill, the old Google Action skill is unfortunately no longer available. That is the reason I created this plugin.

                  What happens on the HS side of the plugin ? It receives the web callback and do performs the actions you defined in the plugin:
                  • Run an event
                  • Run a script
                  • Future: CAPI control of device
                  • Future: JSON api call

                  Comment


                  • #54
                    Hope to release the next version this week if i get the issue with the textbox on the config page fixed.

                    New features will include
                    • web configuration page
                    • Regex matching
                    • Ability to run single script line without the need to create a script
                    • Basic authentication for the webhook

                    Comment


                    • #55
                      You asked if anyone would like this functionality? Pretty much any HS user with assistant.

                      Even if they don't know they need it, they do and will.
                      We love GH. I'd like to use this, but after reading this thread along with the documentation, I doubt I'd be successful. I suspect a youtube of configuration video would help. I find videos make a process not so daunting even if they don't cover everything. I wish more plugins were documented on youtube. Hardly any are.



                      If you finish the GUI part of this working, I'd happily pay for the app

                      Comment


                      • #56
                        I just got a google home mini and messing around with possibilities and stumbled on your plugin. I do some C# and VB developing at work so I'm not too put off by the setup here, BUT I live in Denmark and I noticed that DialogFlow only lists "Danish" as supported in "Speech to Text" and not "Text to speech".

                        From your your example with coffee I'm guessing it requires Text-to-speech as well.

                        Comment


                        • #57
                          Went ahead and began installing working on HTTPS at the moment, but I did have a problem activating the plugin:
                          feb-07 22:47:02 Error Initializing plug-in(2): DialogFlow Instance::Trial ExpiredSTACK: Server stack trace: Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessag e(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(Mess ageData& msgData, Int32 type) at HomeSeerAPI.IPlugInAPI.InitIO(String port) at Scheduler.clsHSPI.CheckInterfaces()
                          feb-07 22:47:02 DialogFlow Entering InitIO
                          feb-07 22:47:02 Starting Plug-In Initializing plugin DialogFlow ...
                          feb-07 22:47:02 Info Plugin DialogFlow has connected. IP:127.0.0.1:54165
                          feb-07 22:46:55 Updater Install/Update of package Google Assistant DialogFlow Beta was successful.
                          feb-07 22:46:55 Updater Copying file: C:\Program Files (x86)\HomeSeer HS3\HSPI_DialogFlow.exe
                          feb-07 22:46:55 Updater Copying file: C:\Program Files (x86)\HomeSeer HS3\HSPI_DialogFlow.exe.config
                          feb-07 22:46:55 Updater Warning File exists, not overwriting: C:\Program Files (x86)\HomeSeer HS3\Config\\HSPI_DialogFlow.ini
                          feb-07 22:46:55 Updater Copying file: C:\Program Files (x86)\HomeSeer HS3\bin\HSPI_DialogFlow\Newtonsoft.Json.license.txt
                          feb-07 22:46:55 Updater Copying file: C:\Program Files (x86)\HomeSeer HS3\bin\HSPI_DialogFlow\Newtonsoft.Json.dll
                          feb-07 22:46:55 Updater Copying file: C:\Program Files (x86)\HomeSeer HS3\bin\HSPI_DialogFlow\IniParser.dll
                          feb-07 22:46:55 Updater Installing package Google Assistant DialogFlow Beta
                          feb-07 22:46:55 Updater Using local file: HSPI_DialogFlow-0.0.2.0.zip
                          feb-07 22:46:55 Updater Downloading update: Google Assistant DialogFlow Beta

                          Comment


                          • #58
                            Can you download the file again from the opening post ? I extended the Trial period. Still strugglin with getting the editbox to work ok ...

                            Comment

                            Working...
                            X