Announcement

Collapse
No announcement yet.

I got "Hey Google, ask HomeSeer _____" working, with custom responses

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • I got "Hey Google, ask HomeSeer _____" working, with custom responses

    Edit: I've updated this post with new files and screenshots required to enable conversation functionality. If HS returns a response with a ? question mark in it, the Google device will listen again for another command. This is most useful with Jon00's Alexa helper but can also be used to confirm certain HS commands that require confirmation, such as "Run event _____".

    Changes include:
    1) Update Default Welcome Intent to deselect "Set this intent as end of conversation"
    2) Updates to index.js and package.json in the attached zip file

    -------------------------------------------------------------------------------------------------------------

    It was much harder than it could have been, but I think I figured it out. Here are the steps as best I remember them. If you encounter troubles let me know and I'll amend this as needed.

    FYI, I use Jon00's Alexa helper for the custom responses.

    This solution utilizes HomeSeer's JSON API for request=voicecommand

    1) Create Dialogflow project

    2) Under Intents, click "Default Welcome Intent" to edit it. Make it look like this:

    Click image for larger version  Name:	screen1.PNG Views:	26 Size:	291.0 KB ID:	1309399
    Click image for larger version  Name:	screen2.PNG Views:	23 Size:	138.2 KB ID:	1309400

    3) Now go to the Fulfillment page and enable the inline editor. Paste in the code I supplied in the attached files, after editing the URL and username/password in index.js. I'm using a direct URL to my HS machine, but you should be able to use your MyHS URL as well.

    4) Test in Dialogflow to make sure your custom webhook is working correctly (see caveat below about making http calls)

    5) Go to Integrations, click on Google Assistant, and then Test. You will be taken to Actions on Google where you can test using the complete Assistant experience. Run some tests

    6) Go to Invocation and change Display Name to "HomeSeer". You will get an error. Edit the pronunciation and enter "Home Seer" in two words. The error should go away. Now you can invoke your action using "Tell HomeSeer", "Ask HomeSeer", etc., however you will hear an annoying introductory message about getting your test app. You can get rid of this later by clicking on the Device icon in the Simulator tab located in the upper right to disable testing on your devices. Note: you could choose a different word/name for the Invocation if you wanted to.

    7) Go to Release, and Deploy your action to Alpha with you and anyone else that you want as the whitelisted testers. You will need to fill out a bunch of fields elsewhere before it will let you deploy an alpha. I just entered "test" and other garbage information. Nobody should ever see this. When entering testers, copy and email yourself the opt-in link. You will need to open this on a phone with Assistant installed. Do so and accept the Alpha tester status. You may need to wait until the deployment is complete to finish this step. Deployment can take some time.

    8) Be sure to disable Device Testing in Actions on Google in the Simulator by clicking on the device icon in the upper right. If this is enabled you will continue to receive the introductory test message when invoking your action.

    I think that's it. Let me know if you hit any snags.

    Some caveats:

    - When you first start using this action, Google may say, "Ok, getting HomeSeer" before starting the action. Once you use the action enough times, this introductory message is removed.

    - You need to link a Google billing account to this Google Project before you can make http calls to your HomeSeer machine from the code in the Inline editor. From my understanding, this type of code/traffic will never go beyond the free and/or credited use. I've been using this for months now and have not paid anything.

    - You must store a HS username and password in the code in plain text (if someone knows how to obfuscate, please let me know)

    - After your command, you will hear a 'ding', the HS response, then another 'ding'. This is to let you know your Action is starting/stopping

    - The spoken HS response will not use the typical Assistant voice. You must select 1 from a list in Actions on Google Invocation tab. I disabled the option that only showed US voices so I could select a British GB voice

    - I have inadvertently re-enabled device testing by poking around either Dialogflow or Actions on Google, which re-enables that introductory message. I think it is due to the option in Dialogflow under Assistant Integrations that automatically pushes changes. If that happens to you, just return to Actions on Google Simulator tab and disable device testing again.

    I hope it works for you. Enjoy!
    Attached Files
    Last edited by mrceolla; June 7th, 2019, 06:17 PM. Reason: Added conversation functionality. If the response from HS contains a "?", the Google device will listen again.

  • #2
    FYI, I was able to eliminate the need to say "Ask HomeSeer..." for specific commands by creating a custom "Routine" in the Google Home app. Unfortunately this method seems to take longer to execute and hear the response. Not sure why yet.

    Comment


    • #3
      This sounds awesome. I started to follow your steps but the zip contents appear to be a dead link. Is there any way you could share that again or post it in a different format?

      thanks in advance.

      Comment


      • #4
        Originally posted by thejtdude View Post
        This sounds awesome. I started to follow your steps but the zip contents appear to be a dead link. Is there any way you could share that again or post it in a different format?

        thanks in advance.
        Thanks for letting me know. Please try to download the .zip from the Attached Files section at the bottom of the post. I tried using the editor tools to insert a link to the file into the body of the post, but that must not have worked correctly. I just removed that link.

        Comment


        • #5
          I wanted to add some notes about the URL in the index.js file. I personally target my HS machine directly whenever I can just because I don't want to rely on myHS being operational. When targeting your machine directly, especially outside of your home, you should use https. I had trouble setting up HS's built in SSL, and then they removed it altogether, so someone helped me setup an Apache proxy:

          https://forums.homeseer.com/forum/ho...r-poodle/page2

          I tell you all of this to perhaps help you, but mainly to explain this line in index.js:

          process.env.NODE_TLS_REJECT_UNAUTHORIZED='0'; // allows self-signed certificate https requests

          I did not pay for a certificate, so it is self-signed. You may be able to delete this line if it does not apply to how you are connecting to your HS machine.

          That said, if you want to keep it secure, yet simple, I believe you could just target your myHS via connected2.homeseer.com according to:

          https://forums.homeseer.com/forum/ho...085#post962085

          Though I have not tried that. If using myHS, you should be able to delete that UNAUTHORIZED line.

          Comment


          • #6
            I like the local connection idea. That is always my preference as well. New link works well. Thanks for the quick update!

            Comment


            • #7
              Originally posted by thejtdude View Post
              I like the local connection idea. That is always my preference as well.
              I believe that code in index.js runs out on the cloud somewhere, so the URL or IP address used would need to be a public URL or IP. I could be mistaken. I did not try a local IP address.

              Comment


              • #8
                Not that anyone cares, but I've updated the information in the first post to allow for conversations. Before, you could only send one command and then the conversation was over. Now, if the response from HS contains a "?", the Google device will listen for a follow-up command. This could go on indefinitely.

                I've been using this for months now and I love it. Combined with Jon00's Alexa helper, it is powerful. I can now ask for information from HS, and HS can ask follow-up questions. For commands I use often, I setup a Google Home routine so that I can skip saying "Tell Homeseer...".

                Examples:

                Background: I have HSM200s around the house that light up different colors for different things. They light up if a door is open or unlocked.
                Command 1: Hey Google, which door is unlocked?
                Answer 1: The front entry door is closed and unlocked. Would you like me to lock it?
                Command 2: Yes
                Answer 2: Ok, the front door is now locked.

                Command 1: Hey Google, what is the temperature in the house?
                Answer 1: The temperature is 75 degrees. Would you like me to turn on the air conditioning?
                Command 2: Yes
                Answer 2: Ok, the air conditioning is now on and set to 72 degrees. Would you like to adjust the temperature?
                Command 3: Yes, set to 70 degrees.
                Answer 3: Ok, setting air conditioning to 70 degrees.


                I hope someone else finds this useful.

                Comment


                • #9
                  Great idea but what a nightmare of a sparse documentation for the needed steps. PLENTY of pitfalls. Took me 2 hours just to get the fulfillment inline editor to work with all the steps necessary to activate firebase and activate sorage space within firebase. Now I'm stuck at the intents. Have everything EXACTLY as shown in the screenshot, yet any phrase I type in the "try it now" field on the top right yields a DEFAULT FALLBACK INTENT with the default response of WHAT WAS THAT? and the action being "input.unknown".
                  Looks like there is another important part of the documentation missing on how does one get the Default Welcome Event to trigger on ANY phrase entered ??

                  Comment


                  • #10
                    Hmm. Sorry you're having trouble. I did minimal documentation above since it was from memory and I wasn't sure what the interest would be like. But I'm happy to try to help.

                    Are you testing the flow on Dialogflow or Actions on Google (Google Assistant)? Be sure to test it in Google Assistant via the link below the "Try it now" textbox. I don't think the "Try it now" activates the webhook.

                    If it still doesn't work there, make sure that your firebase code is running? You can view the log to confirm (I think the log is on yet another site, but link should be there). If not, make sure "Enable webhook call for this intent" is enabled as in the screenshot.

                    If you're sure your code is running, next make sure it is returning something. Perhaps there is a communication problem with your HS machine and nothing is getting returned. Hard code a return value if you need to. Use console.log() to make sure the expected areas of code are executing. If you're not familiar with coding or having trouble figuring out index.js, let me know.

                    Lastly, make sure to invoke your Action using the right words, like "Tell HomeSeer" followed by a command.

                    Let me know if you have any questions about any of that.

                    Comment


                    • #11
                      Unfortunately not working
                      If I put anything else in the default welcome intent under training phrases like "Water" and then type any sentence that includes the word "water" my script will run, I see a log entry on my homeseer and an error comes back because homeseer won't understand the sentence. As soon as I make the intent look like the sceenshot, it will never activate, no matter what I type.
                      Next, I tried to follow the steps with the "see how it works in google assistant" and there I am in the simulator where no matter what I enter it always says "Sorry I don't understand" and no log entry on the homeseer server. So I know it is again not firing the intent and instead going to the default fallback intent.

                      I have never encountered anything as cryptic and unintuitive as this Google platform LOL!
                      And yes I can code. I have written my own plugin for HS3 before and Windows and Mac applications as well as microcontroller code.

                      It seems though as if HS3 has a major flaw in itself anyway. The regular commandline for testing works fine:

                      "mydomain.org/JSON?request=voicecommand&phrase=turn off kitchen counter"


                      but as soon as you put something like turn off kitchen lights I get an error "More than one device was found with that name combination."

                      In the regular Google assistant / MyHS link you can say "turn off the kitchen lights" and all lights in the kitchen "room" will turn off which is really nice. I have 5, so if it's not possible to make HS3 turn all of them on or off at the same time using its built in voice parsing, then this is all useless anyway

                      Comment


                      • #12
                        Sorry you're still having trouble. There is further setup required in the Google Assistant area. Did you perform those steps in #5 and #6 above, and are you using the correct Invocation phrase in the test area based on #6?

                        I just tested again in Dialogflow and there you DO NOT want to use "Ask HomeSeer..." or whatever the invocation phrase is, but in Google Assistant you do. I also noticed in Dialogflow that I couldn't see HomeSeer's response anywhere, but the action was executed on HS just fine. This is why I thought in my previous post you couldn't do testing in there anymore. Turns out you should be able to. I fear you performed an extra step somewhere or your setup does not quite match the screenshot. Note, my fields inside Context are blank. I just logged into Dialog flow and my setup still matches the screenshot. Feel free to post a screenshot of your setup page if you'd like another set of eyes to look it over.

                        Regarding your light(s) issue... I highly recommend creating virtual devices or lighting scenes/groups all renamed within Google Home to "Lights" or "The Lights" and assigned to each room of your house that has a Google Home device. Then you can say generic commands like "turn on the lights" and it will turn on that 1 device, which in turn can do whatever you want (ie. set however many lights in that 1 room to 75%). It enables generic commands to be location based yet fine tuned to what you want. I almost never want my lights at 100% in any room of my house, so without this, generic commands would be useless to me.

                        Do you use Insteon or Z-wave for lighting? I use Insteon and they have "Groups" which is the best way to control multiple lights at one time. It represents as a single device but controls however many you want to whatever percent you want at whatever ramp rate you want. And of course it can turn them all off simultaneously as well.

                        Comment


                        • #13
                          Click image for larger version

Name:	Screenshot 2019-08-28 21.53.22.png
Views:	106
Size:	340.8 KB
ID:	1323215


                          Here is my Dialogflow screenshot. Exactly like yours. Contexts is empty.


                          Next screenshot is what always happens, no matter what I type in the "try it" text box. The script NEVER fires because the WELCOME intent is NEVER matched. It always matches the fallback intent which gives the generic answer that the input was not understood.

                          Click image for larger version

Name:	Screenshot 2019-08-28 21.56.25.png
Views:	110
Size:	264.3 KB
ID:	1323216

                          Comment


                          • #14
                            I think I spotted it.

                            Note your dollar sign in $textCommand is not highlighted yellow. Note underneath it. In my setup, the value under Resolved Value is "$textCommand" with the dollar sign. So there is something wrong with your Training phrases section.

                            Click image for larger version  Name:	Capture.PNG Views:	0 Size:	107.5 KB ID:	1323335

                            I just tested creating a new phrase. I typed "$textCommand2" then hit enter. The bottom row then appeared. I could edit the parameter name and entity. To change the Resolved Value I needed to drag a handle which was previously only around the $ of $textCommand2.

                            Hopefully that solves your problem.

                            Comment


                            • #15
                              What do you mean drag a handle? I can't drag anything anywhere. All it does is select the text in the browser ?

                              [UPDATE]: I got it to show as you described. Just fiddled around with it, deleted and entered new a few times until I finally got that "$" to show up. What a mess of an unintuitive interface! OK next problem.
                              HS3 now "sees" the input correctly but still nothing works. I'm getting an unhandled rejection, whatever that means.
                              Here's the snippet from the firebase log:
                              12:07:34.045 PM
                              dialogflowFirebaseFulfillment Dialogflow Request body: {"responseId":"8991b996-3f45-4de8-af5b-f87d7fcfcd73-ee1dc704","queryResult":{"queryText":"turn off kitchen counter","action":"input.welcome","parameters":{"textCommand ":"turn off kitchen counter"},"allRequiredParamsPresent":true,"fulfillmentMessag es":[{"text":{"text":[""]}}],"intent":{"name":"projects/hs3-2019-08-26/agent/intents/7a9c2a62-216b-472e-bc74-fff77833b7cc","displayName":"Default Welcome Intent"},"intentDetectionConfidence":0.3,"languageCode":"en" },"originalDetectIntentRequest":{"payload":{}},"session":"pr ojects/hs3-2019-08-26/agent/sessions/ecc0c282-6a4f-b5fa-3c55-52d94ca51fad"}
                              12:07:34.691 PM
                              dialogflowFirebaseFulfillment Unhandled rejection
                              12:07:34.691 PM
                              dialogflowFirebaseFulfillment
                              TypeError: Cannot read property 'close' of null at rp.get.then.jsonBody (/srv/index.js:42:26) at <anonymous> at process._tickDomainCallback (internal/process/next_tick.js:229:7)
                              12:07:34.699 PM
                              dialogflowFirebaseFulfillment Function execution took 1783 ms, finished with status: 'crash'

                              Comment

                              Working...
                              X