No announcement yet.

Recovering from devices that didn't respond

  • Filter
  • Time
  • Show
Clear All
new posts

    Originally posted by Furious View Post

    Could I also have a copy please? I've been playing with something similar with MQTT, problem is its way too involved
    Same. It gets way too complex as is. I haven't touched it in a few days now. There were two routines; one to mirror the real device when its status changes, the other to "reflect" the status back when the real device becomes invalid status. The problem I'm having is not triggering both events repeatedly when one or the other device status changes. It needs more work before I'll feel comfortable with distributing it.
    Real courage is not securing your Wi-Fi network.


      Originally posted by Wadenut View Post

      Same. It gets way too complex as is. I haven't touched it in a few days now. There were two routines; one to mirror the real device when its status changes, the other to "reflect" the status back when the real device becomes invalid status. The problem I'm having is not triggering both events repeatedly when one or the other device status changes. It needs more work before I'll feel comfortable with distributing it.
      No worries - I'll crack on with my MQTT/node red flow to see if I can attack the problem from a different angle


        Originally posted by Furious View Post

        No worries - I'll crack on with my MQTT/node red flow to see if I can attack the problem from a different angle
        Different approaches might get us further in the long run. Without the work of Joseph Swan, Warren de La Rue, Nicola Tesla and others, Thomas Edison might have given up on the light bulb.

        The approach is this:
        Create a Mirror device to contain the assumed state of the Z-Wave device. Whenever the state of the Z-Wave device changes, and does not become "Invalid" an event is triggered to copy the state of that device to the mirror. When the state of the Z-Wave device does become invalid, another event is triggered to command the Z-Wave device to change to the same state as the Mirror.
        So, if you have 50 Z-Wave devices, you also need to create 50 Mirror devices and two sets of 50 Events. You can see how this already could become very complex, and while it sounds simple enough in theory once implemented, in practice it isn't. I've seen problems in working with only a single Lock. The lock can become unresponsive, either due to too many commands received, or the Z-Wave network having too many commands queued at once. The result: other Z-Wave devices also appear to become sluggish.
        That said, check your mail.
        Real courage is not securing your Wi-Fi network.


          Ok, I've come to a bit of a dead end with normal API methods - JSON or the ASCII APIs both only give results of a command - in other words, there is no way to trap for the last command sent.

          JSON is a bit better as the status will eventually change to unknown, and it's trivial to store the last known state - but it limits what you'd want to do as you'd have to assume that the device is changing from off to on or vice versa.

          Scraping the log for entries is the only avenue.
          Node red makes this easy - what I can do is install the sqlite node, then its as simple as defining the db file (homeseer_log.hsd) in read only mode, then firing a query off every 5 seconds with something like this:
          select Log_Entry from log where Log_Type = 'Device Control' AND Log_Entry LIKE 'Device:%to%' AND Log_DateTime between datetime('now', 'localtime','-5 seconds') AND datetime('now','localtime');
          This gives me the last 5 seconds of entries where HS has tried to set a device. Parse the log entry and keep a table of values which are devices and their set state.

          What I can then do is periodically dump device status every half a second using a JSON query. Then, if it sees an unknown status perform some logic and try 3 times to set the device again every 5 seconds via MQTT. It could then check if its still unknown and maybe send an alert?


            In HS2 there was a facility to perform debug logging at various detail levels. I'm sure this exists in HS3 as well.

            I'm thinking of using this to retrieve whatever the last command to a failed device might have been. This would still require an event to trigger on "Invalid Status" for a particular device; thus one event per device; then a script could query the debug log and resend the last failed command to that device. Thus we avoid the need for Mirror devices.
            The only thing is that once enabled, debug logs can grow rather large very quickly, so the script would also need to prune or delete the log when executed. There would be a small chance of sharing violations on the log file, but this is easy to overcome.
            Real courage is not securing your Wi-Fi network.


              Scrap that.
              In HS3 (not so in HS2), all the debug logs are also written to the main log.
              Real courage is not securing your Wi-Fi network.


                I've got something in place now, and testing - node red flow does a compare old and new values for a device, and if the status goes to "Unknown", triggers the following:
                1. Runs the sqlite query to get the device control log entry
                2. Parses that to get the value being set
                3. Resends the value to the device
                4. Waits for 4 seconds and checks the device status via JSON
                5. If it's still "Unknown", increment a counter
                6. Loop steps 3-5 two more times
                7. Send a notification (MQTT) with the device ref and name

                I'll leave it in place to see how it fares. Key point is to only check the log if the device status is changed to unknown, so the log doesn't need to be constantly polled.


                  Won't that require logging all device actions? That's something I'd wanted to avoid.
                  Real courage is not securing your Wi-Fi network.


                    The node red flow does a JSON status request every 600ms and that gets all devices and status. It's low impact, and I've not seen any noticeable cpu usage from doing so. It also facilitates all device control via MQTT.


                      Ah. Sounds promising then

                      I'll look forward to trying your method.
                      Real courage is not securing your Wi-Fi network.


                        Node red installed and working on a server
                        The Mosca and sqlite nodes added to the palette.
                        MQTT server configured in the mosca node to pull/push MQTT updates
                        SQLite node to point to your homeseer_log.hsd file - relax, it's only in read only mode
                        JSON turned on in homeseer

                        Excuse the mess of it, it's not optimised for looking pretty or efficiency. The sqlite output was particularly weird to work with as it didn't seem to parse into the payload object nicely.
                        [{"id":"b0cf0fa.f22f8f","type":"tab","label":"Homeseer to MQTT control","disabled":false,"info":""},{"id":"b5fac3f5.ad881","type":"http request","z":"b0cf0fa.f22f8f","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"","tls":"","persist":false,"proxy":"","authType":"","x":730,"y":240,"wires":[["b321fafd.505d98","8579c89c.7ff628","446b08c4.7198e8"]]},{"id":"785ae009.ad819","type":"inject","z":"b0cf0fa.f22f8f","name":"Initial Request","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":true,"onceDelay":"0.5","x":160,"y":240,"wires":[["6b33fa9f.4c91e4"]]},{"id":"b321fafd.505d98","type":"function","z":"b0cf0fa.f22f8f","name":"Logic","func":"// Variables\nvar homeseerdevices = JSON.parse(msg.payload);\n\n// Clear msg.payload\nmsg.payload = {}\n\nfor(var i =0; i<homeseerdevices.Devices.length;i++){\n    // Setup some variables\n    var prevchange = context.get('dev'+i);\n    var lastchange = homeseerdevices.Devices[i].last_change;\n    var ref = homeseerdevices.Devices[i].ref;\n    var name = homeseerdevices.Devices[i].name;\n    var value = homeseerdevices.Devices[i].value;\n    var device_type_string = homeseerdevices.Devices[i].device_type_string;\n    var status = homeseerdevices.Devices[i].status;\n    var status_image = homeseerdevices.Devices[i].status_image;\n    var associated_devices = homeseerdevices.Devices[i].associated_devices[0];\n    var location = homeseerdevices.Devices[i].location;\n    var location2 = homeseerdevices.Devices[i].location2;\n    \n    // Check if the device has a battery and identify the root device\n    if (name.includes(\"Battery\") || device_type_string.includes(\"Battery\") || status_image.includes(\"battery\") ){\n        context.set('batt'+associated_devices,value);\n    }\n    \n    // Check if device has new data\n    if (typeof prevchange == 'undefined' || prevchange === null){\n        var prevchange = lastchange;\n        context.set('dev'+i,lastchange);\n    }\n    // Output new data\n    if (prevchange != lastchange) {\n        msg.payload.ref = ref;\n = name;\n        msg.payload.value = value;\n        msg.payload.status = status;\n        msg.payload.location = location;\n        msg.payload.location2 = location2;\n        // Check if should attach battery\n        var batt = context.get('batt'+associated_devices);\n        if (typeof batt !== 'undefined' && batt !== null && batt >= 1){ msg.payload.battery = batt;}\n        context.set('dev'+i,lastchange);\n        return msg;\n    }\n}","outputs":1,"noerr":0,"x":890,"y":240,"wires":[["5120289f.8eb888","13907b0f.5afef5"]]},{"id":"8579c89c.7ff628","type":"function","z":"b0cf0fa.f22f8f","name":"Clear","func":"msg.payload = {}\nreturn msg;","outputs":1,"noerr":0,"x":410,"y":280,"wires":[["9b36035b.411e6"]]},{"id":"9b36035b.411e6","type":"delay","z":"b0cf0fa.f22f8f","name":"","pauseType":"delay","timeout":"600","timeoutUnits":"milliseconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":550,"y":280,"wires":[["b5fac3f5.ad881"]]},{"id":"5120289f.8eb888","type":"mqtt out","z":"b0cf0fa.f22f8f","name":"","topic":"homeseer/out","qos":"","retain":"","broker":"bc36b861.c8f2a8","x":1060,"y":240,"wires":[]},{"id":"740c7b7c.c05ec4","type":"comment","z":"b0cf0fa.f22f8f","name":"Output HomeSeer device status to MQTT","info":"","x":180,"y":200,"wires":[]},{"id":"e030cd8a.58e8f","type":"comment","z":"b0cf0fa.f22f8f","name":"Control HomeSeer devices via MQTT","info":"","x":170,"y":80,"wires":[]},{"id":"8911283a.d298a8","type":"mqtt in","z":"b0cf0fa.f22f8f","name":"","topic":"homeseer/in","qos":"2","datatype":"auto","broker":"bc36b861.c8f2a8","x":130,"y":120,"wires":[["36c6ecc0.97b484"]]},{"id":"36c6ecc0.97b484","type":"function","z":"b0cf0fa.f22f8f","name":"Logic","func":"// Variables\nvar homeseerjson = JSON.parse(global.get('homeseerjson'));\nvar mqtt = JSON.parse(msg.payload);\nvar ref = mqtt.ref;\nvar value = mqtt.value;\nvar label = mqtt.label;\nvar name =;\n\n// Convert name to ref (if applicable)\nif (typeof name !== 'undefined' && name !== null){\n    for (i = 0; i < homeseerjson.Devices.length; i=i+1) {\n        var hsname = (homeseerjson.Devices[i].name);\n        if (name == hsname){\n            var ref = homeseerjson.Devices[i].ref;\n        }\n    }\n}\n\n// Control by label\nif (typeof label !== 'undefined' && label !== null){\nmsg.url = ('http://'+global.get('homeseerserver')+'/JSON?request=controldevicebylabel&ref='+ref+'&label='+label);\n}\n\n// Control by value\nif (typeof value !== 'undefined' && value !== null){\nmsg.url = ('http://'+global.get('homeseerserver')+'/JSON?request=controldevicebyvalue&ref='+ref+'&value='+value);\n}\n\n// Send message integrity check (did we get a ref?)\nif (typeof ref !== 'undefined' && ref !== null){\nreturn msg;\n}","outputs":1,"noerr":0,"x":270,"y":120,"wires":[["1376f75a.2f6179"]]},{"id":"1376f75a.2f6179","type":"http request","z":"b0cf0fa.f22f8f","name":"Make HTTP Request","method":"GET","ret":"txt","url":"","tls":"","x":440,"y":120,"wires":[[]]},{"id":"446b08c4.7198e8","type":"delay","z":"b0cf0fa.f22f8f","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"30","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":true,"x":140,"y":400,"wires":[["292343ec.6e567c"]]},{"id":"9883999f.a41798","type":"comment","z":"b0cf0fa.f22f8f","name":"Globally store all HomeSeer devices on interval (used for name instead of ref control feature)","info":"","x":340,"y":360,"wires":[]},{"id":"292343ec.6e567c","type":"function","z":"b0cf0fa.f22f8f","name":"Store Globally","func":"global.set('homeseerjson',msg.payload);\nreturn msg;","outputs":1,"noerr":0,"x":320,"y":400,"wires":[[]]},{"id":"13907b0f.5afef5","type":"switch","z":"b0cf0fa.f22f8f","name":"Check for unknown","property":"payload.status","propertyType":"msg","rules":[{"t":"cont","v":"Unknown","vt":"str"}],"checkall":"true","repair":false,"outputs":1,"x":1070,"y":360,"wires":[["75b262e7.3e65dc"]]},{"id":"75b262e7.3e65dc","type":"function","z":"b0cf0fa.f22f8f","name":"Construct Device name for log","func":"flow.set('LogNamedDevice',msg.payload.location2+' '+msg.payload.location+' ';\nflow.set('Refdevice',msg.payload.ref);\nmsg.topic = \"select Log_Entry from log where log_type = 'Device Control' AND Log_entry LIKE '%\"+flow.get('LogNamedDevice')+\"%' AND Log_DateTime between datetime('now', 'localtime','-8 seconds') AND datetime('now','localtime');\"\nreturn msg;","outputs":1,"noerr":0,"x":1110,"y":400,"wires":[["afddd439.910db8"]]},{"id":"afddd439.910db8","type":"sqlite","z":"b0cf0fa.f22f8f","mydb":"c1d096d1.6fb868","sqlquery":"msg.topic","sql":"select Log_Entry from log where log_type = 'Device Control' AND Log_entry LIKE 'Device:%to%' AND Log_DateTime between datetime('now', 'localtime','-5 seconds') AND datetime('now','localtime');","name":"Homeseer Log DB","x":1070,"y":440,"wires":[["5b4c0556.37f2ac"]]},{"id":"788cc16a.95669","type":"comment","z":"b0cf0fa.f22f8f","name":"Source -","info":"","x":710,"y":20,"wires":[]},{"id":"8ecfc7f7.3878b8","type":"function","z":"b0cf0fa.f22f8f","name":"Parse payload value + format msg to send MQTT","func":"var deviceval = msg.responseUrl;\nvar deviceval2 = deviceval.split(\"(\");\nvar deviceval3 = deviceval2[1].split(\")\");\nflow.set('DevValue',deviceval3[0]);\nmsg.payload.value = deviceval3[0];\nmsg.payload.ref = flow.get('Refdevice');\nflow.set('count',0);\nreturn msg;","outputs":1,"noerr":0,"x":1430,"y":500,"wires":[["b34217f9.0e4848","93b9d1ef.daac7"]]},{"id":"b03d8ebe.c70f8","type":"mqtt out","z":"b0cf0fa.f22f8f","name":"","topic":"homeseer/in","qos":"2","retain":"false","broker":"bc36b861.c8f2a8","x":2110,"y":500,"wires":[]},{"id":"b34217f9.0e4848","type":"delay","z":"b0cf0fa.f22f8f","name":"","pauseType":"delay","timeout":"4","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":1060,"y":580,"wires":[["8061676f.ac0558"]]},{"id":"84b55829.658a98","type":"http request","z":"b0cf0fa.f22f8f","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"","tls":"","persist":false,"proxy":"","authType":"","x":1550,"y":580,"wires":[["802e10f6.1d939"]]},{"id":"8061676f.ac0558","type":"function","z":"b0cf0fa.f22f8f","name":"Set URL and clear payload","func":"msg.url = \"http://'+global.get('homeseerserver')+'/JSON?request=getstatus&ref=\"+flow.get('Refdevice');\nmsg.payload = {};\nreturn msg;","outputs":1,"noerr":0,"x":1280,"y":580,"wires":[["84b55829.658a98"]]},{"id":"802e10f6.1d939","type":"function","z":"b0cf0fa.f22f8f","name":"Parse status again, increment count if unknown","func":"// Variables\nvar homeseerdevicesvalidate = JSON.parse(msg.payload);\n\nvar status = homeseerdevicesvalidate.Devices[0].status;\n\nif (status == 'Unknown'){\n    flow.set('count', flow.get('count')+1);\n    }\n    else\n    {\n    flow.set('count',0);\n    }\nreturn msg;","outputs":1,"noerr":0,"x":1860,"y":580,"wires":[["42d3c13.33dd24"]]},{"id":"42d3c13.33dd24","type":"switch","z":"b0cf0fa.f22f8f","name":"","property":"count","propertyType":"flow","rules":[{"t":"gt","v":"0","vt":"num"},{"t":"eq","v":"0","vt":"num"}],"checkall":"true","repair":false,"outputs":2,"x":1610,"y":660,"wires":[["629c1dd3.d0c7a4"],[]]},{"id":"629c1dd3.d0c7a4","type":"switch","z":"b0cf0fa.f22f8f","name":"","property":"count","propertyType":"flow","rules":[{"t":"eq","v":"3","vt":"num"},{"t":"lt","v":"3","vt":"num"}],"checkall":"true","repair":false,"outputs":2,"x":1770,"y":660,"wires":[["41dce533.7936bc"],["93b9d1ef.daac7","b34217f9.0e4848"]]},{"id":"ecf31ed3.a455e","type":"mqtt out","z":"b0cf0fa.f22f8f","name":"","topic":"Notify","qos":"","retain":"","broker":"bc36b861.c8f2a8","x":2090,"y":660,"wires":[]},{"id":"41dce533.7936bc","type":"function","z":"b0cf0fa.f22f8f","name":"","func":"msg.payload = {};\nmsg.topic = \"Notify/6\";\nmsg.payload = \"Device ref \"+flow.get('Refdevice')+\" (\"+flow.get('LogNamedDevice')+\") has a problem!\";\nreturn msg;","outputs":1,"noerr":0,"x":1930,"y":660,"wires":[["ecf31ed3.a455e"]]},{"id":"5b4c0556.37f2ac","type":"change","z":"b0cf0fa.f22f8f","name":"","rules":[{"t":"set","p":"responseUrl","pt":"msg","to":"payload[0].Log_Entry","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1100,"y":500,"wires":[["8ecfc7f7.3878b8"]]},{"id":"93b9d1ef.daac7","type":"function","z":"b0cf0fa.f22f8f","name":"Construct payload for MQTT control","func":"msg.payload = {};\nmsg.payload.value = flow.get('DevValue');\nmsg.payload.ref = flow.get('Refdevice');\nreturn msg;","outputs":1,"noerr":0,"x":1880,"y":500,"wires":[["b03d8ebe.c70f8"]]},{"id":"1a17000e.db7b1","type":"comment","z":"b0cf0fa.f22f8f","name":"Only send changes to  MQTT","info":"","x":900,"y":200,"wires":[]},{"id":"2474071c.911c98","type":"comment","z":"b0cf0fa.f22f8f","name":"Branch off if status has been set to Unknown","info":"","x":1090,"y":320,"wires":[]},{"id":"cda262a5.8fe05","type":"comment","z":"b0cf0fa.f22f8f","name":"Store values for looping","info":"","x":1460,"y":460,"wires":[]},{"id":"f5839054.659ec","type":"comment","z":"b0cf0fa.f22f8f","name":"Construct MQTT to try device again","info":"","x":1960,"y":460,"wires":[]},{"id":"7e2e7f6d.b09af","type":"comment","z":"b0cf0fa.f22f8f","name":"Loop 3 times and send notification","info":"","x":1740,"y":700,"wires":[]},{"id":"6b33fa9f.4c91e4","type":"function","z":"b0cf0fa.f22f8f","name":"homeseer IP","func":"global.set('homeseerserver',\"192.168.x.x:81\");\nreturn msg;","outputs":1,"noerr":0,"x":430,"y":240,"wires":[["39754eb5.71b522"]]},{"id":"b59c108.df85af","type":"comment","z":"b0cf0fa.f22f8f","name":"Set Homeseer server","info":"Format is:\n\"192.168.x.x:port\"","x":440,"y":200,"wires":[]},{"id":"39754eb5.71b522","type":"function","z":"b0cf0fa.f22f8f","name":"Set url","func":"msg.url = 'http://'+global.get('homeseerserver')+'/JSON?request=getstatus';\nreturn msg;","outputs":1,"noerr":0,"x":570,"y":240,"wires":[["b5fac3f5.ad881"]]},{"id":"bc36b861.c8f2a8","type":"mqtt-broker","z":"","name":"MQTTSERVER","broker":"","port":"1883","clientid":"myClient","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"c1d096d1.6fb868","type":"sqlitedb","z":"","db":"C:\\Program Files (x86)\\HomeSeer HS3\\Logs\\homeseerlog.hsd","mode":"RO"}]
                        There is one node with a comment above it - set homeseer server. Change the IP and port in there to what your homeseer server is.

                        Once deployed, the flow does the following:
                        Every 600ms (that can be reduced/increased based on responsiveness or server load), poll homeseer for all device status and dump to a global object
                        Compare new dump of status to existing stored statuses
                        If different, send difference to MQTT homeseer/out
                        If status = "Unknown" then branch off to:
                        1.Get homeseer log for specific device entry
                        2.Parse log entry to get value trying to be set - always in ()s
                        3.Send a MQTT message to send the same command again to the device
                        4.Wait 4 seconds
                        5.Get status for just the device again - if still "Unknown", then go to 1 and increment counter
                        6.If counter gets to 3, give up and send a notification.

                        Note - I have the zwave timeout set to 4 seconds - from observation from testing, the normal state of events is:
                        HS sends zwave command
                        Timeout elapses
                        HS stamps the log with the failure message
                        HS updates the zwave device to "Unknown"

                        So, if you have zwave set to x seconds, then you WILL have to increase the log query to accomodate - if its 10 seconds, then you'll need at least 11 seconds of log entries.

                        Additional thoughts:
                        The specificity of the log file extraction is down to the device naming - the device ref is already known, however the log doesn't stamp that, it stamps the full name (location2 location name). So, potentially you can derail the whole process by having a device with a duplicate name being set at around the same time and send the wrong value.
                        No way I'm going to try and get around that scenario - why the hell would you have devices with the same name, you'd never work out which is which anyway


                          Had it this morning with one module that consistently fails; usually on an Off command so I've replaced it with another module.
                          I brought the problem module next to the Z-Net and tried it here. The module seems to have the same problem even when in proximity with the interface. Time will tell if the new one functions well at it's location.
                          Just wondering if this problem is related to a failing radio as it seems here at least, it's always the same three devices that consistently have issues.
                          Real courage is not securing your Wi-Fi network.