Announcement

Collapse
No announcement yet.

A script to log device changes to InfluxDB for use with Grafana

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #16
    OK, it looks like the initial post request is timing out sometimes. Is your InfluxDB server the same machine as your Homeseer server or is it remote? Does it always throw errors when logging the same devices or does it change?

    Code:
    Mar-28 2:53:30 PM         L2DB-InfluxDB    Error: System.Net.WebException: [B]The  operation has timed[/B] out at  Microsoft.VisualBasic.CompilerServices.Symbols.Container.Inv   okeMethod(Method TargetProcedure, Object[] Arguments, Boolean[]  CopyBack, BindingFlags Flags) at  Microsoft.VisualBasic.CompilerServices.NewLateBinding.Object   LateGet(Object Instance, Type Type, String MemberName, Object[]  Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[]  CopyBack) at  Microsoft.VisualBasic.CompilerServices.NewLateBinding.LateGe  t(Object  Instance, Type Type, String MemberName, Object[] Arguments, String[]  ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack) at  scriptcode7.VBWrapper.Main(Object Parms)
    In the example you posted, it looks like the post requests are valid with no errant spaces or commas.
    Code:
    388,name=Pushover\ 3P\ Application\ Messages\ Remaining,location=Pushover,location2=System,type=Pushover\ 3P\ Device,address=P-4-Remaining value=6059
    571,name=CC-UPS-01\ Last\ Updated,location=APCUPSD,location2=Power\ Systems,type=Timestamp,address=UPS1_003 value=1522248725
    523,name=Front\ Walk\ Sensor\ Binary\ Motion,location=Front\ Walk,location2=Motion\ Sensor,type=Z-Wave\ Sensor\ Binary,address=E1D61766-011-Q52 value=255
    To answer your question, the line "postdata = postdata.Replace(" ", "\ ")" replaces the space character " " with a backslash and a space "\ " every time a space appears inside the postdata variable.

    Could the logging script be causing your other issues? I don't know. It stands to reason that it might but I don't know the intricacies of Homeseer scripting. I seem to recall running into similar issues when debugging the script on my own system.

    Comment


      #17
      InfluxDB is on separate server but what the heck I am installing it now on Homeseer server to test again. I will let you know my results but yes does look like different device error each time not always the same. The data that does make it to Influx however is inconsistant. When I see post request logged that shows no error I can not find that data in Influx or will get number of items with no metric data just blank when the script showed value of "10" was logged. Will try local install of InfluxDB and let you know the outcome.

      Thanks,
      Morgan

      Comment


        #18
        Running on Windows now. Any ideas with this error?

        Error: System.IO.FileNotFoundException: Could not find file 'C:\Program Files (x86)\HomeSeer HS3\System.Net.ConnectStream'. File name: 'C:\Program Files (x86)\HomeSeer HS3\System.Net.ConnectStream' at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean checkHost) at System.IO.StreamReader..ctor(String path) at scriptcode4.VBWrapper.Main(Object Parms)

        Comment


          #19
          Well I thought the script by reidfo was only for Linux but I grabbed the bottom portion of that and now everything is working with no errors. Just need to figure out how to make device selection work in the inverse where I manually specify the devices I want logged vs exempting the device ID. I have far few devices I want to log and exemption list is getting long. Thanks for all the help this is far less complex than implimenting MQTT or SQL for logging. Included some of my sample graphs and no going to get to work on Grafana to clean things up.

          Thanks,
          Morgan
          Attached Files
          Last edited by morgan.rinehart; March 29, 2018, 11:48 AM. Reason: Pictures

          Comment


            #20
            Originally posted by morgan.rinehart View Post
            Just need to figure out how to make device selection work in the inverse where I manually specify the devices I want logged vs exempting the device ID.
            Replace the following block of code:
            Code:
            	If SKIP_LIST <> "" Then 
            		Dim skips = SKIP_LIST.split(New Char() {","c})
            		Dim skip as String
            		For Each skip in Skips
            			If devref = skip
            				Exit Sub
            			End If
            		Next
            	End If
            With this instead:
            Code:
                    Dim allowed As Boolean = False
                    If ALLOW_LIST <> "" Then
                        Dim allows = ALLOW_LIST.Split(New Char() {","c})
                        If allows.Contains(devref) Then
                            allowed = True
                        End If
                    End If
                    If Not allowed Then Exit Sub
            And also replace
            Code:
            dim SKIP_LIST = "" 'comma separated list of device references to skip logging
            With This:
            Code:
                    Dim ALLOW_LIST = "" 'Comma separated list of device references to log
            Put your allowed device references in the ALLOW_LIST separated by commas.
            HS Pro 3.0 | Linux Ubuntu 16.04 x64 virtualized under Proxmox (KVM)
            Hardware: Z-NET - W800 Serial - Digi PortServer TS/8 and TS/16 serial to Ethernet - Insteon PLM - RFXCOM - X10 Wireless
            Plugins: HSTouch iOS and Android, RFXCOM, BlueIris, BLLock, BLDSC, BLRF, Insteon PLM (MNSandler), Device History, Ecobee, BLRing, Kodi, UltraWeatherWU3
            Second home: Zee S2 with Z-Wave, CT101 Z-Wave Thermostat, Aeotec Z-Wave microswitches, HSM200 occupancy sensor, Ecolink Z-Wave door sensors, STI Driveway Monitor interfaced to Zee S2 GPIO pins.

            Comment


              #21
              Thanks now I am 100% up and running. This has put big smile on my face to finally have this data workable format. Homeseer is great but lacks the piece I have been looking for on the data collection. Now I have nice graphs and starting tonight running analytics on my data. Thanks again Reidfo made my day!

              Comment


                #22
                GREAT! I'm glad you are up and running. I'm a bit of a data geek and have wanted logging, charting, and analytics for a long time. Was happy to find the original post that got me started. I'm happy that the script by default logs all devices since sometimes I want to look at trends I hadn't thought of monitoring before. Like I said, I'm a data geek.
                HS Pro 3.0 | Linux Ubuntu 16.04 x64 virtualized under Proxmox (KVM)
                Hardware: Z-NET - W800 Serial - Digi PortServer TS/8 and TS/16 serial to Ethernet - Insteon PLM - RFXCOM - X10 Wireless
                Plugins: HSTouch iOS and Android, RFXCOM, BlueIris, BLLock, BLDSC, BLRF, Insteon PLM (MNSandler), Device History, Ecobee, BLRing, Kodi, UltraWeatherWU3
                Second home: Zee S2 with Z-Wave, CT101 Z-Wave Thermostat, Aeotec Z-Wave microswitches, HSM200 occupancy sensor, Ecolink Z-Wave door sensors, STI Driveway Monitor interfaced to Zee S2 GPIO pins.

                Comment


                  #23
                  thank you so much for this , always wanted to use Grafana with Homeseer

                  I have trouble setting it up though , database and script done , i have zero experience with this so i don't know how to check what's wrong , database is showing in grafana so it means i did it right , but i'm not sure how to import the data or build the graph , doesn't pull anything

                  is there anyway you have a step by step guide ?

                  Comment


                    #24
                    Hi,

                    First of all, thanks for this! After trying a host of different workarounds this solution seems to be perfect for the logging i would like to do.

                    Had issues running the version that needed to specify the devices to include, switching to the normal one where you only specify to exclude works fine. Initially had some issues with errors being returned on certain devices, however this turned out to be where there were a comma in the device name in HomeSeer. After replacing this for another character this script now posts all devices correctly.
                    Last edited by torhaala; April 15, 2018, 01:30 PM. Reason: Solved

                    Comment


                      #25
                      A total noob on influx/grafana, but the charts looks so good, so I will give it a try.


                      Can this be done with the default influxdb.conf ? Or is there some "must-do" changes to the config?

                      Is "Telegraf" a must? I cant see what purpouse it does. Its the .vb script that is pushing the data right?

                      Comment


                        #26
                        I have influxdb and grafana running now.

                        Trying to get the vb.script to post data.

                        It fails with 2 (3 at first) errors.

                        First there was an error of an unused declared variable 'problem' So i just removed the declaration.

                        The next two errors are these:


                        Apr-23 15:12:34 Error Compiling script L2DB-influxdb.vb: Expected declaration
                        Apr-23 15:12:34 Error Compiling script L2DB-influxdb.vb: The namespace or type specified in System.Core does not contain any common member or does not exist. Verify that the name range or type is defined and contains at least one common member. Make sure that the imported item name does not use any aliases
                        EDIT:

                        The above errors are only there on the "iverse" version of the script (with an allow_list in stead of an skip_list)
                        So I tried the normal version now, but now it gives this error:

                        Error: System.IO.FileNotFoundException: Finner ikke filen C:\Program Files (x86)\HomeSeer HS3\System.Net.ConnectStream.
                        The same as the user morgan.rhinehart also reported in post #18

                        Comment


                          #27
                          This script will be a great addition to my Grafana!
                          Does this script need to be passed anything? I tried just creating a manual event to run the script but it seems to throws an error that doesn't give a lot to go on.

                          Apr-23 10:23:03 PM Error 1 Running script C:\Program Files (x86)\HomeSeer HS3\scripts\L2DB-influxdb.vb :Exception has been thrown by the target of an invocation.Index was outside the bounds of the array.
                          Apr-23 10:23:03 PM L2DB-InfluxDB Script running
                          Apr-23 10:23:02 PM Event Running script and waiting: C:/Program Files (x86)/HomeSeer HS3/scripts/L2DB-influxdb.vb
                          Apr-23 10:23:02 PM Event Event Trigger "Monitoring Script1"

                          Comment


                            #28
                            Originally posted by Rob2791 View Post
                            This script will be a great addition to my Grafana!
                            Does this script need to be passed anything? I tried just creating a manual event to run the script but it seems to throws an error that doesn't give a lot to go on.

                            Apr-23 10:23:03 PM Error 1 Running script C:\Program Files (x86)\HomeSeer HS3\scripts\L2DB-influxdb.vb :Exception has been thrown by the target of an invocation.Index was outside the bounds of the array.
                            Apr-23 10:23:03 PM L2DB-InfluxDB Script running
                            Apr-23 10:23:02 PM Event Running script and waiting: C:/Program Files (x86)/HomeSeer HS3/scripts/L2DB-influxdb.vb
                            Apr-23 10:23:02 PM Event Event Trigger "Monitoring Script1"
                            When you run it via the startupscript, it uses the hs.RegisterStatusChangeCB.
                            Its an callback that adds these 4 params:

                            parm(0) = house code of the device that changed status
                            parm(1) = device code of the device that changed status
                            parm(2) = the X10 status that the device changed to.
                            parm(3) = the device reference number. Can be used with GetDevice to find the DeviceClass of the specific device.

                            Untested, but it should work if you add those params manually when calling it.

                            Comment


                              #29
                              Originally posted by gjelsvik View Post
                              When you run it via the startupscript, it uses the hs.RegisterStatusChangeCB.
                              Its an callback that adds these 4 params:

                              parm(0) = house code of the device that changed status
                              parm(1) = device code of the device that changed status
                              parm(2) = the X10 status that the device changed to.
                              parm(3) = the device reference number. Can be used with GetDevice to find the DeviceClass of the specific device.

                              Untested, but it should work if you add those params manually when calling it.
                              I'm sorry, i didn't understand that "hs.RegisterStatusChangeCB" would run this script every time a device changed by just having that in the startup script. I missed that part when I looked everything over, and it was late at night and nothing was changing status in the house so i never found it being called in the logs until today, to my surprise i looked at the logs and found all the "L2DB-InfluxDB" log entries.

                              So now I'm getting a "real" error in the logs every time a device changes state, but everything seems to be logging. I got some graphs up and working. I've commented out the hs.writelog line (see below), is this something I should worry about?


                              Dim httpResponse = DirectCast(HttpWebRequest.GetResponse(), HttpWebResponse)
                              Using streamReader = New StreamReader(httpResponse.GetResponseStream().ToString)
                              Dim responseText = StreamReader.ReadToEnd()
                              hs.WriteLog("L2DB-InfluxDB", "Response: " & responseText)
                              End Using
                              Catch ex As Exception
                              'hs.WriteLog("L2DB-InfluxDB", "Error: " & ex.ToString())
                              hs.WriteLog("L2DB-InfluxDB", "Request was: " & postdata)
                              End Try


                              Here is the error:
                              Error: System.IO.FileNotFoundException: Could not find file 'C:\Program Files (x86)\HomeSeer HS3\System.Net.ConnectStream'. File name: 'C:\Program Files (x86)\HomeSeer HS3\System.Net.ConnectStream' at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean checkHost) at System.IO.StreamReader..ctor(String path) at scriptcode13.VBWrapper.Main(Object Parms)

                              Comment


                                #30
                                Originally posted by Rob2791 View Post
                                So now I'm getting a "real" error in the logs every time a device changes state, but everything seems to be logging. I got some graphs up and working. I've commented out the hs.writelog line (see below), is this something I should worry about?
                                I had that error aswell. My google-fu told me that the line:

                                Using streamReader = New StreamReader(httpResponse.GetResponseStream().ToString)
                                Is trying to cast a byte stream over to a string. (Or something like that)
                                I never figured it out. I also had a lot of trouble with the influxDB, so I found a other script that uses MySQL.

                                I have also changed the way the script is called, as the RegisterStatusChangeDB is not triggered when a device value is set to the same value that it was. (I want continous graphs, even though the value doesnt change all the time) So i am calling the scrip every 5 minutes.
                                Added a list of devicerefs that i want to log, and loop through them, putting the values into mysql.

                                Comment

                                Working...
                                X