Announcement

Collapse
No announcement yet.

A script to log device changes to InfluxDB for use with Grafana

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #61
    Im using the linux updated version of the script on my pc, seems to work ok except im also getting dropped metrics:
    May-13 10:06:57 Warning Dropping event callbacks due to full queue (Type: VALUE_SET Device Ref: 73) (2000 entries), system may be too busy, plugins and HSTouch may not receive all device updates
    May-13 09:59:09 L2DB-InfluxDB Error: System.Net.ProtocolViolationException: You must provide a request body if you set ContentLength>0 or SendChunked==true. Do this by calling [Begin]GetRequestStream before [Begin]GetResponse. at Microsoft.VisualBasic.CompilerServices.Symbols.Container.Inv okeMethod(Method TargetProcedure, Object[] Arguments, Boolean[] CopyBack, BindingFlags Flags) at Microsoft.VisualBasic.CompilerServices.NewLateBinding.Object LateGet(Object Instance, Type Type, String MemberName, Object[] Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack) at Microsoft.VisualBasic.CompilerServices.NewLateBinding.LateGe t(Object Instance, Type Type, String MemberName, Object[] Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack) at scriptcode3.VBWrapper.Main(Object Parms)
    May-13 09:59:09 L2DB-InfluxDB Error: System.Net.WebException: The operation has timed out at Microsoft.VisualBasic.CompilerServices.Symbols.Container.Inv okeMethod(Method TargetProcedure, Object[] Arguments, Boolean[] CopyBack, BindingFlags Flags) at Microsoft.VisualBasic.CompilerServices.NewLateBinding.Object LateGet(Object Instance, Type Type, String MemberName, Object[] Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack) at Microsoft.VisualBasic.CompilerServices.NewLateBinding.LateGe t(Object Instance, Type Type, String MemberName, Object[] Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack) at scriptcode3.VBWrapper.Main(Object Parms)
    Can you advise where exactly to place the connection limiting line of code?
    Would be great to get this script pushed into Git so we can amend properly.

    Ive used this script since it came out, its been a huge help so thanks for that! Sadly it got upset when i migrate HS to a new PC.

    Comment


      #62
      https://gist.github.com/OliPassey/03...5016fabb80f48f
      I now have it working without error, chucked it at GitHub in the hope it helps someone

      Comment


        #63
        Hi is started implementing this script, I am running HS3 pro on an mini PC (celeron quad core 4gb ram)
        And influxDB on a docker container, on a Synology NAS
        The script is running ok, but after a while I got the same errors in the log as posted in this forum.
        Probably installing influxDB on the windows machine could solve my problems.
        But influxDB on the NAS is the way to go for me.
        So I enabled UDP on influxDB port 8091
        Edit the influxdb.conf file which is located in /volume1/docker/influxdb.conf

        I added the following line :


        [[udp]]
        enabled = true
        bind-address = ":8091"
        database = "homeseer"
        retention-policy = ""
        precision = "s"
        batch-size = 100
        batch-pending = 5
        batch-timeout = "1s"
        read-buffer = 0

        no I adjusted the script to use UDP instead on TCP


        Code:
        
        
        'VB.Net script to write Homeseer 3 values to InfluxDB
        'Created by Brian based on code found on the Homeseer forums. No warranty. Use at your own risk.
        'Uncomment the log statements if you're having problems to try to track down the error.
        
        'Installation Instrustions: 
        '    0: Install InfluxDB & get it running
        '    1: Modify the variables below to fit your system.
        '    2: Add this line to Homeseer\Scripts\Startup.vb
        '    hs.RegisterStatusChangeCB("L2DB-influxdb.vb","Main")
        '    3: Restart Homeseer
        
        
        Imports System.Net
        Imports System.IO
        Imports System.Text
        Imports System.Net.Sockets 
        
        Sub Main(ByVal Parms As Object)
        
            '==========================================================
            'Modify these to fit your system
            dim INFLUX_DB_SERVER_IP = "192.168.0.6"
            dim INFLUX_DB_SERVER_PORT = 8091
            dim SKIP_LIST = "183" 'comma separated list of device references to skip logging
            '==========================================================
        
            dim device_name, device_location, device_location2, deviceObj, device_type, problem
            dim dev_address as string
            Dim device_value As Double
            Dim devRef As Integer
            Dim udpClient As New UdpClient
            Dim IP As IPAddress 
            Dim PORT As Integer 
            Dim bytCommand As Byte() = New    Byte() {}
            Dim pRet 
            'hs.WriteLog("L2DB-InfluxDB", "Script running")
        
            'Get device info from Homeseer
            dev_address = Parms(1) 'address of device. 
            device_value = Parms(2) 'new value of device
            devRef = Parms(4) 'Device reference of the device 
            deviceObj = hs.GetDeviceByRef(devRef)
            device_name = deviceObj.name(hs)
            device_location = deviceObj.location(hs)
            device_location2 = deviceObj.location2(hs)
            device_type = deviceObj.Device_Type_String(hs)
        
            If device_type = "Timer" Then 'Don't log timers
                Exit Sub
            End If
        
            If SKIP_LIST <> "" Then 
                Dim skips = SKIP_LIST.split(New Char() {","c})
                Dim skip as String
                For Each skip in Skips
                    If devref = skip
                        Exit Sub
                    End If
                Next
            End If
        
            'hs.WriteLog("L2DB-InfluxDB", "Logging to Database: Reference: " & devref & " Address: " & dev_address & ", Device Name: " & device_name & ", New Value: " & device_value)
        
        
            'Build the post data string
            Dim postdata as String = devref
            If device_name <> "" Then 
                postdata = postdata & ",name=" & device_name
            End If
            If device_location <> "" Then 
                postdata = postdata & ",location=" & device_location
            End If
            If device_location2 <> "" Then 
                postdata = postdata & ",location2=" & device_location2
            End If
            If device_type <> "" Then 'Types are optional and not all devices have them
                postdata = postdata & ",type=" & device_type
            End If
            If dev_address <> "" Then 'Addresses are optional and not all devices have them
                postdata = postdata & ",address=" & dev_address
            End If
        
            postdata = postdata.Replace(" ", "\ ")
            postdata = postdata & " value=" & device_value
        
            hs.WriteLog("L2DB-InfluxDB", postdata)
        
        
            Try
            IP = IPAddress.Parse(INFLUX_DB_SERVER_IP)
            PORT = INFLUX_DB_SERVER_PORT
            udpClient.Connect(IP, PORT)
            bytCommand = Encoding.ASCII.GetBytes(postdata)
            pRet = udpClient.Send(bytCommand, bytCommand.Length)
        
             'hs.WriteLog("L2DB-InfluxDB", "No of bytes send " & pRet)
        
            Catch ex As Exception
                hs.WriteLog("L2DB-InfluxDB", "Error: " & ex.ToString())
            End Try
        
        
        
        End Sub
        Hopefully this is running more stable for me..

        Gr.

        Comment


          #64
          Does anyone know how I could amend the script to have an option to send a metric on a time schedule rather than just when the value gets updated?
          Im monitoring a UPS, but with the battery sitting constantly at 100% im getting no graph when the time period in grafana moves on..

          Comment


            #65
            Originally posted by OliP View Post
            Does anyone know how I could amend the script to have an option to send a metric on a time schedule rather than just when the value gets updated?
            Im monitoring a UPS, but with the battery sitting constantly at 100% im getting no graph when the time period in grafana moves on..
            Rather than trigger the script by a status change you can run it recurring every minute.
            Take the line out of startup.vb ( hs.RegisterStatusChangeCB("L2DB-influxdb.vb","Main") from step 2 in the installation).

            Here's my script....different from the above script but you get the idea. Basically my script runs every minute and only send devices values from device numbers that are in the list. I didn't want a massive influx database of all my devices, 95% of which are not worth logging.

            Code:
            'Call this script by runiing &hs.runScriptFunc("L2DB-influxdbV4.vb","Main",0,False,False)
            'VB.Net script to write Homeseer 3 values to InfluxDB
            'from this post of the Homeseer forums: https://forums.homeseer.com/showthread.php?t=194145
            'Use at your own risk. Edited by Marty.
            'Uncomment (hs.WriteLog) the log statements if you're having problems to try to track down the error.
            
            
            'V4 sends to new Raspberry Pi 2 which runs only influx and grafana, note the IP address and database name
            
            
            Imports System.Core
            Imports System.Web
            Imports System.Net
            Imports System.IO
            Imports System.Text
            
            Sub Main (ByVal Parms As Integer)
            
                '==========================================================
                'Modify these to fit your system
                Dim INFLUX_DB_SERVER_IP = "xxx.xxx.xxx.xxx"
                Dim INFLUX_DB_SERVER_PORT = "xxxx"
                Dim INFLUX_DB_DATABASE_NAME = "HomeSeer"
            
                'Comma separated list of device references to log
                Dim DEVICE_LIST = {
                    1886,
                    2755,
                    2733,
                    2750,
                    2749,
                    2728,
                    2727,
                    2699,
                    2700,
                    2704,
                    2705,
                    2714,
                    2715,
                    2689,
                    2690,
                    2694,
                    2695,
                    3065,
                    2710
                    }
            
            
            
                '==========================================================
            
                Dim device_name, device_location, device_location2, deviceObj, device_type
                Dim device_value As Double
            
                'hs.WriteLog("L2DB-InfluxDB", "Script running L2DB-influxdbV4")
            
                'Build the post data string
                Dim postdata As String = ""
            
                For Each devRef As Integer In DEVICE_LIST
            
                    deviceObj = hs.GetDeviceByRef(devRef)
                    device_name = deviceObj.name(hs)
                    device_location = deviceObj.location(hs)
                    device_location2 = deviceObj.location2(hs)
                    device_type = deviceObj.Device_Type_String(hs)
                    device_value = hs.DeviceValueEx(devRef)
                    postdata = postdata & devRef
                    If device_name <> "" Then 
                        postdata = postdata & ",name=" & device_name
                    End If
                    If device_location <> "" Then 
                        postdata = postdata & ",location=" & device_location
                    End If
                    If device_location2 <> "" Then 
                        postdata = postdata & ",location2=" & device_location2
                    End If
                    If device_type <> "" Then 'Types are optional and not all devices have them
                        postdata = postdata & ",type=" & device_type
                    End If
                    postdata = postdata & "~value=" & device_value    '~ will be changed to a space later
                    postdata = postdata & Chr(10)     'insert line feed for next set of values
                Next
            
            
                'Strip invalid characters and change ~ to a space
                postdata = postdata.Replace(" ", "_")
                postdata = postdata.Replace("[", "")
                postdata = postdata.Replace("]", "")
                postdata = postdata.Replace("~", " ")
            
                'hs.WriteLog("L2DB-InfluxDB", "Postdata: " & postdata)
            
                'Set up the Webrequest
                Dim url = "http://" & INFLUX_DB_SERVER_IP & ":" & INFLUX_DB_SERVER_PORT & "/write?db=" & INFLUX_DB_DATABASE_NAME
                Dim httpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
            
                httpWebRequest.ContentType = "application/x-www-form-urlencoded"
                httpWebRequest.Method = "POST"
                ServicePointManager.DefaultConnectionLimit = 20
                Dim encoding As New System.Text.UTF8Encoding
            
                'Make the request to the database
                Try
                    Dim data As Byte() = encoding.GetBytes(postdata)
                    httpWebRequest.ContentLength = data.Length
                    Dim myStream As Stream = httpWebRequest.GetRequestStream()
            
                    If data.Length > 0 Then
                        myStream.Write(data, 0, data.Length)
                        myStream.Close()
                    End If
                Catch ex As Exception
                    hs.WriteLog("L2DB-InfluxDB", "Error1: " & ex.ToString())
                End Try
            
                Try
                    Dim httpResponse = DirectCast(HttpWebRequest.GetResponse(), HttpWebResponse) 
                    dim myReader As StreamReader 
                    dim myStream As Stream
                    myStream = httpResponse.GetResponseStream()
                    myReader = New StreamReader(myStream) 
                    Dim responseText = myReader.ReadToEnd() 
                    If responseText <> "" Then
                        hs.WriteLog("L2DB-InfluxDB", "Response: " & responseText) 
                    End If
            
                Catch ex As Exception
                    hs.WriteLog("L2DB-InfluxDB", "Error2: " & ex.ToString())
                    hs.WriteLog("L2DB-InfluxDB", "Request was: " & postdata)
                End Try
            
            
            End Sub
            iCore5 Win 10 Pro x64 SSD

            HS3 Pro Edition 3.0.0.435 Windows

            BLOccupied:,Device History:,Yamaha:,UltraMon3:,mcsXap:,Restart:,UltraNetatmo3:, UltraM1G3:,Ultra1Wire3:,BLBackup:,Harmony Hub:,DoorBird:,UltraECM3:,Nanoleaf 3P:,UltraRachio3:,Z-Wave:,SDJ-Health:,BLGarbage:,Blue-Iris:,Chromecast:,Pushover 3P:,EasyTrigger:

            Comment


              #66
              Brilliant, thank you!
              Will take a look at this tonight, think i'll look to get both running monitoring separate devices, some need logging every minute, other just at change.
              Thanks again.
              Oli

              Comment


                #67
                How exactly are you running this new script?

                I see the command &hs.runScriptFunc("L2DB-influxdb-cron.vb","Main",0,False,False) but do you put this in the startup script or run it from an event?
                I've tried both, and it seems to run without error but i see no metrics in Grafana for the IDs selected.

                Edit:
                Clearly not the startup script, that error'd out.
                Ran this from an event but get
                Date/Time Pri Type/Error Message/Source
                Jun-06 16:52:51 Warning Not running script since its already running: C:/Program Files (x86)/HomeSeer HS3/scripts/L2DB-influxdb-cron.vb Single instance option enabled in event properties
                Jun-06 16:52:51 Event Running script in background: C:/Program Files (x86)/HomeSeer HS3/scripts/L2DB-influxdb-cron.vb
                Jun-06 16:52:51 Event Event Trigger "CRON & IFTTT Log to InfluxDB"
                Jun-06 16:52:51 Event Event CRON & IFTTT Log to InfluxDB triggered by the event page 'Run' button.
                Jun-06 16:52:05 Warning VB.Net script exception(0), re-starting: Object reference not set to an instance of an object.
                Jun-06 16:52:05 Event Running script in background: C:/Program Files (x86)/HomeSeer HS3/scripts/L2DB-influxdb-cron.vb
                Jun-06 16:52:05 Event Event Trigger "CRON & IFTTT Log to InfluxDB"
                Edit 2:

                I'm now running this as an event with the execute immeadiate script command, with the line as per your instructions.
                I've also enabled logging to get some more info. The script seems to be running fine now, but no metrics are making it over to influx.
                Checked IP / Port / Db are all as before.
                Nothing additional in logs to suggest an issue.
                Jun-06 17:02:53 L2DB-InfluxDB Script running L2DB-influxdbV4
                Jun-06 17:02:53 Event Running script statement immediately: &hs.runScriptFunc("L2DB-influxdb-cron.vb","Main",0,False,False)
                Jun-06 17:02:53 Event Event Trigger "CRON & IFTTT Log to InfluxDB"

                Solved:
                These metrics have underscores instead of spaces. :facepalm:

                Thanks for a great script!

                Comment


                  #68
                  I run it as a timed event every 5 minutes (or whatever you want).

                  In Grafana you need to select the database when "Add Query" ("HomeSeer" in my case, as per the database name). Then when you click on the "select measurement" box in the FROM line you should get a drop down list of the device numbers that you have in the list of the script. Then when you click the "+" on the WHERE line you select "name" and you should see the device name when you click 'select tag value'.

                  In the SELECT line you need "field(value)" and "distinct()" and you should then get the graph.

                  See attached screenshots.

                  Click image for larger version  Name:	Screenshot 2019-06-07 11.22.35.png Views:	0 Size:	68.4 KB ID:	1309203Click image for larger version  Name:	Screenshot 2019-06-07 11.25.31.png Views:	0 Size:	194.1 KB ID:	1309204


                  Ohhh......just saw the bit where you solved it.........facepalm for me too....lol.

                  Glad you got it working.
                  iCore5 Win 10 Pro x64 SSD

                  HS3 Pro Edition 3.0.0.435 Windows

                  BLOccupied:,Device History:,Yamaha:,UltraMon3:,mcsXap:,Restart:,UltraNetatmo3:, UltraM1G3:,Ultra1Wire3:,BLBackup:,Harmony Hub:,DoorBird:,UltraECM3:,Nanoleaf 3P:,UltraRachio3:,Z-Wave:,SDJ-Health:,BLGarbage:,Blue-Iris:,Chromecast:,Pushover 3P:,EasyTrigger:

                  Comment


                    #69
                    I am back to things cutting out after about 6 or 7 hours of homeseer being up.

                    Getting this in my logs:
                    Jul-22 16:33:19 L2DB-InfluxDB Error2: System.Net.ProtocolViolationException: You must provide a request body if you set ContentLength>0 or SendChunked==true. Do this by calling [Begin]GetRequestStream before [Begin]GetResponse. at Microsoft.VisualBasic.CompilerServices.Symbols.Container.Inv okeMethod(Method TargetProcedure, Object[] Arguments, Boolean[] CopyBack, BindingFlags Flags) at Microsoft.VisualBasic.CompilerServices.NewLateBinding.Object LateGet(Object Instance, Type Type, String MemberName, Object[] Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack) at Microsoft.VisualBasic.CompilerServices.NewLateBinding.LateGe t(Object Instance, Type Type, String MemberName, Object[] Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack) at scriptcode5.VBWrapper.Main(Int32 Parms)
                    Jul-22 16:33:19 L2DB-InfluxDB Error1: System.Net.WebException: The operation has timed out at Microsoft.VisualBasic.CompilerServices.Symbols.Container.Inv okeMethod(Method TargetProcedure, Object[] Arguments, Boolean[] CopyBack, BindingFlags Flags) at Microsoft.VisualBasic.CompilerServices.NewLateBinding.Object LateGet(Object Instance, Type Type, String MemberName, Object[] Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack) at Microsoft.VisualBasic.CompilerServices.NewLateBinding.LateGe t(Object Instance, Type Type, String MemberName, Object[] Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack) at scriptcode5.VBWrapper.Main(Int32 Parms)
                    Jul-22 16:33:07 Warning Dropping event callbacks due to full queue (Type: CONFIG_CHANGE) (2000 entries), system may be too busy, plugins and HSTouch may not receive all device updates
                    Am i logging to many devices or just too often - or something else?
                    Any advice would be appreciated.

                    Im running two versions of the script, one via CRON, one via StartUp script, and have seperate the devices so each script is logging unique devices.

                    Comment


                      #70
                      What are you using to run InfluxDB and Grafana? A PC or a Pi?

                      Have you checked the integrity of the Influx database? I started getting errors once when the Influx database became corruputed. Maybe backup the database and then flush it. Can't remember off the top of my head how to do that but you can do a little bit of searching on Google for the InfluxDB commands.

                      M
                      iCore5 Win 10 Pro x64 SSD

                      HS3 Pro Edition 3.0.0.435 Windows

                      BLOccupied:,Device History:,Yamaha:,UltraMon3:,mcsXap:,Restart:,UltraNetatmo3:, UltraM1G3:,Ultra1Wire3:,BLBackup:,Harmony Hub:,DoorBird:,UltraECM3:,Nanoleaf 3P:,UltraRachio3:,Z-Wave:,SDJ-Health:,BLGarbage:,Blue-Iris:,Chromecast:,Pushover 3P:,EasyTrigger:

                      Comment


                        #71
                        Influx is running on another server and is fine, it takes metrics from multiple sources and all others continue to log.
                        This feels like I’m logging too much for .net’s 20 connection limit.
                        Reduced the cron logging to 5mins from 1m but doesn’t seem to have helped.

                        Comment


                          #72
                          Resolved by drastically reducing the amount of metrics i was logging.

                          Comment


                            #73
                            It’s a .Net script. Save is as a .vb file instead of .txt.

                            EDIT: I guess the post I replied to was deleted, or I can't find it. This was in reference to the script throwing an error about a function without an As clause.

                            Sent from my iPhone using Tapatalk
                            HS Pro 3.0 | Linux Ubuntu 16.04 x64 virtualized under Proxmox (KVM)
                            Hardware: Z-NET - W800 Serial - Digi PortServer TS/8 and TS/16 serial to Ethernet - Insteon PLM - RFXCOM - X10 Wireless
                            Plugins: HSTouch iOS and Android, RFXCOM, BlueIris, BLLock, BLDSC, BLRF, Insteon PLM (MNSandler), Device History, Ecobee, BLRing, Kodi, UltraWeatherWU3
                            Second home: Zee S2 with Z-Wave, CT101 Z-Wave Thermostat, Aeotec Z-Wave microswitches, HSM200 occupancy sensor, Ecolink Z-Wave door sensors, STI Driveway Monitor interfaced to Zee S2 GPIO pins.

                            Comment


                              #74
                              Originally posted by reidfo View Post
                              It’s a .Net script. Save is as a .vb file instead of .txt.

                              EDIT: I guess the post I replied to was deleted, or I can't find it. This was in reference to the script throwing an error about a function without an As clause.

                              Sent from my iPhone using Tapatalk
                              Hi reidfo Yes I deleted my post as I found I had some copy/paste errors when I was editing the script. I have it working now thanks.


                              Andy

                              Comment


                                #75
                                Thanks for this script. I have been using Influx to log my devices for the last 5 months now and it has worked very well.

                                Comment

                                Working...
                                X