Announcement

Collapse
No announcement yet.

HomeSeer - Grafana/InfluxDB integration for non developers using MQTT

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    HomeSeer - Grafana/InfluxDB integration for non developers using MQTT

    Hello,

    Just sharing a simple way to publish device status to InfluxDB to use it however you see fit. From a non developer for non developers.

    Installation of Grafana, Node Red, HomeSeer, MCSMqtt, InfluxDB and Mosquitto is out of the scope of this post. And I have almost NO experience with either InfluxDB and Grafana and have been using HomeSeer for less than 6 months today so, you might find a lot of ways to improve on this post. On the upside, If I could configure it, it means it is a very easy task to do.

    This 10 minutes video might be useful to explain the procedure.

    Hope you find it of any use. I used Michael McSharry's spectacular plugin but tested with the also splendid plugin named Big5.


    #2
    The neatest part of this is taking the first step with several pieces of software. The hardest part is always getting something to work to build confidence to expand usage. I have no node-red, influx, or grafana experience. This shows how easily they can be integrated.

    Comment


      #3
      Originally posted by Michael McSharry View Post
      The neatest part of this is taking the first step with several pieces of software. The hardest part is always getting something to work to build confidence to expand usage. I have no node-red, influx, or grafana experience. This shows how easily they can be integrated.
      Thanks Michael,

      I have been some days with this weird method... I'm happy to report it works flawlessly all the time.

      I even made another dashboard taking readings of the current power consumption taken from an Aeotec Z-Wave Plug and then combining in the same dashboard with some infor taken from telegraph on windows to combine power consumption with performance:

      Click image for larger version

Name:	Screen Shot 2019-08-24 at 2.08.57 PM.png
Views:	867
Size:	423.3 KB
ID:	1322580

      I'm running Mosquitto, Grafana and Influx on another Asus but this time an Asus PN40 on Ubuntu Desktop 18.04, As you can see, both servers are almost sleeping:

      Click image for larger version

Name:	Screen Shot 2019-08-23 at 11.10.39 PM.png
Views:	762
Size:	449.9 KB
ID:	1322581
      Using MQTT enables me to choose exactly what I want to publish on the database from the chosen sensor.

      So far so good... I'm getting better on Grafana by the way. I'm using two servers because I have two. None of those two are particularly expensive but I'm pretty sure all the software could be running very smoothly on the PN60 under Windows.







      Comment


        #4
        Here's my work in progress dashboard using Grafana, Telegraf and the mcsMQTT plugin to send HS3 values and status.

        Comment


          #5
          Thank you for the video Pepe! I was able to get a humidity sensor to report to influx DB and then display in Grafana thanks to it!

          I've noticed one problem though, and it's even present in your video. While Grafana will display a graph of the information, it only does so when using the "distinct" SELECT value option "Mean" "Median" "Mode" will not work.

          This is why you had to edit your query in the video at around 8:30. if you had deleted "mean" and selected "distinct" instead it would have drawn the graph for you.

          In addition, when you mouse over the graph, it should show you the numeric value of the graph next to it's alias, and this does not happen. (10:16 in the video for example.)

          Have you had any luck in fixing this? Like you, I've got no idea what I'm doing. I suspect that it's related to the configuration of mcsMQTT.

          Michael McSharry has written the absolute best documentation I've ever seen for something like this, but it's over 300 pages long and I don't yet comprehend how to use the variables and wildcards he talks about in his documentation.

          Comment


            #6
            Originally posted by TankHankerous View Post
            Thank you for the video Pepe! I was able to get a humidity sensor to report to influx DB and then display in Grafana thanks to it!

            I've noticed one problem though, and it's even present in your video. While Grafana will display a graph of the information, it only does so when using the "distinct" SELECT value option "Mean" "Median" "Mode" will not work.

            This is why you had to edit your query in the video at around 8:30. if you had deleted "mean" and selected "distinct" instead it would have drawn the graph for you.

            In addition, when you mouse over the graph, it should show you the numeric value of the graph next to it's alias, and this does not happen. (10:16 in the video for example.)

            Have you had any luck in fixing this? Like you, I've got no idea what I'm doing. I suspect that it's related to the configuration of mcsMQTT.

            Michael McSharry has written the absolute best documentation I've ever seen for something like this, but it's over 300 pages long and I don't yet comprehend how to use the variables and wildcards he talks about in his documentation.
            Thanks for all your feedback and my apologies for taking this long to answer. I think I can make a new video with a lot of improvements in my knowledge but, i’ve been traveling non stop. I think I have fixed most of my problems just by learning a little bit more about Grafana.

            Anyway, will post the video link as son as I’m able to make it

            Comment


              #7
              Originally posted by taylormia View Post
              Here's my work in progress dashboard using Grafana, Telegraf and the mcsMQTT plugin to send HS3 values and status.

              Wow... that’s fantastic!!! Thanks for sharing!!! Do you send the video snapshots using mqtt?

              Comment


                #8
                Originally posted by ppespepe View Post

                Wow... that’s fantastic!!! Thanks for sharing!!! Do you send the video snapshots using mqtt?
                Nope. I use the AJAX plugin for Grafana from here https://grafana.com/grafana/plugins/ryantxu-ajax-panel and input the Blue Iris URL of the camera.

                Comment


                  #9
                  Originally posted by ppespepe View Post

                  Thanks for all your feedback and my apologies for taking this long to answer. I think I can make a new video with a lot of improvements in my knowledge but, i’ve been traveling non stop. I think I have fixed most of my problems just by learning a little bit more about Grafana.

                  Anyway, will post the video link as son as I’m able to make it
                  That would be awesome! There is a script floating around that will export data to influxDB but It chokes and dies when it tries to send data from the ~164 devices on my network that I'm trying to log and graph. I've got an Aeotec multisensor 6 in every room of the house, one outside for exterior conditions, Light switches, Power switches monitoring amps, watts, voltage kWh, virtual thermostats... and most of it reports once a minute. I just recently migrated over from Z-wave.me and it had a beautiful plugin that let me export directly to influxDB with no middlemen. Getting Homeseer data into influx so I can work with it in Grafana is one of the last pieces I'm missing. I used to have beautiful graphs of temperature and humidity, power usage and lighting status and Homeseer just doesn't have the same ability

                  taylormia 's inclusion of security camera snapshots has made me want more though. I think my Ubiquiti cameras might be capable of doing the same thing. I will have to test and find out!

                  Comment


                    #10
                    Originally posted by TankHankerous View Post

                    That would be awesome! There is a script floating around that will export data to influxDB but It chokes and dies when it tries to send data from the ~164 devices on my network that I'm trying to log and graph. ...... Getting Homeseer data into influx so I can work with it in Grafana is one of the last pieces I'm missing. ......
                    If you want it, depending on which OS you are running HS. I'm been doing this for ~780 devices / virtual devices for some time now. Before these scripts/plugins existed.

                    I use the ASCII control interface of HS and a perl script to send all changes to telegraf to influxdb for grafana consumption. telegraf is configured to write all hs_* entries to the homeseer influxdb database. Device reference numbers are converted to location 1 location 2 device name when sending to influx.

                    I start the script via Startup.vb by triggering a HS startup event. Script only needs to be restarted if you rename an device or location.

                    My HS system has been up for only 12 days, the perl script has just under 7 minutes of cpu time on it so it's light weight on the system.

                    I'm pumping 400-800 z-wave changes per hour, not counting the timer updates which are nearly every second via my script. Ran under Windows with perl installed. Currently running on Ubuntu 16.04.6 LTS on esxi and recently on a HS-Pi4.

                    Click image for larger version

Name:	Capture.PNG
Views:	810
Size:	140.7 KB
ID:	1326272

                    Code:
                    #!/usr/bin/perl
                    
                    use IO::Socket::INET;
                    use Data::Dumper;
                    
                    # flush after every write
                    $| = 1;
                    my $debug;
                    
                    our %refStatusXref;
                    
                    our $telegraf = new IO::Socket::INET (
                            PeerHost => 'localhost',
                            PeerPort => '8094',
                            Proto => 'tcp',
                    ) or die "$0:> ERROR in influx Socket Creation : $!\n";
                    
                    
                    my $now = localtime;
                    our $socket = new IO::Socket::INET (
                            PeerHost => 'homeseer-ip-or-hostname-or-localhost',
                            PeerPort => '11000',
                            Proto => 'tcp',
                    ) or die "$0:> ERROR in homeseer Socket Creation : $!\n";
                    
                    print "$0:> Homeseer TCP Connection Success. $now\n";
                    
                    
                    $data = 'au,*****hs-id*********,**********hs-password************';
                    print $socket "$data\r\n\n";
                    
                    while (my $data = <$socket>) {
                        $now = localtime;
                        chomp $data;
                        $data =~ s/\r//g;
                        my ($action, $ref, $newValue, $oldValue) = split (/,/, $data);
                    
                        next unless ($action eq 'DC');
                    
                        if (not defined $refStatusXref{$ref}) {
                           print qq($0:> $action, $ref, $newValue, $oldValue\n) if ($debug);
                    
                           print $socket "gs,$ref\r\n\n";
                           $data = <$socket>;
                           chomp $data;
                           $data =~ s/\r//g;
                           my ($refStatus, undef, undef, $device, $floor, $room) = split(/,/, $data);
                    
                           print "$0:> getDevice: $refStatus, $floor $room $device\n" if ($debug);
                           $refStatusXref{$refStatus}->{'floor'} = $floor;
                           $refStatusXref{$refStatus}->{'room'} = $room;
                           $refStatusXref{$refStatus}->{'device'} = $device;
                        }
                    
                        if (defined $refStatusXref{$ref}) {
                           my $device = qq($refStatusXref{$ref}->{'floor'} $refStatusXref{$ref}->{'room'} $refStatusXref{$ref}->{'device'});
                           $device =~ s/\s+$//;
                           $device =~ s/^\s+//;
                           $device =~ s/\s/\\ /g;
                           print $telegraf qq(hs_device,device=$device value=$newValue\n);
                           print "\n$0:> $now: $device\t$oldValue\t-->\t$newValue\n" if ($debug);
                        }
                    
                    }
                    
                    $socket->close();
                    $telegraf->close();

                    Comment


                      #11
                      Originally posted by lveatch View Post

                      If you want it, depending on which OS you are running HS. I'm been doing this for ~780 devices / virtual devices for some time now. Before these scripts/plugins existed.

                      I use the ASCII control interface of HS and a perl script to send all changes to telegraf to influxdb for grafana consumption. telegraf is configured to write all hs_* entries to the homeseer influxdb database. Device reference numbers are converted to location 1 location 2 device name when sending to influx.

                      I start the script via Startup.vb by triggering a HS startup event. Script only needs to be restarted if you rename an device or location.

                      My HS system has been up for only 12 days, the perl script has just under 7 minutes of cpu time on it so it's light weight on the system.

                      I'm pumping 400-800 z-wave changes per hour, not counting the timer updates which are nearly every second via my script. Ran under Windows with perl installed. Currently running on Ubuntu 16.04.6 LTS on esxi and recently on a HS-Pi4.
                      That sounds like exactly what I need! I'm running HS3 on a Debian stretch box installed on an old laptop. If I can get rid of node red, that would simplify my life.

                      Comment


                        #12
                        Hi,

                        This looks very interesting, is there a version of the script for a homeseer running on a window box. I have a PI3 set up with Influxdb and Grafana.
                        Thanks

                        Liam

                        Comment


                          #13
                          Originally posted by slayer View Post
                          Hi,

                          This looks very interesting, is there a version of the script for a homeseer running on a window box. I have a PI3 set up with Influxdb and Grafana.
                          Thanks

                          Liam
                          Yes, you just need Perl installed (free) on your windows box. Strawberry Perl is a good choice. There there are a few additional Perl modules needed for the script to work. I'll PM you the details I sent to TankHankerous.

                          Comment


                            #14
                            Iveatch's Script is the bees knees. It tracks all the things without a problem.

                            Got a thread showing my results over here

                            Comment


                              #15
                              Iveatch
                              Thanks for the PM, currently working through.

                              Regards

                              Liam

                              Comment

                              Working...
                              X