Announcement

Collapse
No announcement yet.

How Can mcsMQTT Pull Data Into HomeSeer Energy?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    How Can mcsMQTT Pull Data Into HomeSeer Energy?

    I have a whole house energy monitor that can measure the energy being used at any given time. These values are published to a mosquito MQTT broker. How can I use something like mcsMQTT to pull these energy values into HS4's Energy DB? My goes is to be able to use the graphs in: HS4's "/energy.html". I notice that certain ZWave plug in switches that report energy use automatically get pulled into HS4's Energy DB, how can I add my own non-ZWave values?

    Thanks for any help or pointing me in the right direction.

    Jean-Marie Vaneskahian
    ---------------------------------------------------
    Jean-Marie G. Vaneskahian
    jean@vaneskahian.com
    ---------------------------------------------------

    #2
    Any MQTT device can be mapped into HS device and option exists in mcsMQTT select the device value to be also stored in the HS Energy database. Take a look in mcsMQTT user manual and search for keyword such as energy and it will show the setting to select it on the MQTT Page, Edit Tab. I do not know how familiar you are with mcsMQTT so I do not know the level of guidance you need at this point. Ask again if you need more help with it.

    http://mcsSprinklers.com/mcsMQTT.pdf.

    Comment


      #3
      That is awesome... I used the HS3 version of mcsMQTT to just make HomeSeer data available to the mosquito MQTT borker on a raspberry Pi and then used NodeRed, InfluxDB, and Grafana to graph the data. But I then learned that HS4 has some built in graphing capability with the Energy DB. So I thought I was on my own to pull in the data that I have on energy use externally into HomeSeer HS4 and integrate it in the Energy DB. I had no idea you had already done all that work too!

      So then what is the best setup here for graphing things and storing them in a time series database? What are the best options and approach here?
      1. I can continue to run on the Raspberry Pi: Mosquito MQTT, Node Red, InfluxDB, and Grafana
      2. I can publish the Energy Values from my main breaker panel to the Mosquito MQTT broker on the Raspberry Pi
      3. I can then use mcsMQTT on HS4 to subscribe the Energy values from the panel that are published in the Mosquito MQTT broker
      4. You are now telling me that the same mcsMQTT will also store those subscribed Energy values in the HomeSeer Energy DB, correct?
      My other goals are to graph things like Temperature sensors that exist in HomeSeer. But to do that I need to export those to the Raspberry Pi via MQTT so I can get them into Node Red and InfluxDB so Grafana can plot them. Is there a better way to do this?

      I have various piece parts but do not know the best approach to integrating it. I know I can make it work, but what is the most efficient / elegant? If HS4 had a good native graphing capability the mess with Grafana would not be needed.

      Thanks for all of your help Michael!!!
      ---------------------------------------------------
      Jean-Marie G. Vaneskahian
      jean@vaneskahian.com
      ---------------------------------------------------

      Comment


        #4
        I have read posts where the HS Energy database operations are flaky. When I was implementing it the documentation and the behavior were not the same so I reverse-engineered to figure out how to get the desired watt/KW vs. KWH data stored in the database. It was also a chore to figure out what all was needed to be able to just store directly into the database. I do not use it myself so my real experience is just implementing the interface with mcsMQTT that was requested by other user(s).

        Since you want to expand beyond just energy for history retention and display then I would suggest using something that has a generalized device to database solution. Your easiest way to get data into a database is with mcsMQTT. It stores any and all HS DeviceValue data directly into InfluxDB,InfluxDB2, MS SQL Server, MySQL and SQLite databases with a simple selection of data to be stored. If your only use of Node-Red is to get data into InfluxDB, then it can be bypassed as mcsMQTT stores directly.

        For display there are various options with the tradeoffs typically being how elegant to you want the chart to look vs. how easy you want to pick and show the data. The internal mcsMQTT charting options are designed to minimize the effort to show data. It produces line charts of fixed size with data on one or two Y axis. No user customization beyond selecting which devices on each Y axis and duration on X axis. It shows the trend lines and the current value. The chart below is from data in InfluxDB.

        Click image for larger version  Name:	0x.png Views:	0 Size:	53.0 KB ID:	1563198

        Grafana is a charting package so has much customization and can produce attractive charts. You know about this one. It is not as interactive and dynamic as mcsMQTT charting but will yield very nice results. It requires that one do their setup outside of the HS environment, but once setup the resultant chart can be produced from HS integration.

        Jon00 has a charting package that I have never used. There is also a Device History plugin that does both storage and display. I believe Device History uses SQLite database and has a more flexibility in chart formatting, but not to the degree of Grafana. I have done other charting solutions in earlier versions of HS and have migrated to ease-of-use as being what works best for me. There are likely other ones available that I have overlooked. I know a user that imports data into Excel and makes charts from it programmatically. Another does Google charts via cloud.

        Comment


          #5
          If you want to have the best graphing for any value you want to store beyond energy, grafana and an external database is your best option. I am using mcsmqtt to push many values to an external mysql database, and then using grafana to make it look pretty. I found grafana to have a steep learning curve. I was able to get some basic line graphs but have not had time to go back and really use grafana to its potential. I am pushing temp values on various sensors, energy stats and thermostat values to eventually graph out run times. Previously was using jon00s database charting utility and that worked well and was pretty easy to get going for basic all around device value graphs.

          Comment


            #6
            Michael McSharry, thanks so much for the advice. You have hit the nail on the head for lots of this stuff.

            The ONLY reason I was using Node Red was to parse a little data (HomeSeer is much better at this) and then feed it to InfluxDB for Grafana to use! But if mcsMQTT can now write HomeSeer device values directly into InfluxDB, then there is zero need for Node Red at all! This is great. It does open some questions then based on the mcsMQTT documentation:
            1. I would like to since it will remove extra network communications, but should I run Mosquito MQTT on my Windows 10 dedicated HomeSeer computer?
              1. What is the downside of running Mosquito MQTT on Windows (Same PC as HomeSeer) vs running it on Linux on a separate independent computer on the network? (My Windows 10 HomeSeer PC has tons of available resources and is up and running 24x7)
            2. I would like to since it will remove extra network communications, but should I run InfluxDB 2 on my Windows 10 dedicated HomeSeer computer?
              1. What is the downside of running InfluxDB 2 on Windows (Same PC as HomeSeer) vs running it on Linux on a separate independent computer on the network? (My Windows 10 HomeSeer PC has tons of available resources and is up and running 24x7)
            3. I see that I can even run Grafana on Windows too.

            Do you run any of these on Windows? What is your experience? It looks like with mcsMQTT, much of my graphing needs might be met and I might not even need the HS4 Energy component. And if I still have a need to Grafana, it is still an option on top of the mcsMQTT charting.

            My goal now based on your data is to hopefully run almost everything on the same very robust Windows PC that HS4 runs on, and is imaged / backed-up nightly, and move away from using the Raspberry Pi at all for this. This would keep all the communications off the network and in principle be faster. But what is the down side? Are these packages stable on Widows 10?

            What are your thoughts and experience?



            ---------------------------------------------------
            Jean-Marie G. Vaneskahian
            jean@vaneskahian.com
            ---------------------------------------------------

            Comment


              #7
              1. What is the downside of running Mosquito MQTT on Windows (Same PC as HomeSeer) vs running it on Linux on a separate independent computer on the network? (My Windows 10 HomeSeer PC has tons of available resources and is up and running 24x7)
              Functionally there is absolutely no difference. The only consideration in my mind is availability. In a distributed architecture then you want MQTT Broker to always be running so individual nodes in the architecture do not loose communications. In an architecture that depends upon HS being the core of everything then the computer that runs HS can be the same one that runs Mosquitto without implications. Mosquitto has very little overhead and is very mature.

              1. What is the downside of running InfluxDB 2 on Windows (Same PC as HomeSeer) vs running it on Linux on a separate independent computer on the network? (My Windows 10 HomeSeer PC has tons of available resources and is up and running 24x7)
              The difference here is the provisions you make for disk failure where your InfluxDB data could be lost in failure. Whatever is easier for you develop a backup solution is the primary criteria. If you do run on Linux, then dont use SD for the data storage. Use SSD. SD is designed for occasional write such as pictures from a camera. It is not designed for constant beating and it lifetime write cycles will be exhausted.

              My goal now based on your data is to hopefully run almost everything on the same very robust Windows PC that HS4 runs on, and is imaged / backed-up nightly, and move away from using the Raspberry Pi at all for this. This would keep all the communications off the network and in principle be faster. But what is the down side? Are these packages stable on Widows 10?

              What are your thoughts and experience?
              I have a hybrid environment with many distributed data collection and interface microcontrollers. HS has not been a big player in this architecture. What I did not like was HS running on a RPi-class computer due to the responsiveness of the user interface. It worked, but not ideal. This past year I obtained a Beelink GR9 to run HS and Plex under windows. An excellent decision. I think your direction of putting major software packages on a central Windows computer is good for the reasons you stated. I have no experience to base robustness of InfluxDB or Grafana, but I would not be concerned because they are well supported products. Often in the Linux world you get cutting edge goodies, but if you are late to the game they could be obsolete with no support.

              Comment


                #8
                Awesome! I will work at getting those setup on the Windows 10 HomeSeer PC then. HomeSeer is the cornerstone of almost ALL my automation. As a consequence it was the source of all the MQTT data (except for one item). So the ability to consolidate everything in Windows 10 sounds more elegant to me since a distributed solution where HS4 was down, would not really buy me anything.

                The HS4 computer runs Windows 10. Has the Boot, OS, and HS4 are on RAID 1 mirrored 1TB NVMe. This should be plenty of disk space even with Influx DB 2. I agree about the SD card on the RPi, but again it was not doing anything critcal, but moving it to windows will be much better. The backup on the HS4 Windows 10 PC is done through an nightly FULL backup job on Acronis that takes a complete snapshot of the entire PC and backs it up to a NAS with RAID 6. This has in the past allowed me to do a bare metal restore. The 12th Gen i7 processor and 64GB of RAM are overkill for the HS4 PC, but it lets me not worry about resources being an issue (plus now I can move more things that make sense to the HS4 PC).

                Thanks again for everything that you do and the help! I will keep you posted on my progress or any issues I come acorss.

                I installed the HS4 mcsMQTT plug-in directly from the ADD plugin menu and the HS store. It installed just fine.

                I would very much like to make a donation for all your work and help! How can I do that for you?

                Jean-Marie Vaneskahian
                ---------------------------------------------------
                Jean-Marie G. Vaneskahian
                jean@vaneskahian.com
                ---------------------------------------------------

                Comment


                  #9
                  I would very much like to make a donation for all your work and help! How can I do that for you?
                  You already have. Knowledge that I am able to help others is all I want.

                  Comment


                    #10
                    Michael McSharry, I wanted to let you know, I now have Mosquito MQTT setup and fully working and running on my HomeSeer HS4 computer running Windows 10.

                    The installation was easy and it even runs as a Windows Service. The only caveat what might be needed for others is that once you install it you must edit the ".conf" file that the service uses to allow for external connections with the following lines at the top of the ".conf" file:

                    Path to the file "C:\Program Files\mosquitto\mosquitto.conf"

                    Code:
                    listener 1883
                    allow_anonymous true
                    mcsMQTT immediately saw that the broker was local and pointed back to the loopback address in the "MQTT Broker Operations" area.

                    I wanted to ask some basic questions to see how best to accomplish what I want to do:
                    1. Do you use InfluxDB or InfluxDB2? Is there a benefit of one over the other?
                    2. I want to have some HomeSeer temperature sensors publish to the local Mosquito MQTT server with a topic like "Jarvis/Temperature Sensors/Kitchen" and "Jarvis/Temperature Sensors/Bedroom" etc. What is the best way to publish a device's value any time that device value gets set, even if the value did not change?
                      1. I want these temperature values in the device to be pushed to InfluxDB each time the device value is set, even if the value is the same as the previous value.
                    3. I will have an external device publishing the whole house energy usage to the Mosquito MQTT now running on the HomeSeer HS4 computer to a topic like "Jarvis/Energy/Whole House/Phase A" and "Jarvis/Energy/Whole House/Phase B"
                      1. I will need to combine these for the total Energy use
                      2. Can I have mcsMQTT pull those values and save them to the InfluxDB, just like it does the temperature sensors?

                    Thanks for any help here or pointing me in the right direction! I am making slow but steady progress
                    ---------------------------------------------------
                    Jean-Marie G. Vaneskahian
                    jean@vaneskahian.com
                    ---------------------------------------------------

                    Comment


                      #11
                      Michael McSharry, I created a video of where I am at and was hoping for some guidance. I have made some progress but am stuck on a few things.

                      I have done the following and have these issues:
                      1. Installed Mosquito MQTT as a Windows Service on my HS4 computer. It is tested and working.
                      2. Installed InfluxDB 2 as a Windows Service on my HS4 computer. It brings up the InfluxDB 2 GUI, but I do not know if mcsMQTT is using it.
                      3. I do not know how to get a single HomeSeer device (temp sensor) into the MQTT Broker with a specific Topic name.
                      Here is a video of what I have done and what I am running into:

                      ---------------------------------------------------
                      Jean-Marie G. Vaneskahian
                      jean@vaneskahian.com
                      ---------------------------------------------------

                      Comment


                        #12
                        mcsMQTT immediately saw that the broker was local and pointed back to the loopback address in the "MQTT Broker Operations" area.
                        You need to specify the actual network IP in mcsMQTT General tab for Broker IP. The loopback network 127.0.0.1 means you are running the MQTT Broker that is internal to mcsMQTT. Mosquitto is a more mature broker, and not that you have it installed it is best to use it.

                        As a note, Mosquitto version 1 does not need the edits in the conf file to provide the desired network passage. Mosquitto version 2 supports MQTT standard 5 and earlier while version 1 supports standard 3 and earlier. Either will work in the mcsMQTT environment.

                        Do you use InfluxDB or InfluxDB2? Is there a benefit of one over the other?
                        I use InfluxDB2. Which is better is like asking if HS3 or HS4 is better. InfluxDB2 will have long term support.

                        I want to have some HomeSeer temperature sensors publish to the local Mosquito MQTT server with a topic like "Jarvis/Temperature Sensors/Kitchen" and "Jarvis/Temperature Sensors/Bedroom" etc. What is the best way to publish a device's value any time that device value gets set, even if the value did not change?
                        mcsMQTT storage to InfluxDB is triggered by the HS Event callback. The callback being generated is based upon the MISC setting shown below. When checked only changes of value will generate the callback and store to InfluxDB so you will want it unchecked. If it is a plugin device (created by mcsMQTT) then use the mcsMQTT Edit tab to uncheck this MISC setting.

                        Click image for larger version

Name:	0x.png
Views:	241
Size:	15.9 KB
ID:	1563376

                        1. I will have an external device publishing the whole house energy usage to the Mosquito MQTT now running on the HomeSeer HS4 computer to a topic like "Jarvis/Energy/Whole House/Phase A" and "Jarvis/Energy/Whole House/Phase B"
                          1. I will need to combine these for the total Energy use
                          2. Can I have mcsMQTT pull those values and save them to the InfluxDB, just like it does the temperature sensors?
                        mcsMQTT only writes to InfluxDB based upon the HS Event Trigger. It does store in the short-term database SQLite any/selected MQTT traffic, but you want long term storage so you will need to associate Jarvis/Energy/Whole House/Phase A and Jarvis/Energy/Whole House/Phase B to HS devices. Each time each is received they will then be routed to InfluxDB if selected to go to long term storage.

                        mcsMQTT supports expressions so it you want just want total energy you can sum them in the expression before storing into HS device. I will need more info on exactly how these two are delivered. Are they in a JSON payload so both received at the same time or are they in independent topics and then what is the relationship/sequence of the topics?

                        Comment


                          #13
                          It looks like we both posted updates at the same time
                          ---------------------------------------------------
                          Jean-Marie G. Vaneskahian
                          jean@vaneskahian.com
                          ---------------------------------------------------

                          Comment


                            #14
                            So here is more progress but also a problem with the Attic Temperature sensor that I cannot figure out. First, I did what you said I changed to the IP address to the IP of the host itself and not the loopback. I stopped mcsMQTT and made sure Mosquito MQTT service was running. I even connected to the HS4 computer with my MQTT Explorer with mcsMQTT stopped and it worked fine telling me that the broker responding was in fact the Mosquito MQTT service on HS4. I then started mcsMQTT and made sure it was pointing to the LAN IP address of the HS4 computer it was running on. All looked good.

                            Here are screenshots of the setup.

                            Click image for larger version

Name:	Screenshot_2022-08-25_21-55-46.jpg
Views:	231
Size:	47.6 KB
ID:	1563382

                            As you can see from this screenshot I have 2 devices that I am trying to publish. Ref 844 and 847. But 847 has a last date of 1900-01-01??? Yet when we look at the device, it is fine.

                            Click image for larger version

Name:	Screenshot_2022-08-25_21-54-24.jpg
Views:	188
Size:	9.1 KB
ID:	1563383

                            Click image for larger version

Name:	Screenshot_2022-08-25_21-54-36.jpg
Views:	185
Size:	44.7 KB
ID:	1563384

                            It is also set to disable Last Change Time Updates on Status Change Only (same as the other device 844 that works fine)

                            Click image for larger version

Name:	Screenshot_2022-08-25_21-57-06.jpg
Views:	188
Size:	23.9 KB
ID:	1563385
                            Click image for larger version

Name:	Screenshot_2022-08-25_21-57-17.jpg
Views:	188
Size:	51.1 KB
ID:	1563386

                            As you can see in my MQTT Explorer, only the 844 device (not the Attic one) get published.. Why? Also why do the payload not display on the mcsMQTT screen?

                            Why can I not add the short term and long term checkboxes to the devices in mcsMQTT?
                            ---------------------------------------------------
                            Jean-Marie G. Vaneskahian
                            jean@vaneskahian.com
                            ---------------------------------------------------

                            Comment


                              #15
                              You also asked about how the Current Cost Envi publishes its Energy use data to the Mosquito Broker now on the HS4 server. I have a Raspberry Pi that has an RTL_433 SDR (Software Defined Radio) that runs some code that pulls in the energy use of the Current Cost Envi meter that is setup at my breaker panel. It comes into the Raspberry Pi and it in turn publishes the current Wattage one each Phase of the power line. They update every minute.

                              Click image for larger version

Name:	Screenshot_2022-08-25_23-15-04.jpg
Views:	228
Size:	48.9 KB
ID:	1563394

                              Click image for larger version

Name:	Screenshot_2022-08-25_23-13-28.jpg
Views:	184
Size:	66.2 KB
ID:	1563395

                              Click image for larger version

Name:	Screenshot_2022-08-25_23-13-35.jpg
Views:	186
Size:	61.6 KB
ID:	1563396

                              The Topics where these are published are:

                              Code:
                              rtl_433/rpi4-iot/devices/CurrentCost-TX/1643/power0_W
                              rtl_433/rpi4-iot/devices/CurrentCost-TX/1643/power1_W
                              What is the best way to:
                              1. Pull these into HomeSeer HS4 as devices that get updated by mcsMQTT?
                              2. Add these 2 Values together and Publish that as something like "Total House Wattage"
                              3. Write the "Total House Wattage" to InfluxDB
                              ---------------------------------------------------
                              Jean-Marie G. Vaneskahian
                              jean@vaneskahian.com
                              ---------------------------------------------------

                              Comment

                              Working...
                              X