Announcement

Collapse
No announcement yet.

How Can mcsMQTT Pull Data Into HomeSeer Energy?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #16
    Well, I have made even more progress. Since I got everything moved off the Raspberry Pi to the HS4 Windows server except my RTL_433 SDR decoder (it decodes radio signals from IoT devices and allows me to publish the values to an MQTT Broker), I worked last night on getting to moved to the HS4 Windows server as well. After some driver issues and looking for a windows binary of the RTL_433 package, I got it working as a Windows Service!

    I now have all the following running on the HS4 Windows server as Windows Services:
    • Mosquito MQTT Broker
    • InfluxDB 2
    • Grafana
    • RTL_433
    Click image for larger version

Name:	Screenshot_2022-08-26_09-43-42.jpg
Views:	177
Size:	184.5 KB
ID:	1563435

    With this and mcsMQTT running in HS4, I think I should have everything I need to pull in the data, publish it to MQTT, save it to InfluxDB and graph it on Grafana. Now I need to "wire it all together" via software and settings which is where I am having problems.

    My biggest issue is the lack of understanding on how to configure mcsMQTT to:
    1. Take an existing specific HS4 device that has a value I want to save those values to InfluxDB.
    2. I realize now I don't actually care about MQTT itself. In the past it was just the vehicle to get data into InfluxDB so it could be graphed.
    3. I also need to be able to read those two Whole House Wattage values from the MQTT Topic and write those to InfluxDB.
    4. Create a chart of some of the data with mcsMQTT (I will also do the same with Grafana, but I know how to do that one)
    I am loving that I have everything now under Windows on a single, high availability system with lots of resources.

    I really appreciate your help with this and feel like I am in the home stretch!

    Jean-Marie Vaneskahian
    ---------------------------------------------------
    Jean-Marie G. Vaneskahian
    jean@vaneskahian.com
    ---------------------------------------------------

    Comment


      #17
      What you need to do is on the association tab of mcsmqtt, change the filter to include non plugin hs devices and then on the device select the "h" checkbox to the right of that device to write it to the database

      Comment


        #18
        Dweber85rc ,

        Originally posted by Dweber85rc View Post
        What you need to do is on the association tab of mcsmqtt, change the filter to include non plugin hs devices and then on the device select the "h" checkbox to the right of that device to write it to the database
        Click image for larger version

Name:	Screenshot_2022-08-26_11-06-36.jpg
Views:	90
Size:	11.1 KB
ID:	1563448
        Click image for larger version

Name:	Screenshot_2022-08-26_11-06-53.jpg
Views:	89
Size:	37.9 KB
ID:	1563449

        I did that, but while I can check the "h" value, I cannot set the short term and long term database logging. Why is this?
        ---------------------------------------------------
        Jean-Marie G. Vaneskahian
        jean@vaneskahian.com
        ---------------------------------------------------

        Comment


          #19
          844 and 847 are likely listed in HS as Devices. In HS4 Devices are only containers and only Features can generate HS Event callbacks so no ability for plugin to get an event trigger from a Device so checkboxes are not shown. I should not use the same dependency in HS3 and need to update the HS3 plugin to allow the S and L checkboxes.

          The basic strategy is to associate both topics a HS device. Let us say HS devices are 100 for power0 and 101 for power1. Click the 101 ref button on Association tab to bring up Edit popup. Select a Rate Device to be created with a weighting factor of 1.0. This will create device 102. 100 and 102 will now get updated with the individual power values. In the Expression text box on Edit tab for the 101 device enter the line below to sum the payload of power1 with the power0. It is possible that inline expression designation is needed and if so then double < and > should be placed around the expression. I did not try before posting this.
          Code:
          $$PAYLOAD:+$$PAYLOAD:(rtl_433/rpi4-iot/devices/CurrentCost-TX/1643/power0_W):
          Select the L checkbox on power0 topic on the Association tab so it goes into InfluxDB. Edit the Device names in HS so they make sense to you. If the power0 is delivered after power1 then swap the rate device creation to 100 rather than 101 and user power1 in the expression.

          Comment


            #20
            Michael McSharry, so for the sensors that I have that are "Root" devices and not features in HS4, what are my options for getting mcsMQTT to go their values to InfluxDB? Should I create new devices with a feature that gets its value from these "Root" devices so that mcsMQTT will put those into InfluxDB? If I use this approach, what is the best way to update the values on these newly created device features (an event that checks the device time of the old root and then does an update?)

            Also, I will work on the second issue with adding the Power0 and Power1.
            ---------------------------------------------------
            Jean-Marie G. Vaneskahian
            jean@vaneskahian.com
            ---------------------------------------------------

            Comment


              #21
              Here is a video of the problem I am running into with the instructions on adding the two power0 and power1 into a new 3rd device that you gave me.



              Click image for larger version

Name:	Screenshot_2022-08-26_15-39-58.jpg
Views:	98
Size:	38.3 KB
ID:	1563525

              Click image for larger version

Name:	Screenshot_2022-08-26_15-40-34.jpg
Views:	93
Size:	52.4 KB
ID:	1563526

              Any ideas why it will not create the new device and stay checked?
              ---------------------------------------------------
              Jean-Marie G. Vaneskahian
              jean@vaneskahian.com
              ---------------------------------------------------

              Comment


                #22
                First problem is I assumed, incorrectly, that it was HS3 plugin you are using. From the video and response, it is clear that it is HS4. How were you able to run HS4 with 844 and 847 as root devices? I thought HS4 would not allow such things. I can remove the constraint and always show the S and I checkboxes, but not clear to me if HS4 will generate a HS Event trigger for a root device.

                The problem with the rate device results form the difference in HS3 and HS4 where the first returns "checked" and the second returns "true" when a checkbox is checked. I missed this on the rate device so it still had the HS3 expectation in the HS4 plugin. It is fixed in 5.26.2.1.

                You can change the room, floor and name on either the Edit tab or on the HS Devices page. The plugin does not depend upon these for any logic. You can also regroup features under a device and create new devices for this purpose on the Edit tab Grouping row. It is not necessary, but just making you aware of it.

                The attached 5.26.2.1 is unzipped into the \bin\mcsMQTT folder.

                Attached Files

                Comment


                  #23
                  Yes, I apologize if I was not clear on what I was running. I am HS4 (and the HS4 version of the Plugin). I am running it on Windows 10.

                  I installed the binaries you sent me and they worked great. The checkbox stayed and the new device was created. The problem is the value.

                  Just so we are on the same page.
                  1. The value of the new device should simply be the total Wattage of Power0 + Power1 right?
                  2. If Power0 = 2000 and Power1 = 1000 then the new device should have a value of 3000
                  3. This Rate concept asks what time interval I want? (Second / Minute / Hour), How does this apply?
                  4. I get crazy numbers on the new device (like negative number)

                  Click image for larger version

Name:	Screenshot_2022-08-26_17-52-11.jpg
Views:	96
Size:	25.2 KB
ID:	1563545
                  Click image for larger version

Name:	Screenshot_2022-08-26_17-50-33.jpg
Views:	87
Size:	38.2 KB
ID:	1563546
                  ---------------------------------------------------
                  Jean-Marie G. Vaneskahian
                  jean@vaneskahian.com
                  ---------------------------------------------------

                  Comment


                    #24
                    Still like to understand
                    How were you able to run HS4 with 844 and 847 as root devices? I thought HS4 would not allow such things. I can remove the constraint and always show the S and I checkboxes, but not clear to me if HS4 will generate a HS Event trigger for a root device.
                    Looking at the code I see that it was not as simple to use the rate device for this purpose. I updated in 5.26.3.0 (attached) to add a rate option of None which will not consider the time interval and then the function becomes essentially whatever you want to put in the Expression textbox. If you leave this textbox blank then the HS device will be updated to the same as the device from which the rate device was derived with the low pass filtering still being applied.

                    The most straightforward process and the one I tested is similar, but not exactly the process I first described.
                    1. Associate the power0 and power1 to get two HS Features with Ref.
                    2. On the one that come later in the MQTT message delivery, use the Edit tab to create a rate device, sensitivity filter 1.0, None for the rate interval.
                    3. A new HS feature will be created and if the Association table is refreshed it will be visible as a new row in the table.
                    4. Use Edit tab on this rate device row to enter the expression in the Expression textbox. Other settings can be the default. The expression is easiest to understand if both power0 and power1 are explicitly specified, but the original expression should work too.

                    Below is my test case where Temperature 1 (Ref 9) and 2 (Ref 7) are summed in the rate device (Ref 8) . The long term storage can be selected for any of them on the Association table.

                    Code:
                    $$PAYLOAD:(DS18B20:DS18B20-1:Temperature):+$$PAYLOAD:(DS18B20:DS18B20-2:Temperature):
                    Click image for larger version

Name:	0x.png
Views:	100
Size:	50.8 KB
ID:	1563559

                    Click image for larger version

Name:	1.png
Views:	82
Size:	46.3 KB
ID:	1563560

                    Click image for larger version

Name:	3.png
Views:	83
Size:	92.3 KB
ID:	1563562

                    Click image for larger version

Name:	2.png
Views:	85
Size:	22.5 KB
ID:	1563561

                    Comment


                      #25
                      Thanks so much! It makes perfect sense in terms of how the new device is created, but unfortunately I do not see (5.26.3.0) in your previous posting anywhere. I think you might have forgotten to attach it.

                      Also, while I was waiting to implement the sum of the two power devices (Power0 and Power1), I went ahead and wrote a script to just add their two values and put the result in third device. I then created an event that triggers when Power0 is updated or set and runs the script 2 seconds later. That way I do not have to worry about which updates first (Power0 or Power1). It is not as clean as your proposed solution but it seems to work.

                      Here is the new device I created whose only purpose is to store the data of the sum of Power0 and Power1

                      Click image for larger version  Name:	Screenshot_2022-08-26_18-30-51.jpg Views:	0 Size:	6.9 KB ID:	1563572

                      Here is the script I wrote that adds the two values for Power0 and Power1 and then writes the resulting sum in that new device (Ref# 1420).

                      Click image for larger version  Name:	Screenshot_2022-08-26_18-30-31.jpg Views:	0 Size:	23.5 KB ID:	1563573

                      Code:
                      public enum Jarvis_Variables
                      {
                          Energy_Whole_House_Phase_A = 1413,
                          Energy_Whole_House_Phase_B = 1414,
                          Energy_Whole_House = 1421
                      }
                      
                      public void Update_Energy_Whole_House(object Parms)
                      {
                          double Energy_Whole_House_Phase_A = (double)hs4.GetPropertyByRef((int)Jarvis_Variables.Energy_Wh ole_House_Phase_A, HomeSeer.PluginSdk.Devices.EProperty.Value);
                          double Energy_Whole_House_Phase_B = (double)hs4.GetPropertyByRef((int)Jarvis_Variables.Energy_Wh ole_House_Phase_B, HomeSeer.PluginSdk.Devices.EProperty.Value);
                          double Energy_Whole_House = Energy_Whole_House_Phase_A + Energy_Whole_House_Phase_B;
                          hs4.UpdateFeatureValueByRef((int)Jarvis_Variables.Energy_Who le_House, Energy_Whole_House);
                          hs.SetDeviceLastChange((int)Jarvis_Variables.Energy_Whole_Ho use, DateTime.Now);
                      }
                      Here is the event trigger that waits for Power0 to have a value set or changed and then waits 2 seconds and fires off the script.

                      Click image for larger version  Name:	Screenshot_2022-08-26_18-31-24.jpg Views:	0 Size:	23.5 KB ID:	1563574

                      Here is the result. You can see that the new device with the sum of Power0 and Power1 (called "Whole House Energy", Ref# 1420) is updated 2 seconds after Power0 and Power1.

                      Click image for larger version  Name:	Screenshot_2022-08-27_07-10-24.jpg Views:	0 Size:	20.0 KB ID:	1563575

                      I then went into mcsMQTT and added this new device, "Whole House Energy" (Ref# 1420) to associations and and checked the h, s, l checkboxes. I then checked my InfluxDB and sure enough, it was there along with Power0 and Power1!

                      Click image for larger version  Name:	Screenshot_2022-08-27_07-18-05.jpg Views:	0 Size:	54.5 KB ID:	1563576

                      I still would very much like to implement the solution you brought up as soon as I get the updated plugin with the rate option of none.

                      As to your second point on:

                      How were you able to run HS4 with 844 and 847 as root devices?
                      These devices were created by a HS3 Legacy Plugin called "RFXCOM". It pulls in wireless sensors (in this case Oregon Scientific Temperature and Humidity Sensors) and creates a device with features. The plugin unfortunately creates the Temperature Sensor portion as the "Root" parent device and the Humidity and Battery levels as child "features".

                      Here is a screenshot of the devices:

                      Click image for larger version  Name:	Screenshot_2022-08-27_07-26-01.jpg Views:	0 Size:	30.0 KB ID:	1563577

                      As you can see Ref# 847 and Ref#844 are Parent / Root Devices, but they have the temperature data that I am interested in getting into InfluxDB via mcsMQTT.

                      I can 100% confirm that an event created that fires off when the parent / root is set or changes value does work. From this perspective, there is nothing different about the Parent / root device vs. the feature / children. Values can can be set and changed and events can respond to those values being set and changed. All I want to be able to do is log the values of these parent / root temperature sensors into InfluxDB via mcsMQTT.

                      Does that answer what you were looking for?

                      Is there anything else I can do?

                      I really appreciate your help and updates.
                      ---------------------------------------------------
                      Jean-Marie G. Vaneskahian
                      jean@vaneskahian.com
                      ---------------------------------------------------

                      Comment


                        #26
                        In the attached I removed the check for device vs. feature for the L and S checkboxes. I believe the 5.26.3.0 attachment is at the bottom of post 24, but 5.26.3.1 is the one you want to include 844 and 847.

                        Using the event and script of course works so it a matter of preference. Doing manipulation of raw data is something that is a commonly needed and that is why I have included the two derived devices to mcsMQTT on the Edit tab. I did not realize until now that this simple addition of multiple payload items could not be accomplished without the None option for the Rate device.
                        Attached Files

                        Comment


                          #27
                          I have installed the new version of the plugin and things most everything works!!! Thank you so much!!! I do have a small issue but it is no big deal.

                          I got the new Rate device based on Power0 set to "None" and then on the new device I created an evaluation of:

                          Code:
                          $$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power0):+$$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power1):
                          That populated the new rate device with the value old value of Power1 + the new value of Power0

                          I then tried:

                          Code:
                          $$PAYLOAD:+$$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power1):
                          Same Problem

                          I then did the same thing but this time created the rate device based on Power1 and the evaluations of the new rate device were:

                          Code:
                          $$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power0):+$$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power1):
                          Now the problem was

                          That populated the new rate device with the value old value of Power0 + the new value of Power1

                          also tried

                          Code:
                          $$PAYLOAD:+$$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power0):
                          Same problem.

                          This appears to be a timing problem as both Power0 and Power1 get published to the MQTT broker seemingly simultaneously.

                          The way the script I wrote gets around this problem is by waiting 2 seconds after Power0 gets updated before adding Power0 and Power1 so that both values are current.

                          mcsMQTT is very fast and is able to grab the value and update before both have populated. Not sure...

                          Maybe if there was some kind of delay.. But honestly it is working just fine with the script.

                          I cannot thank you enough!

                          A quick question:
                          1. I see that I can publish a HomeSeer device to the MQTT broker with any topic name I want, and that is great! However, it writes the HomeSeer Device name to the InfluxDB. Is there a way to give the Device a custom name for the DB entry (same way we can give the MQTT topic a custom name)? For Example my Temperature Sensor in the attic in HomeSeer is called: "TS - Attic" and with mcsMQTT it is published as "Jarvis/Temperature/Attic_1" but it is stored in InfluxDB as "TS - Attic". Is there a way for me to give it a name like I gave it in MQTT like "Attic_1" or something else?
                          It is no big deal if this is not possible as I know what to query. It is just that while some names make sense for a wild HomeSeer Ecosystem, they do not need to be as complex for a DB name.

                          This plugin and moving everything to the HS4 HomeSeer computer on Windows is EXACTLY what I needed. Next I will tackle charting on both mcsMQTT itself and Grafana.

                          You are awesome!! Thanks again!
                          ---------------------------------------------------
                          Jean-Marie G. Vaneskahian
                          jean@vaneskahian.com
                          ---------------------------------------------------

                          Comment


                            #28
                            Messages received from the Broker are queued in the order received and worked off in sequence order. Processing this queue is its own single thread so there should be no timing issues. mcsMQTT provided a debug facility from the top of MQTT Page General Tab. The resultant file is \data\mcsMQTT\mcsMQTT Debug.txt

                            When I ran my test I used a JSON message with three keys and the rate device was created off the third of these. In the debug file the following is shown. The timing starts with ProcessMessage and ends with ActOnMessageForTrigger. In your case I expect two ProcessMessage/ActOnMessageForTrigger sequences and the second will have the ApplyExpression to sum the two payloads. Run your test for the case where the second message is the one where the rate device was created. Post the file.
                            Code:
                            8/27/2022 12:41:29 PM 152357 | ProcessMessage DS18B20
                            8/27/2022 12:41:29 PM 152493 | Update Accepted 9 to 10 StatusType=2 Payload=10
                            8/27/2022 12:41:29 PM 152550 | Updating Device from 10 PayloadNumeric=True
                            8/27/2022 12:41:29 PM 152551 | Update Accepted 5 to 33.5 StatusType=2 Payload=33.5
                            8/27/2022 12:41:29 PM 152554 | Updating Device from 33.5 PayloadNumeric=True
                            8/27/2022 12:41:29 PM 152555 | Update Accepted 7 to 50.8 StatusType=2 Payload=50.8
                            8/27/2022 12:41:29 PM 152557 | Updating Device from 30.8 PayloadNumeric=True
                            8/27/2022 12:41:30 PM 153712 | ApplyExpression 50.8+33.5 Result=84.3
                            8/27/2022 12:41:30 PM 153714 | ActOnMessageForTrigger QueueSize=1 ,Topic DS18B20,Payload {"Time":"2022-01-24T11:19:22","DS18B20-1":{"Id":"012025907848","Temperature":10.0},"DS18B20-2":{"Id":"01204C350836","Temperature":33.5},"DS18B20-3":{"Id":"01204C3DFEDA","Temperature":50.8},"TempUnit":"F "}
                            For the InfluxDB field names you have two applicable settings on the History Tab of the MQTT Page

                            Click image for larger version

Name:	0x.png
Views:	0
Size:	0
ID:	1563651

                            Five options are provided for how you want to identify the HS DeviceValue in InfluxDB. Default is by floor, room and name. Using Ref guarantees uniqueness, but you have no control of Ref number while you do have control of floor, room and name.

                            If you want additional information stored along with DeviceValue then those are specified in the Extra Identification Fields. mcsMQTT.pdf will provide more guidance on using this field.

                            Comment


                              #29
                              I will play with looking at the debug to understand how the final value is computed. Maybe Power1 and Power0 and not being stored by the RTL_433 decoder in the same order every time?

                              As to the name being stored, I will look at the options. These are both very minor issues. Thanks again so much!!!
                              ---------------------------------------------------
                              Jean-Marie G. Vaneskahian
                              jean@vaneskahian.com
                              ---------------------------------------------------

                              Comment


                                #30
                                Michael McSharry, it is funny, I got so caught up on the mechanics of this that I never figured out how to do the main topic of this thread.

                                Now that I have a HomeSeer HS4 device that has the whole house Wattage use at any given time and that HS4 mcsMQTT is aware of and logs into InfluxDB, how do pull that into HomeSeer Energy DB for use in HS4?

                                Click image for larger version

Name:	Screenshot_2022-08-28_14-17-10.jpg
Views:	102
Size:	22.3 KB
ID:	1563767

                                Click image for larger version

Name:	Screenshot_2022-08-28_14-17-31.jpg
Views:	75
Size:	41.1 KB
ID:	1563768

                                Click image for larger version

Name:	Screenshot_2022-08-28_14-17-42.jpg
Views:	62
Size:	33.3 KB
ID:	1563769

                                HS4 seems to know about my ZWave power devices natively. How do I go about adding my new Whole House Wattage device to it?

                                Since my values are in Watts at any given time and the final displayed energy use is in Kilowatt Hours, does HomeSeer automatically time the time series and add them up?

                                This is my last question for this thread...
                                ---------------------------------------------------
                                Jean-Marie G. Vaneskahian
                                jean@vaneskahian.com
                                ---------------------------------------------------

                                Comment

                                Working...
                                X