How Can mcsMQTT Pull Data Into HomeSeer Energy?

[jeanv]

Awesome! Now I understand when I could not find it. That field for "HS Energy Database" is only visible to devices that are created by mcsMQTT from the topic it is subscribed to on the MQTT broker. If it is a native HomeSeer device that mcsMQTT did not create but has an association in mcsMQTT then the "HS Energy Database" field is not available.

Here are my devices that mcsMQTT created and I renamed as needed (and put them in a new Location and Floor)



So it is Device Ref# 1435 that I want in the HomeSeer Energy DB. I went to the Edit for the Ref# 1435 in mcsMQTT



When I hit Edit I get



The Expression that is now finally working is:

$$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power0):+$$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power1):

And I have checked save as Watts for the HomeSeer Energy Database

Everything should now be working. But I still do not see this new REF# 1435 "Watts - Used In Whole House" (See above) in the HomeSeer Energy page. How long do do I need to wait before that new device is available in HomeSeer Energy DB?

[Michael McSharry]

From mcsMQTT.pdf. The radio to select it is near the bottom of the Edit tab. Note there was reverse engineering done to figure out what needed to be done to store the data in the format that HS actually uses. If you see something that does not look correct then let me know.

HS maintains a database to record energy utilization. It will accept the Watt consumption or production over sensed periods of time. To utilize this database the HS Energy Database row of the Edit tab is used. The preferred source is a total energy sensor source. The plug-in will handle a daily energy sensor as well, but energy consumed between the last sample of prior day and midnight may introduce minor variance, depending upon the frequency of the sensor’s sampling and reporting.

[jeanv]

Michael McSharry, it is funny, I got so caught up on the mechanics of this that I never figured out how to do the main topic of this thread.

Now that I have a HomeSeer HS4 device that has the whole house Wattage use at any given time and that HS4 mcsMQTT is aware of and logs into InfluxDB, how do pull that into HomeSeer Energy DB for use in HS4?







HS4 seems to know about my ZWave power devices natively. How do I go about adding my new Whole House Wattage device to it?

Since my values are in Watts at any given time and the final displayed energy use is in Kilowatt Hours, does HomeSeer automatically time the time series and add them up?

This is my last question for this thread...

[jeanv]

I will play with looking at the debug to understand how the final value is computed. Maybe Power1 and Power0 and not being stored by the RTL_433 decoder in the same order every time?

As to the name being stored, I will look at the options. These are both very minor issues. Thanks again so much!!!

[Michael McSharry]

Messages received from the Broker are queued in the order received and worked off in sequence order. Processing this queue is its own single thread so there should be no timing issues. mcsMQTT provided a debug facility from the top of MQTT Page General Tab. The resultant file is \data\mcsMQTT\mcsMQTT Debug.txt

When I ran my test I used a JSON message with three keys and the rate device was created off the third of these. In the debug file the following is shown. The timing starts with ProcessMessage and ends with ActOnMessageForTrigger. In your case I expect two ProcessMessage/ActOnMessageForTrigger sequences and the second will have the ApplyExpression to sum the two payloads. Run your test for the case where the second message is the one where the rate device was created. Post the file.

8/27/2022 12:41:29 PM 152357 | ProcessMessage DS18B20
8/27/2022 12:41:29 PM 152493 | Update Accepted 9 to 10 StatusType=2 Payload=10
8/27/2022 12:41:29 PM 152550 | Updating Device from 10 PayloadNumeric=True
8/27/2022 12:41:29 PM 152551 | Update Accepted 5 to 33.5 StatusType=2 Payload=33.5
8/27/2022 12:41:29 PM 152554 | Updating Device from 33.5 PayloadNumeric=True
8/27/2022 12:41:29 PM 152555 | Update Accepted 7 to 50.8 StatusType=2 Payload=50.8
8/27/2022 12:41:29 PM 152557 | Updating Device from 30.8 PayloadNumeric=True
8/27/2022 12:41:30 PM 153712 | ApplyExpression 50.8+33.5 Result=84.3
8/27/2022 12:41:30 PM 153714 | ActOnMessageForTrigger QueueSize=1 ,Topic DS18B20,Payload {"Time":"2022-01-24T11:19:22","DS18B20-1":{"Id":"012025907848","Temperature":10.0},"DS18B20-2":{"Id":"01204C350836","Temperature":33.5},"DS18B20-3":{"Id":"01204C3DFEDA","Temperature":50.8},"TempUnit":"F "}

For the InfluxDB field names you have two applicable settings on the History Tab of the MQTT Page



Five options are provided for how you want to identify the HS DeviceValue in InfluxDB. Default is by floor, room and name. Using Ref guarantees uniqueness, but you have no control of Ref number while you do have control of floor, room and name.

If you want additional information stored along with DeviceValue then those are specified in the Extra Identification Fields. mcsMQTT.pdf will provide more guidance on using this field.

[jeanv]

I have installed the new version of the plugin and things most everything works!!! Thank you so much!!! I do have a small issue but it is no big deal.

I got the new Rate device based on Power0 set to "None" and then on the new device I created an evaluation of:

$$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power0):+$$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power1):

That populated the new rate device with the value old value of Power1 + the new value of Power0

I then tried:

$$PAYLOAD:+$$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power1):

Same Problem

I then did the same thing but this time created the rate device based on Power1 and the evaluations of the new rate device were:

$$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power0):+$$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power1):

Now the problem was

That populated the new rate device with the value old value of Power0 + the new value of Power1

also tried

$$PAYLOAD:+$$PAYLOAD:(rtl_433//devices/CurrentCost_TX/power0):

Same problem.

This appears to be a timing problem as both Power0 and Power1 get published to the MQTT broker seemingly simultaneously.

The way the script I wrote gets around this problem is by waiting 2 seconds after Power0 gets updated before adding Power0 and Power1 so that both values are current.

mcsMQTT is very fast and is able to grab the value and update before both have populated. Not sure...

Maybe if there was some kind of delay.. But honestly it is working just fine with the script.

I cannot thank you enough!

A quick question:

1. I see that I can publish a HomeSeer device to the MQTT broker with any topic name I want, and that is great! However, it writes the HomeSeer Device name to the InfluxDB. Is there a way to give the Device a custom name for the DB entry (same way we can give the MQTT topic a custom name)? For Example my Temperature Sensor in the attic in HomeSeer is called: "TS - Attic" and with mcsMQTT it is published as "Jarvis/Temperature/Attic_1" but it is stored in InfluxDB as "TS - Attic". Is there a way for me to give it a name like I gave it in MQTT like "Attic_1" or something else?

It is no big deal if this is not possible as I know what to query. It is just that while some names make sense for a wild HomeSeer Ecosystem, they do not need to be as complex for a DB name.

This plugin and moving everything to the HS4 HomeSeer computer on Windows is EXACTLY what I needed. Next I will tackle charting on both mcsMQTT itself and Grafana.

You are awesome!! Thanks again!

[Michael McSharry]

In the attached I removed the check for device vs. feature for the L and S checkboxes. I believe the 5.26.3.0 attachment is at the bottom of post 24, but 5.26.3.1 is the one you want to include 844 and 847.

Using the event and script of course works so it a matter of preference. Doing manipulation of raw data is something that is a commonly needed and that is why I have included the two derived devices to mcsMQTT on the Edit tab. I did not realize until now that this simple addition of multiple payload items could not be accomplished without the None option for the Rate device.

[jeanv]

Thanks so much! It makes perfect sense in terms of how the new device is created, but unfortunately I do not see (5.26.3.0) in your previous posting anywhere. I think you might have forgotten to attach it.

Also, while I was waiting to implement the sum of the two power devices (Power0 and Power1), I went ahead and wrote a script to just add their two values and put the result in third device. I then created an event that triggers when Power0 is updated or set and runs the script 2 seconds later. That way I do not have to worry about which updates first (Power0 or Power1). It is not as clean as your proposed solution but it seems to work.

Here is the new device I created whose only purpose is to store the data of the sum of Power0 and Power1



Here is the script I wrote that adds the two values for Power0 and Power1 and then writes the resulting sum in that new device (Ref# 1420).

public enum Jarvis_Variables
{
    Energy_Whole_House_Phase_A = 1413,
    Energy_Whole_House_Phase_B = 1414,
    Energy_Whole_House = 1421
}

public void Update_Energy_Whole_House(object Parms)
{
    double Energy_Whole_House_Phase_A = (double)hs4.GetPropertyByRef((int)Jarvis_Variables.Energy_Wh ole_House_Phase_A, HomeSeer.PluginSdk.Devices.EProperty.Value);
    double Energy_Whole_House_Phase_B = (double)hs4.GetPropertyByRef((int)Jarvis_Variables.Energy_Wh ole_House_Phase_B, HomeSeer.PluginSdk.Devices.EProperty.Value);
    double Energy_Whole_House = Energy_Whole_House_Phase_A + Energy_Whole_House_Phase_B;
    hs4.UpdateFeatureValueByRef((int)Jarvis_Variables.Energy_Who le_House, Energy_Whole_House);
    hs.SetDeviceLastChange((int)Jarvis_Variables.Energy_Whole_Ho use, DateTime.Now);
}

Here is the event trigger that waits for Power0 to have a value set or changed and then waits 2 seconds and fires off the script.



Here is the result. You can see that the new device with the sum of Power0 and Power1 (called "Whole House Energy", Ref# 1420) is updated 2 seconds after Power0 and Power1.



I then went into mcsMQTT and added this new device, "Whole House Energy" (Ref# 1420) to associations and and checked the h, s, l checkboxes. I then checked my InfluxDB and sure enough, it was there along with Power0 and Power1!



I still would very much like to implement the solution you brought up as soon as I get the updated plugin with the rate option of none.

---

As to your second point on:

How were you able to run HS4 with 844 and 847 as root devices?

These devices were created by a HS3 Legacy Plugin called "RFXCOM". It pulls in wireless sensors (in this case Oregon Scientific Temperature and Humidity Sensors) and creates a device with features. The plugin unfortunately creates the Temperature Sensor portion as the "Root" parent device and the Humidity and Battery levels as child "features".

Here is a screenshot of the devices:



As you can see Ref# 847 and Ref#844 are Parent / Root Devices, but they have the temperature data that I am interested in getting into InfluxDB via mcsMQTT.

I can 100% confirm that an event created that fires off when the parent / root is set or changes value does work. From this perspective, there is nothing different about the Parent / root device vs. the feature / children. Values can can be set and changed and events can respond to those values being set and changed. All I want to be able to do is log the values of these parent / root temperature sensors into InfluxDB via mcsMQTT.

Does that answer what you were looking for?

Is there anything else I can do?

I really appreciate your help and updates.

[Michael McSharry]

Still like to understand

How were you able to run HS4 with 844 and 847 as root devices? I thought HS4 would not allow such things. I can remove the constraint and always show the S and I checkboxes, but not clear to me if HS4 will generate a HS Event trigger for a root device.

Looking at the code I see that it was not as simple to use the rate device for this purpose. I updated in 5.26.3.0 (attached) to add a rate option of None which will not consider the time interval and then the function becomes essentially whatever you want to put in the Expression textbox. If you leave this textbox blank then the HS device will be updated to the same as the device from which the rate device was derived with the low pass filtering still being applied.

The most straightforward process and the one I tested is similar, but not exactly the process I first described.
1. Associate the power0 and power1 to get two HS Features with Ref.
2. On the one that come later in the MQTT message delivery, use the Edit tab to create a rate device, sensitivity filter 1.0, None for the rate interval.
3. A new HS feature will be created and if the Association table is refreshed it will be visible as a new row in the table.
4. Use Edit tab on this rate device row to enter the expression in the Expression textbox. Other settings can be the default. The expression is easiest to understand if both power0 and power1 are explicitly specified, but the original expression should work too.

Below is my test case where Temperature 1 (Ref 9) and 2 (Ref 7) are summed in the rate device (Ref 8) . The long term storage can be selected for any of them on the Association table.

$$PAYLOAD:(DS18B20:DS18B20-1:Temperature):+$$PAYLOAD:(DS18B20:DS18B20-2:Temperature):

[jeanv]

Yes, I apologize if I was not clear on what I was running. I am HS4 (and the HS4 version of the Plugin). I am running it on Windows 10.

I installed the binaries you sent me and they worked great. The checkbox stayed and the new device was created. The problem is the value.

Just so we are on the same page.

1. The value of the new device should simply be the total Wattage of Power0 + Power1 right?
2. If Power0 = 2000 and Power1 = 1000 then the new device should have a value of 3000
3. This Rate concept asks what time interval I want? (Second / Minute / Hour), How does this apply?
4. I get crazy numbers on the new device (like negative number)

[Michael McSharry]

First problem is I assumed, incorrectly, that it was HS3 plugin you are using. From the video and response, it is clear that it is HS4. How were you able to run HS4 with 844 and 847 as root devices? I thought HS4 would not allow such things. I can remove the constraint and always show the S and I checkboxes, but not clear to me if HS4 will generate a HS Event trigger for a root device.

The problem with the rate device results form the difference in HS3 and HS4 where the first returns "checked" and the second returns "true" when a checkbox is checked. I missed this on the rate device so it still had the HS3 expectation in the HS4 plugin. It is fixed in 5.26.2.1.

You can change the room, floor and name on either the Edit tab or on the HS Devices page. The plugin does not depend upon these for any logic. You can also regroup features under a device and create new devices for this purpose on the Edit tab Grouping row. It is not necessary, but just making you aware of it.

The attached 5.26.2.1 is unzipped into the \bin\mcsMQTT folder.

[jeanv]

Here is a video of the problem I am running into with the instructions on adding the two power0 and power1 into a new 3rd device that you gave me.

Any ideas why it will not create the new device and stay checked?

[jeanv]

Michael McSharry, so for the sensors that I have that are "Root" devices and not features in HS4, what are my options for getting mcsMQTT to go their values to InfluxDB? Should I create new devices with a feature that gets its value from these "Root" devices so that mcsMQTT will put those into InfluxDB? If I use this approach, what is the best way to update the values on these newly created device features (an event that checks the device time of the old root and then does an update?)

Also, I will work on the second issue with adding the Power0 and Power1.

[Michael McSharry]

844 and 847 are likely listed in HS as Devices. In HS4 Devices are only containers and only Features can generate HS Event callbacks so no ability for plugin to get an event trigger from a Device so checkboxes are not shown. I should not use the same dependency in HS3 and need to update the HS3 plugin to allow the S and L checkboxes.

The basic strategy is to associate both topics a HS device. Let us say HS devices are 100 for power0 and 101 for power1. Click the 101 ref button on Association tab to bring up Edit popup. Select a Rate Device to be created with a weighting factor of 1.0. This will create device 102. 100 and 102 will now get updated with the individual power values. In the Expression text box on Edit tab for the 101 device enter the line below to sum the payload of power1 with the power0. It is possible that inline expression designation is needed and if so then double < and > should be placed around the expression. I did not try before posting this.

$$PAYLOAD:+$$PAYLOAD:(rtl_433/rpi4-iot/devices/CurrentCost-TX/1643/power0_W):

Select the L checkbox on power0 topic on the Association tab so it goes into InfluxDB. Edit the Device names in HS so they make sense to you. If the power0 is delivered after power1 then swap the rate device creation to 100 rather than 101 and user power1 in the expression.

[jeanv]

Dweber85rc ,

What you need to do is on the association tab of mcsmqtt, change the filter to include non plugin hs devices and then on the device select the "h" checkbox to the right of that device to write it to the database

I did that, but while I can check the "h" value, I cannot set the short term and long term database logging. Why is this?