Publish-Subscribe options

[Slab0]

When HS device change occurs only the 'payload' is affected. The 'source' is the subscribe topic. The 'topic' is the publish topic. At the point of publishing a payload a check is made for 'source' = 'topic' and in that case the publish is skipped. Note this happens only when HS reports a device value has changed. It is not bypassed when HS controls a mcsMQTT device. In your case the following is the expected scenario.
1. external node published Light=ON
2. mcsMQTT uses CAPI to tell HS to forward ON command to Zwave plugin
3. Zwave plugin controls its hardware
4. Zwave plugin updates HS Devices status
5. HS delivers Device change event to mcsMQTT
6. mcsMQTT recognnizes the device has a topic to be published, but does not publish because source (subscribe topic) = topic (publish topic)

Thanks very much for the insight. Obviously that represents a challenge in my case, because all devices have the same topic for sub/pub.

As mentioned previously, I just wanted to stop the re-publishing of a device value that is received via the sub. The external node already set the topic value, HS doesn't need to send it back out on the same topic (and indeed, this could create an issue with external nodes). I do need HS to always publish when it (gui/script/event) changes the value of a device (vs. receives it from the sub)... regardless of what the pub topic is.

From my perspective, the sub=pub approach doesn't need to be the the determining factor. I would be equally happy (maybe more so) with an additional configuration checkbox when editing a given device.... perhaps a "Only publish HS initiated device changes", or "Do not publish subscription topic changes", or something equivalent.

Please let me know if there's anything else you need me to do... happy to help! Thanks much.

[Michael McSharry]

Lots of combination of things that need to be updated on the UI when the plugin tries to be user friendly. I tested many sequences, but likely others will find combinations that I had not anticipated. Let me know if you have any other findings based upon your use cases. 3.4.3.3 is in the updater.

[Slab0]

Great progress! Here are my notes:

1) Fixed...I only have to change the pub topic once now when doing the initial association on the Associations tab. The db is updated appropriately, and it doesn't revert to the default pub template anymore!

2) Awesome change to the Edit/Add tab! The old format was the single biggest source of confusion for me initially. The new format now only displays relevant data for a non-plugin device:



And equally appreciated is the change to the "Delete Sub Topic" button, which I assume doesn't delete the device as well. I had on two occasions previously deleted both the device and topic before I understood that even though it's a real device, it would still get deleted (which meant re-scanning, a new device ID, changing events, etc...). I also no longer see a change to the db topic the first time I go to Edit/Add tab either. Overall, very nice overhaul and much more user friendly.

Unfortunately, nothing ever gets published still. The statistics tab continues to only show "win7pro-VM/mcsMQTT/LWT" as last published.

Fundamentally, there are just two broad sources that can affect a change of a device's status/value: 1) the sub topic and 2) 'anything else' . I'm looking for a way to configure mcsMQTT to do what a device's sub payload says with no other action, and to publish what 'anything else' does to the device so the rest of the environment is aware of current status. Based on my (albeit limited) experience with other bits of mqtt enabled software, they operate in this very manner. They perform tasks based on inbound sub topic values, and they publish topics based on what they do to the device. They don't automatically do both in tandem, no matter what the topics are... or at least I haven't encountered that yet. If I have a need to re-publish an incoming sub payload, either on the same topic or any other number of other topics, I can do that via scripts, events, etc... I would just like to see a means to stop the plugin from doing it automatically.

In looking at your 6 step list describing how the plugin works, it looks like mcsMQTT get's involved either at step 2 or step 5. If not done already, perhaps at step 2 a flag is set indicating that the sub payload was received. When step 5 executes, if the flag is set... don't publish, if not (meaning 'anything else' affected a change) then publish. The sub and pub topics being equal doesn't matter much to me, but I can work with that if that needs to be the determining factor as to publishing an incoming sub payload or not.

If that's not doable without a disruptive overhaul, then if you could just back out the sub=pub check and I'll attempt to deal with the plugin re-plublishing received topic payloads on the external nodes themselves (hopefully ).

Debug file attached... thanks much!

[Michael McSharry]

I did not find anything specific, but did some general accounting to assure HS has been told to report event changes. It is working in my scenarios using HS devices. I also added debug around the event processing that is similar to what was there in prior versions but removed because it was chatty and was no longer looking at the event logic.

Give it a try and include debug if it does not work for you. I have provision for 5 seconds for the Zwave (or other) plugin to act upon the CAPI request and report device value change back through HS. If longer than that then mcsMQTT should publish for the case where source=topic. It will always publish if source and topic are not the same and there is a publish topic.

In updater at 3.4.3.4

[Slab0]

After some brief testing (more to go) it seems everything is back in working order. Both pub and sub appear to be working fine, the db looks good, and I didn't note anything odd in the debug file. Thanks very much!

I have provision for 5 seconds for the Zwave (or other) plugin to act upon the CAPI request and report device value change back through HS. If longer than that then mcsMQTT should publish for the case where source=topic.

To be clear, I assume if CAPI doesn't respond in 5 secs that the plugin doesn't just immediately publish the sub payload, but will publish the next change for the device that CAPI hands to it after the 5 sec provision... correct?

I was going to question the timer approach, but as I gave it more thought I understand the challenge... there really isn't a good way (at least not without a schema change) to determine where a CAPI event was initiated. It's entirely possible that an external node can publish a device change and a HS initiated action could nearly concurrently change the device as well.

A couple more usability nits that I've noted:

1. Initial associations don't appear to inherit the default QOS set on the General tab... they are always set to 'Exactly' instead.

2. I've noted a minor inconsistency on the Associations tab under "Filter Association Table by Category". When checking "Show All Accepted Only" the option "Include MQTT Topics" remains checked, even though only the accepted associations are actually shown. I would think it would be cleared, or alternatively the 'only' word would be removed and accepted would be shown along with MQTT topics.

I'll keep testing... thanks again for the great plugin.

[Michael McSharry]

To be clear, I assume if CAPI doesn't respond in 5 secs that the plugin doesn't just immediately publish the sub payload, but will publish the next change for the device that CAPI hands to it after the 5 sec provision... correct?

It affects only the events HS raises within 5 seconds of the topic being received from broker. If it takes Zwave more than 5 seconds to report success on changing the state of its endpoint then mcsMQTT will publish. Any other changes of the devices will also be published after the 5 second window.

I was going to question the timer approach, but as I gave it more thought I understand the challenge... there really isn't a good way (at least not without a schema change) to determine where a CAPI event was initiated. It's entirely possible that an external node can publish a device change and a HS initiated action could nearly concurrently change the device as well.

For clarity, CAPI is the method one tells a plugin to change the state of a device that the plugin owns. HS will only communicate CAPI to a single plugin. Other plugin find out about state changes if they have registered to have change in any device reported. It is up to the plugin that owns the device if it will change the value based upon a CAPI request.

For mcsMQTT, if a user presses a HS browser button to turn off a Sonoff light then mcsMQTT will publish the request. It will not update the associated Device value. The Device value is only updated when the subscribed topic is received. This means a positive confirmation is needed before the HS Device state is updated. I don't know what approach is used for Zwave plugin. I had thought about making 5 seconds a user parameter, but did not because of the special case nature of subscribe=publish topic.

1. Initial associations don't appear to inherit the default QOS set on the General tab... they are always set to 'Exactly' instead.

I'll look into this

2. I've noted a minor inconsistency on the Associations tab under "Filter Association Table by Category". When checking "Show All Accepted Only" the option "Include MQTT Topics" remains checked, even though only the accepted associations are actually shown. I would think it would be cleared, or alternatively the 'only' word would be removed and accepted would be shown along with MQTT topics.

The intent is to make it easy to see only the accepted topics. I agree that it should be mutually exclusive and the checkboxes shown to reflect exactly what will be done.

[Slab0]

I just updated to version 3.4.4.0 (and as a matter of course over the years, I use the 'disable plugin-apply update-enable plugin' approach). The first thing I did after enabling was to check the db timestamp... it hadn't changed (I didn't expect it to at this point). I then went to the Associations tab to check that all was okay. It wasn't... all the publish topics were changed to the default template, while the sub topics remained correct and unchanged.

I decided to see if the Edit/Add tab showed anything different... and entered one of the device IDs. It looked fine... correct sub and pub topics. Hmmm... went back to the Associations tab, and it showed the device I just looked at in Edit/Add was now correct, but the other devices were still incorrect. And the db was updated (attached, along with the debug file... although debug was not enabled on the General tab). The db has the correct pub topic for the one I looked at on the Edit/Add tab, the others are incorrect.

When I plugged in the other device IDs in Edit/Add... they all had the incorrect pub topic as per the attached db.

** update **
It looks like the pub topic is randomly getting set back to a default template in the db. Not precisely sure what I am doing to trigger this, other than examining the Associations tab and occasionally viewing a device config via the Edit/Add tab.

Also, one of my devices (89) stopped acting on the incoming sub. The stats tab shows that it received the sub, but it doesn't appear to send anything to CAPI. The db doesn't get updated, and indeed the payload field is blank for that device. Publishing from HS to external works fine for this device, and the other devices appear to work fine for both pub and sub. mcsMQTT_2.zip attached...thx much.

[Michael McSharry]

The topic is overwritten in db when a new payload has been received. It does not matter until the plugin restarts and initializes from the database. This has been addressed in 3.4.4.1 as well as your earlier observations.


The debug below shows 89 being processed and the new 5 second window working as expected. In this case Payload = 0. Why do you believe the topic is being ignored? I noticed that payload does not show on the Association tab for non plugin devices. When looking into this I recognize that non plugin devices do not update Association tab Payload. I changed this for the case of subscriptions mapped to non-plugin devices in 3.4.4.1.

See if this resolves your issue of if it is something else.

6/25/2018 9:32:07 AM	54582934	| Update Accepted 89 to 0 StatusType=0  
6/25/2018 9:32:07 AM	54582940	| Command nonPlugin Device 89 to 0  
6/25/2018 9:32:07 AM	54582940	| Command 89 to 0, nValue=0, ControlValue=255, Range=Nothing, ControlType=Button, ControlString=, Label=On, ControlUse=_On  
6/25/2018 9:32:07 AM	54582940	| Command 89 to 0, nValue=0, ControlValue=0, Range=Nothing, ControlType=Button, ControlString=, Label=Off, ControlUse=_Off
  
6/25/2018 9:32:07 AM	54582944	| MessageSourceExists SELECT count(Source) as extant FROM MQTT_MESSAGE WHERE (Source='homeseer/outside/entrance_light')=1  
6/25/2018 9:32:07 AM	54582944	| SaveReceive Updating homeseer/outside/entrance_light  
6/25/2018 9:32:07 AM	54582959	| ActoOnMessageFor Trigger Topic homeseer/outside/entrance_light,Payload=0  

6/25/2018 9:32:07 AM	54583052	| HSEvent Do= True VALUE_CHANGE for Device 89  
6/25/2018 9:32:07 AM	54583053	| HSEvent Add VALUE_CHANGE | 89  
6/25/2018 9:32:07 AM	54583053	| DoHsEvent LastReceivedDate=6/25/2018 9:32:07 AM, Do=False  
6/25/2018 9:32:07 AM	54583054	| DoHsEvent Ref=89 has been bypassed

[Slab0]

The debug below shows 89 being processed and the new 5 second window working as expected. In this case Payload = 0. Why do you believe the topic is being ignored?

Those log entries are from 9:32 AM. Everything appeared to be working fine up until the update to 3.4.4.0, which happened somewhere around 3:00 PM. After changing all the pub topics back to what they should be after the update, the plugin would no longer send received payloads to CAPI for device 89 (or so it appeared, the device didn't change and the HS log showed no entries). The stats tab indicated that the plugin was receiving on the sub, but nothing else happened for that device. I just tried again prior to updating to 3.4.5.0, and it still didn't work.

See if this resolves your issue of if it is something else.

Considering the current state of the db, I started from scratch again... keeping track of the db contents as devices were associated or viewed in the Edit/Add tab. I didn't note anything wrong in the db throughout the process, and no unexpected topics appeared. Everything seems to be working great, including device 89. I'll be doing more testing later today, and we'll see what happens when the next update comes along... I'll post back regardless. I'm going to keep debug enabled on the General tab for a while in case something else crops up.

It looks like the other items that I noted have been addressed, and I'm happy to see the addition of the RegEx fields on the Edit/Add tab... very nice! Thx.

[Slab0]

I updated to 3.4.5.1 and am happy to report that it went smoothly. The only change that I noted to the db is that the LastDate field for the devices got reset to "1900-01-01 00:00:00" and the Payload field was cleared (I think that occurred the first time that I checked either the Edit/Add tab or Associations tab... no matter, there were no adverse affects and everything works fine).

Regarding the Edit/Add tab, I like the color change for consistency but it looks like the format got a bit messed up:



Thx.

[Michael McSharry]

Your observations have been corrected in 3.4.5.2 which is in the updater. Thank you for the help.

[Slab0]

Just updated to 3.4.5.2 and it went perfectly. The Edit/Add tab is fixed, and I see that you even addressed the LastDate/Payload fields getting reset... look's like they were not changed.

I'll keep moving along, but things feel pretty solid/reliable at this point. I'll let you know if anything comes up ...thx much!