Publish-Subscribe options

[Slab0]

Hi,

I have a number of Homeseer managed devices that I also need to control via an external management server (that supports mqtt). The external server needs to know about status changes that are initiated by Homeseer for those devices as well. With that, I've set up this (excellent!) plugin to facilitate, and it's working very well.

The only side effect of this configuration is that mcsMQTT effectively echoes back to the publish topic (which the external server is subscribed to) device status changes that are received from the external server. While the external server does use mySQL to track device change events, fortunately it does not add another entry to the DB if the device status hasn't actually changed from it's last known state, so there's no major issue at this point... just some redundant traffic.

I've chosen to configure mcsMQTT devices and the external server devices to use the same topic for both subscribe and publish. Is there a way to prevent publishing device status changes if received from a topic, when the same topic is used for publishing? Alternatively, can a global (or per device) configuration option be added to only publish device changes that are initiated by Homeseer (vs. received from a subscribed topic)? Or is there some other option that I should consider?

I'm more curious than anything, nothing at all urgent here. As mentioned, everything works great and does what I need. And thanks very much for mcsMQTT... it really is excellent!

[Michael McSharry]

Is there a way to prevent publishing device status changes if received from a topic, when the same topic is used for publishing?

Should be able to do it. Similar to what I do with mcsSprinklers when X10 can come from multiple sources and don't want endless loops.

Glad the plugin works well for you.

[Slab0]

Great...thank you for looking into it!

Concern for loops, or more accurately wondering why I wasn't seeing any, was what motivated me to look into the messaging in my environment in more detail. Fortunately, it appears that the external server only publishes device changes initiated via it's own interface... not from subscribed topics.

Thanks again.

[Michael McSharry]

Now we have question of where truth is. If we have Topic X and HS publishes this topic then the natural assumption is that HS owns the truth for it so if something is received from the same Topic then HS should ignore it.

The second model is like a Wiki where the last one to update a subject becomes the one from where truth comes. Translating this to HS it means that HS should update its Device Status when the broker publishes. This is the one I think you want. Please confirm.

[Slab0]

The second model is like a Wiki where the last one to update a subject becomes the one from where truth comes.

I believe that this scenario is accurate, but to be a bit more specific:

If a device is configured with the same sub and pub topic*

- when a device update arrives from the broker, update the device per payload, but do not publish the update

The assumptions here are 1) any external entities that are subscribed to the topic will update their status regardless of authorized pub source, and 2) Homeseer successfully updates the device as requested by the broker. Everything should be in sync if these are true.

- if the device is changed via Homeseer interface/script/event, publish so that external subscribers can update as they see fit

So as you reference, the last update published (regardless of source authorized to access the broker) should be the current device status for everything.

* As an alternative to using the same pub/sub topic, provide a device config checkbox to only publish when the device status change is initiated by Homeseer, not when received from the sub.

If a device is configured with different sub and pub topics (or the checkbox mentioned above is not checked)

- operate as it does today... update per sub and also pub new status


I should mention that this is my first encounter with MQTT, so if I'm missing something fundamental (and I may well be ) please feel free to educate me. As I said, things appear to be working fine since implemented yesterday. Thx much...

[Michael McSharry]

I'm having a little difficulty understanding your configuration. Is Topic X setup to map into an existing HS3 device or was a new HS3 device created by mcsMQTT when a subscribed Topic was accepted?

If mcsMQTT created it then it will not publish based upon status changes. Only non-mcsMQTT devices publish through the status change interface (HSEvent).

If a Topic is received for mcsMQTT device then mcsMQTT updates HS Device Value.

If a Topic is received for a non-mcsMQTT device (one owned by HS or another plugin) then when the Topic is received it will use the CAPI interface to try to tell HS/Other Plugin to control the device. HS/Other Plugin may or may not change Device Value. If they do then the HSEvent interface comes into play.

For your scenario, are you using mcsMQTT devices or devices that were created by HS/Other plugins?

[Michael McSharry]

I put 3.4.3.0 in the updater based upon what I think you are asking. Give it a try and let me know if it works for you.

[Slab0]

For your scenario, are you using mcsMQTT devices or devices that were created by HS/Other plugins?

I'm only working with HS created devices (all z-wave so far)... I haven't yet approached using mcsMQTT devices.

If a Topic is received for a non-mcsMQTT device (one owned by HS or another plugin) then when the Topic is received it will use the CAPI interface to try to tell HS/Other Plugin to control the device. HS/Other Plugin may or may not change Device Value. If they do then the HSEvent interface comes into play.

Indeed, this is what I've observed. However, the plugin does appear to publish the new status after sending a request off to the CAPI interface (I understand that HS or other plugin is the arbiter as to if the device the status changes at this point). It's that follow-up publish that I'd like to optionally disable.

I'll test the new version as soon as the updater makes it available and will follow up. Thanks again!

[Slab0]

Well...I applied the update, and now nothing works .

First, the update made a bit of mess of my associated devices. It duplicated all of them, and each was missing either the sub of pub or reverted to a default topic format (here's a partial snapshot of what I ended up with post update):



I cleared them all and re-associated my devices like this:



Now, when I publish from the external server I can see the topic/payload is received on the mcsMQTT statistics tab, but the action isn't sent off to CAPI (the device status doesn't change, and the HS log doesn't show the action).

When I turn the device on/off on HS, the statics tab doesn't show anything as published, and the external server receives nothing on it's associated sub.


With the prior release, when I would publish from the external server, the statistics tab always showed it as the last received topic and also the last published topic. Unless I am misinterpreting, the plugin always published what it received. I always want the plugin to send received payloads to CAPI (or update an mcsMQTT managed device if that's used), and I always want the plugin to publish when the device is changed by HS... I was just hoping for an option to disable automatically publishing all received subs.

So, programmatically:

Assume topic xyz, and it's associated with device Light1 in mcsMQTT

1) external server publishes to topic=xyz with payload=255

2) mcsMQTT sub topic=xyz on device Light1 receives payload=255

3) mcsMQTT sends 255 to CAPI Light1 (or mcsMQTT Light1 if that's what it is)
 
[B]4) Current behaviour:[/B]

 publish payload=255 on Light1 pub topic

[B]4) Desired behaviour:[/B]

if Light1 sub=xyz AND Light1 pub=xyz
  done
else
  publish payload=255 on Light1 pub topic

[Michael McSharry]

The database of associations is backed up so you are able to return to an earlier setup by renaming the database file that is at \Data\mcsMQTT. Can you post one before it got messed up and the messed-up one. There should also be a startuptrace.txt file there. Post it as well. What was your prior version of the plugin?

[Slab0]

The database of associations is backed up so you are able to return to an earlier setup by renaming the database file that is at \Data\mcsMQTT. Can you post one before it got messed up and the messed-up one. There should also be a startuptrace.txt file there. Post it as well. What was your prior version of the plugin?

The attached archive contains the current db and the two prior backups. I'm not finding a startuptrace.txt file. The prior version was the last available on the updater prior to updating this morning... apologies, I don't recall what the specific version was.

I should mention that I did the association using the automatic feature and then editing the default topic templates. I found it necessary to change the topics a couple of times before is stuck (it always wanted to revert back to the default template). I'm guessing I should just be doing the associations manually. Not sure if that has anything to do with the duplicates I ended up with.

As I think about it, I suspect the broker (mosquitto in my case) doesn't re-publish a topic if it's locally stored value is already equal to an incoming publish request. If that is indeed the case, there should be no danger of an endless loop scenario as discussed previously, and thus the only value of my change request is possibly saving a minuscule bit of traffic between mcsMQTT and the broker (the redundant publish in step 4 of my prior post when sub and pub topics are the same). I should have confirmed that with the prior version.

[Michael McSharry]

Observations from databases
1. Device 87 house/downstairs/living_room/lr_light is in DB 11 times in old one and 8 times in new one. Some are tagged as plugin devices and some are tagged as non-plugin devices.

2. Same topic house/downstairs/living_room/lr_light shows up as a subscribed 15 time and rejected in new database and not in old database

3. In the old database there were no topics published except win7pro-VM/mcsMQTT/Unknown which was associated with device 86

4. In the new database homeseer/house/security_system was setup as both subscribe and publish on device 20 and homeseer/outside/entrance_light on device 89. It showed up multiple times.

Multiple database entries should not occur so I need to look into how that could happen.

With the old databae 6/17 version I do not understand what was showing up as something published after message received. Which specific topic was giving this behavior?

To better help me run down the root cause can you describe how you are trying to make the associations? The multiple tries I especially dont understand.

You indicated that you have duplicate devices. Can you give me an example of two. Look at the HS Device Management page where Ref property is shown as each device needs to have a unique ref.

[Slab0]

Observations from databases
1. Device 87 house/downstairs/living_room/lr_light is in DB 11 times in old one and 8 times in new one. Some are tagged as plugin devices and some are tagged as non-plugin devices.

2. Same topic house/downstairs/living_room/lr_light shows up as a subscribed 15 time and rejected in new database and not in old database

3. In the old database there were no topics published except win7pro-VM/mcsMQTT/Unknown which was associated with device 86

4. In the new database homeseer/house/security_system was setup as both subscribe and publish on device 20 and homeseer/outside/entrance_light on device 89. It showed up multiple times.

Per the second picture attached in my prior post, all of my devices are setup to subscribe and publish, both on the same topic.

As for the other anomalies, I suspect there is an issue with how I was adding the associations in mcsMQTT. See below.

To better help me run down the root cause can you describe how you are trying to make the associations? The multiple tries I especially dont understand.

1) On the external server, I published on each of the topics I wished to associate an HS device with.

2) They were then discovered and listed under "Association Table for Auto Association of MQTT Topic and HS Device" on the Association tab in mcsMQTT.

3) I selected one from the list by entering the device ID (HS/z-wave device) in the Ref field, which produced 3 editable fields to enter subscribe topic (initially blank), publish topic (pre-filled with a $$xxxx template), and payload template (blank I believe).

4) I filled in the blank sub topic, clicked on the pre-filled pub topic and changed it to match the sub topic. I left the payload field blank.

5) When I refreshed the page by clicking "Show selected associations" OR by clicking on the HS 'PLUG-INS->mcsMQTT->MQTT Setup' menu item to completely refresh, the publish field in the association reverts back to the default $$xxxx template. I had to do it again, or go to the 'Edit/Add' tab and change it there. I believe I had to do this for each device association.

You indicated that you have duplicate devices. Can you give me an example of two. Look at the HS Device Management page where Ref property is shown as each device needs to have a unique ref.

Perhaps duplicate isn't the correct term, but the first picture in my prior post shows what I got immediately after the upgrade. Each device shows up twice... first all in pink, and then again all in green. As per that first picture, none of the devices were correctly configured anymore. I removed the associations (by unchecking the 'A' box), and re-associated so that everything was back as I wanted as per the second picture in my prior post (again, following the procedure outlined above). My guess is that all this activity is why the database has the anomalies that it does.

In the mean time, would it be at all possible to post another 'update' to the updater that effectively is the last version that was on the updater prior to the update I did this AM (the version prior to 3.4.3.0)? Would like to keep my project moving forward ...thanks much (and let me know if you need anything else...happy to help/test/etc...)!

[Michael McSharry]

Your methodology is consistent with how it was designed to operate. The debug file is \Data\mcsMQTT\mcsMQTT Debug.txt. It may have some clues as to why the database has duplicates and why edits are not sticking.

[Slab0]

Your methodology is consistent with how it was designed to operate. The debug file is \Data\mcsMQTT\mcsMQTT Debug.txt. It may have some clues as to why the database has duplicates and why edits are not sticking.

Debug file attached. Note that I turned debug on/off for brief periods throughout initial testing, so not sure how useful this is going to be. Thx.