Does mcsMQTT have a speed restriction?

[billt]

I'm experimenting with getting values from my Emoncms servers into Homeseer using MQTT rather than the scripts that I've been using so far.

At first it seemed to work OK, but I noticed that there was a difference between the values the script was giving and those that mcsMQTT was producing. Looking at the statistics the current queue depth is pretty large and increasing. This implies to me that mcsMQTT is getting overwhelmed by messages and isn't processing them in time. The MQTT messages are very late which would account for the discrepancy between script values and mcsMQTT values.

If there is a limitation it looks as if I'll have to stick to my scripts.

[Michael McSharry]

It does show 74 milliseconds on average between MQTT messages. This seems like something is updating much more frequently than is practical. I suspect your script is not pulling data at 74 millisecond intervals. Do you have a means to throttle the publish rate on your Emoncms servers?

Another thing to consider is if you are a recursive message where mcsMQTT is setup to publish when it receives a topic and the same topic is used on subscription or Emoncms server is publishing upon receipt of what is published by mcsMQTT.

[billt]

At the moment there are 36 values and they are updated every 5 seconds, so that would imply about 140mS per message, which doesn't seem an unreasonably high rate to me.

The script runs every 52 seconds to collect 34 values,1500mS per value. For a laugh I changed it to run every 5 seconds and it seemed to work OK. I've got another 5 scripts that gets 136 values from EDS 1-wire interfaces, tried changing the polling interval for those scripts to 10 seconds and they worked fine; 74ms per value.

I was hoping to change the system to gather those 136 values via MQTT thinking it was a fast, low resource use interface, but, disappointingly, it looks as if it's less practical than the scripts.

I found a way to slow down the mqtt message rate, but it had to be slowed to a message every 30 seconds before mcsMQTT could handle the rate.

[Michael McSharry]

30 seconds is excessive. Your stat is showing 200 milliseconds per message and this is consistent with my RPi message processing time. My system with probably around 20 MQTT nodes is averaging 8 seconds between messages. Most are JSON with probably 10 keys each.

mcsMQTT needs to provide a general solution of assessing events, decoding JSON, and bookkeeping. This constitutes much more than just the handling of MQTT protocol. A script dedicated to a specific function of only storing data in HS devices will be more resource efficient.

[Michael McSharry]

Thinking about this a little I realize it would be very easy to add an express mode of operation where identified topics would bypass all the overhead and simply put the payload in HS device. I should be able to make the updates tomorrow. If you are interested then it would be nice to evaluate in your environment.

[Michael McSharry]

I implemented the express mode. I tested on RPi with an external test program spitting out topics with changing payloads. I could run the the publish loop with the test program down to about 33 milliseconds per topic. If I go to 30 then the receive queue starts to backlog. The receive processing statistic was under 10 milliseconds. This implies that other processes are consuming resources that limit mcsMQTT on my RPi from accepting faster than this rate.

Let me know if you want to evaluate.

[billt]

Thanks for thinking about this.

I first tried 10 second intervals but the queue was still increasing. Just jumped to 30 seconds to check if it would cope with that rate, which it did. I didn't reduce the interval to check the limit - there are a lot of changes to make.

Yes, I would like to try the modified version, thanks.

[Michael McSharry]

The file is at http://mcsSprinklers.com/HSPI_MCSMQTT_3500.zip.

I am in the middle of adding support for changing Topic to Device relationships and have repurposed the Reject checkbox on the Association tab to be the Express checkbox. A topic that is identified as Express will only store numeric payloads in DeviceValue or non-numeric ones in DeviceString. It will be included in the timing statistics. It will not be included in any of the other features of the plugin such as history, mcsMQTT events, JSON expansion, refresh of payloads and update times on mcsMQTT pages, etc.

In your case I would expect most, if not all, the E column checkboxes to be checked which will result in the maximum efficiency of mcsMQTT in providing the service of bridging MQTT to HS Device.

[billt]

Thanks for looking at this.
I've tried this version and it is a lot faster; it can't quite keep up with the native message rate, but it manages if I put a 12 second delay in a few of the messages. As the sendin rate is much higher than necessary anyway, I can add dleays and the plugin will be able to cope with the extra feeds that I will add.

[Michael McSharry]

You can also play with the receive queue depth and sleep times to better characterize your environment. Of course an overall average message rate that is greater than the processing power available needs to be avoided.

I have now submitted version 3.5.2.0 to the updater (and http://mcsSprinklers.com/mcsMQTT_3_5_2_0.zip) and it has a little difference in the capability that the test version that I suspect you used for your evaluation. In particular the Express checkbox is now its own column and the Reject one is restored. It means that everything you identified as Express will be treated as if you had selected Reject. I did not think the Reject was being used by others, but found out that there were uses so I needed to restore the prior capability.

The General tab also has a new setting to use Express mode as the default rather than Full support when a device is associated with a topic. This will save time in clicking the E checkbox if E is what you normally want.