Receive payloads extremely slow or missed.

[Kdiamond]

Hi Michael.

Out of the sudden incoming payloads are missed or being extremely delayed. MQTT Messages Published seem to be instant, but received payloads takes long time. Even a few minutes for status update. Could it be a broker overload? I have about 60 Shelly MQTT devices by now.

What could cause this? Where can I start troubleshooting it?

Thank you!

Br,
Dali

[Kdiamond]

Ahh, Current Queue Depth was over 8000. Of course the system halted. It explains why system is getting less responsive with every device I added.

I changed the settings a bit to process in received messages faster.



Please let me know if it's ok.

It only took me 7 hours to figure it out.

[Michael McSharry]

In some of the more recent plugin updates there was a setting to put a cap on the depth of the queue. Much like a router that will discard TCP packets when it gets backlogged. Of course it means that those messages are lost, but at least it will not grow to the point of never getting processed. The real issue is having the backlog in the first place. Devices typically do not report so often that mcsMQTT/HS cannot keep up.

Something to look out for is excessive use of Retain. The broker needs to keep all these messages and then when a client comes online it will flood the client with these.

There is a performance section in the mcsMQTT manual to give you some ideas of what the timing drivers are and provisions made to help deal with larger volume of messages.

I would look at the message traffic to see what is being sent that is clogging things up.

[Kdiamond]

Thank you for reply Michael.

Devices typically do not report so often that mcsMQTT/HS cannot keep up.

Well I have about 60 Shelly devices that reports by their default settings. So nothing special here I guess.

Of course it means that those messages are lost, but at least it will not grow to the point of never getting processed.

I don't thing that's a good solution. No packet should ever get lost.

There is a performance section in the mcsMQTT manual to give you some ideas of what the timing drivers are and provisions made to help deal with larger volume of messages.I would look at the message traffic to see what is being sent that is clogging things up.

Will do that later today for sure.
With current settings: Receive queue depth: 50, Receive queue interval: 10 Current Queue Depth will jump go up to 20 but will always go back to 0.

What sets the performance limit of the broker? CPU? I now have a cheap 100€ Beelink PC bought from Aliexpress. It runs Windows 10. Beside the HS there is a Torrent client service and PC is being used as storage server as well. Average CPU load is about 25%. So having an latest CPU with 50 times the performance. Those 2 PCs can not have the same broker limits I guess?

About my 7 hours quest to find the problem that could be 5 minutes job. It's my fault without a doubt, but there could be a better way to alert the user.

The absolutely first thing I tried was to alter the Receive queue depth and Receive queue interval in the General tab. The obvious thing. Since the queue was full, changes were not affective immediately as the queue was over 8000 (Max Queue Depth was on Statistics tab) I didn't see the issue and went searching elsewhere.

Overloaded broker means a unresponsive system. Quite big issue, certainly not a thing to ignore. I would suggest the following for you to consider.

1. Clear queue button, for troubleshooting only.
2. Big red shouting alert in Statistics tab saying. You have a problem, broker overloaded, Queue are raising or something.
3. Alert event so we could set and Event to send an email when this happens.

Thank you
Br,
Dali

[Michael McSharry]

You currently have the ability to create HS devices for those items on the Statistics page. You can then setup an event for some queue depth threshold for notification. There are various situations that may occur, such as lost connection with broker, that other events can be created when you start to monitor the MQTT interface. The reason for the Statistics is to help with analysis. It is usually the case when looking for something that it is found in the last place you look. Once you found the issue you are now sensitive to this one situation. The next one will be something different.

I doubt if you have a broker that cannot keep up. It is more likely you have a rogue client that spamming. You need to look at the messages to get an idea of what is happening in your specific case. Tools such as MQTT Explorer are good for independent analysis of MQTT traffic.

For troubleshooting you have the disconnect from broker checkbox on General tab and can restart the plugin to clear the queue.

[Kdiamond]

Thank you for reply Michael.

MQTT Explorer is great and simple! Thank you for suggestion.

By the way. You do this for free? Your plugin does much more than any plugin I paid for. Do you accept any donations?

Br,
Dali

[Michael McSharry]

No need for donation. Knowing that you appreciate is sufficient.