High CPU Usage (60 - 98%) with HSPI_MCSMQTT

[OmegaQuest]

I recently installed mcsMQTT on my HS4 Windows 10 System. Everything seems to be running fine with mcsMQTT except for the CPU is very high. I have Restarted the Plugin and the system CPU drops to under 7%. 10 minutes to 15 minutes later, the CPU Spikes up again and stays that way for Hours (until I restart Plugin again).

I searched through the Forums and I selected he following settings (to see if that helped, and it dosent):

• Listen for Only Associated and MQTT Trigger Topics
• Enumerate HS Devices only with Button
• Save history Of only devices marked With L or S column checkbox

I currently only have about 15 total devices set to Log to InfluxDB, nothing more.

I have attached a Debug Log (Redacted).

[Michael McSharry]

The debug shows little MQTT-related activity. It appears the internal MQTT broker is being used. As a diagnostic change the IP of the broker to something other than 127.0.0.1. It does not matter what IP is used. It is only to prevent the internal broker from being started. There are others that use a VM and the internal broker causes high CPU in their cases. If the CPU use goes down after the diagnostic then the solution is to install a MQTT broker somewhere on the LAN such as Mosquitto. If it does not help then a profile will need to be collected to see which code segment is causing the CPU use.

[OmegaQuest]

I just set it to a fake IP and it now says Broker is offline. I will let it sit for a while and see if the CPU Spikes up again. So basically if this works, the issue is with the MQTT Broker built into mcsMQTT right? And I would need to install Masquitto on my system and use that instead to get stats sent to InfluxDB?

[Michael McSharry]

The only known cases of high CPU are from those who use both internal broker and VM. I am not aware of anybody who has issues with internal broker running on actual rather than virtual machine. If you are using VM then an external broker will need to be used. Otherwise, if you desire, you can run profiling diagnostics to isolate the offending code. Post #31 mcsMQTT using 100% CPU forever - HomeSeer Message Board is one case where the profiling is described.

[OmegaQuest]

I am running my HS4 on a dedicated HP EliteDesk 800 G2 Workstation with the following specs:

Windows 10 Enterprise
Intel i5-6500T (2.5 GHz)
8 GB Ram
125 GB SSD Hard Drive

I will take a look at that Profiling Post

[OmegaQuest]

I have the diagsession. What do I do with it now?

[Jackpost]

I have the exact same behaviour and also not running in VM. Disabeling and enabeling the plugin always brings CPU usage back to normal, unfortunately the high CPU% is always noted when HS no longer responds....

Windows 10 Pro
Intel i7
32 GB Ram
200 GB SSD Hard Drive

I am using Influxdb as well.

[Michael McSharry]

I have the diagsession. What do I do with it now?

zip it up, post it or attach with Private Message or email to mcsSolutions at CenturyTel dot net.

[OmegaQuest]

k, I'm making a new session from a Fresh Restart of MQTT and letting it run until the CPU Spikes. This way it will gather the root cause of the spike and sustained high CPU. Then I will upload it to you.

[OmegaQuest]

I just emailed it to since the size is over the limit for Private Messages.

Thanks

[Michael McSharry]

The profile shows the high utilization is in code outside of mcsMQTT, but invoked by mcsMQTT. Visual Studio tried to load ntsokrnl.exe symbols without success. This implies that the code trace includes it. There is a google suggestion for overcoming it, but these use cases may be different. ntoskrnl.exe High CPU or Disk Usage on Windows 10 [Solved] - Driver Easy

I did not notice the results of the testing that excludes the internal MQTT Broker. Does CPU peg when a fake IP for the Broker is used?

If I had to guess the issue is likely some type of socket-level semaphore that is being used and the event to clear the semaphore is not being raised so the waiting thread just burns CPU cycles.

For those that are using mcsMQTT as the InfluxDB conduit and do not use MQTT traffic then an easy solution is to use a fake IP for the Broker and then use the disconnect from broker checkbox on the General tab. This will have the effect of disabling MQTT and not producing error messages as it tries to connect to the fake broker.

[OmegaQuest]

The CPU dosent peak when the broker is set to a fake IP/Disabled. Only when its enabled/running.

So if the MQTT Broker is not enabled/running, will mcsMQTT still log and send values to InfluxDB? I thought the broker needed to be running for mscMQTT to see/send values to it from other systems and then to InfluxDB. Am I not correct with that assumption?

[Michael McSharry]

So the conclusion is that the library used for internal broker has a failure point for both real and VM systems. For those that use MQTT traffic then an external broker such as Mosquitto needs to be installed if the internal broker experiences this failure mode. It has been several years since the library used for the internal broker has been touched so I do not think it is being maintained. HomeAssistant also has an internal MQTT broker. I never looked at what MQTT library HA is using.

InfluxDB is populated from HS Device changes. It has no relationship to MQTT traffic. Of course if the HS device is being updated by a received MQTT message then there is a secondary relationship. mcsMQTT communicates directly with the InfluxDB server using a HTTP protocol. Flux is used InfluxDB 2. SQL is used for InfluxDB 1.

[rumb]

I have been experincing this issue also every couple of days. I run on the pi box.

So is only solution to use mosquitto?

[Michael McSharry]

The best solution is to run Mosquitto. It is a very easy install on Pi, runs with little resources and no maintenance. It is a mature solution that will not cause you any problems.

I am not going to try to update the MQTT Broker library published on GitHub because the failure mode is subtle. While it is open source, I have not been involved in design and it would be too difficult to get into the minds of the designers. The library on GitHub has had one update to my knowledge and has been inactive for few years. Mosquitto is acvitely supported and has been updated as the MQTT spec has changed.