Dramatic reduction in CPU utilization with Express Mode

[Michael McSharry]

I have benchmarked mcsMQTT 3.5.4.0 to assess the CPU utilization in the various forms of Express mode vs. Full support. The test was done a RPi model 3. No other plugins were running. Mosquitto was also running on same computer.

The chart below shows the reduction in CPU use by using the Express mode settings. In Express mode all history recording is bypassed. This included the history for charting and the history associated with the last payload. If a browser page is visible then the payload/lastdate information is not updated. The transformations using regular expressions, numeric expressions, rate devices and accumulation devices are not available.

Express mode is well suited for production use of MQTT payloads. Full support mode is well suited for investigation and special needs of some topics and payloads.

The selection of Express vs. Full support is done on a Topic by Topic basis so a user can customize the configuration to best balance CPU vs. features needed for each Topic.

Express mode was introduced in version 3.5. Currently 3.5.3.0 is in the updater. It does not support JSON and VSP in Express mode. Express mode is selected on the Association tab with the E column checkbox. The General tab has a setting to default to Full vs. Express when a Topic is associated with a Device.

3.5.4.0 will be released soon and available now at http://mcsSprinklers.com/mcsMQTT_3_5_4_0.zip, but does not yet have the manual updated. This provides the user-selections for including JSON decoding and VSP processing. From the table it can be seen that VSP has a trivial time penalty. JSON decoding into independent devices also has a relatively low CPU penalty. The selections to enable these two features is on the General tab.

                             Full Support          Express       Percentage Reduction
    VSP                       200 ms                 9 ms                     95%                 [ON/OFF toggle]
    Number                    200 ms                 8 ms                     96%                 [Incrementing value]
    JSON 1 Number             310 ms                13 ms                     96%                 [JSON contains 4 keys, one number accepted]
    JSON 2-numbers, VSP, Text 510 ms                32 ms                     94%                 [127 ms/device Full vs. 8 ms/device Express]

[Michael McSharry]

I continued to investigate the CPU utilization within mcsMQTT to try to understand why there was such a large difference in CPU use between Express and Full support modes.

During the analysis the most illuminating discovery is that almost all of the time for processing messages was within HS following the calls to SetDeviceValue or SetDeviceString. When removing these calls, the Express mode times were under 1 millisecond.

During Full support mode calls are made to get the HS device object (dv = hs.GetDeviceByRef) which consumed 12 milliseconds and then each call that used the hs object as a parameter (e.g. dv.Interface(hs)) used a similar time so working with the HS device object became expensive . In version 3.5.5.0 the use of the hs parameter was eliminated in favor of Nothing for the real time processing.

Each time hs.SetDeviceValue was called there was a corresponding call to hs.DeviceString to assure that the DeviceString was empty to allow DeviceStatus to be displayed. This essentially doubled the time to update a Device since most of the time was within HS. The assurance that DeviceString was empty for non-text devices was removed from the real time processing and done during initialization.

These steps resulted in a significant reduction in the CPU utilization for Full support mode. The chart below was taken from the mcsMQTT manual that documents the current capability. Available at http://mcsSprinklers.com/mcsMQTT_3_5_5_0.zip and submitted to Updater.

[Guest]

I continued to investigate the CPU utilization within mcsMQTT to try to understand why there was such a large difference in CPU use between Express and Full support modes.

During the analysis the most illuminating discovery is that almost all of the time for processing messages was within HS following the calls to SetDeviceValue or SetDeviceString. When removing these calls, the Express mode times were under 1 millisecond.

During Full support mode calls are made to get the HS device object (dv = hs.GetDeviceByRef) which consumed 12 milliseconds and then each call that used the hs object as a parameter (e.g. dv.Interface(hs)) used a similar time so working with the HS device object became expensive . In version 3.5.5.0 the use of the hs parameter was eliminated in favor of Nothing for the real time processing.

Each time hs.SetDeviceValue was called there was a corresponding call to hs.DeviceString to assure that the DeviceString was empty to allow DeviceStatus to be displayed. This essentially doubled the time to update a Device since most of the time was within HS. The assurance that DeviceString was empty for non-text devices was removed from the real time processing and done during initialization.

These steps resulted in a significant reduction in the CPU utilization for Full support mode. The chart below was taken from the mcsMQTT manual that documents the current capability. Available at http://mcsSprinklers.com/mcsMQTT_3_5_5_0.zip and submitted to Updater.

Those are crazy numbers rjh

[AshaiRey]

Good detective work.!!
Looking forward for the update to come.
Thx for you hard work.

[bartg]

I upgrade to the lasted version this weekend.
This went without any problem. I have not changed anything in my configuration.
Not only has the CPU load dropped but the memory leak with the older version I had seems to be gone now.
Many thanks for this great plugin.