Version 3.2.x.x

[Michael McSharry]

This version continues to provide an improved user experience and from a code perspective a significant change in the means to manage the non-plugin and plugin associations with MQTT topics. The dynamic update of fields on the Associations, Manual Pub and Manual Sub have much logic and room for some situations to have not been considered. Let me know if any inconsistencies have been observed.

The database has evolved through the versions and assumptions made before are no longer valid for current design of the plugin. A one-time reconstruction of the database is done on the initial run of V3.2.x.x. This may make the startup appear slow, but it is only once.

The Association table is now only rendered on a button push. This is done to improve page load times. Color coding of the rows has been done to distinguish plugin vs. non-plugin MQTT relationships.

Some discussion on CAPI and non-plugin device control via MQTT is contained at https://forums.homeseer.com/showthread.php?t=194817

The change log at the top of the sticky on this forum discloses other updates that were made
https://forums.homeseer.com/showthread.php?t=192675[/URL]

[petez69]

Hi Michael

I upgraded this latest version from 3.1.1.0 and it appears I've lost all of my associations ? I'm not able to control any of my mqtt devices nor receive updates from them.....all disappeared ?

Under filter associations I've ticked show all accepted only and click show selected associations, nothing shows. I've ticked all the filter boxes, nothing shows.

I'm now looking at the statistics page, shows MQTT received and accepted by mcsmqtt is 0, mqtt received and not accepted my mcsmqtt 3532.

As I had previously accepted messages on the earlier version of the plugin, how do I make sure this new version sees those "acceptances" ?

Thankyou..Pete

[petez69]

Hi Michael, I rolled back to 3.1.1.0 and all of my devices show up and are now operable....

Peter

[Michael McSharry]

\data\mcsMQTT.db is the database that contains the association information. Automatic backups are made unless the feature was turned off on the General tab. Renaming a backup to mcsMQTT.db will restore. I suspect what may be happening is the reconstruction of the database to clean it up. Can you post a zipped mcsMQTT.db so I can see what exists.

If there is no Version=3.2.0.0 line in mcsMQTT.ini then it will try to fix the database. Once the line has been inserted by this version then the database update will not be done again. To cause it to happen remove the line from the ini before the plugin is enabled.

Also with 3.2.x.x the use of PED was removed and only the database is used to retain information about the setup. This means an earlier version will go to the PED to get information while 3.2.x.x will go to the database as the primary source of information. This change was made because the information needed for the non-plugin devices could not be put in other plugin's PED so it had to go in the database.

I did not try, but I suspect that if 3.1.4.0 was reinstalled the info would be initialized from the PED as that version was looking for it there. I am not going to go back to using the PED so we need to understand why the database did not capture what you have.

[petez69]

Hi Michael, thanks for the response.

I'll get back to you in 48 hours with the requested debug, I was just glad to get it working as I did an update before bed last night and all the lighting refused to respond

Cheers..Peter

[Michael McSharry]

V 3.2.2.0 has been placed in the updater.

The primary change is to address high message reception bursts that can occur typically upon initial connection the the broker. To address this a queue has been setup to buffer the incoming messages.

There are two General Tab items and three Status Tab items related to the queue. On the Status Tab you will see the current depth, max depth and the average time it takes to process a received message. In a normal situation the current queue size will hover near 0. If one observes that the queue size is growing over time it means that the current setup is not able to handle the rate of incoming messages.

The max size has no special significance, but is information to understand the nature of message traffic being received.

The average processing time can be used to make decisions on your computers ability to support all the features provided by mcsMQTT.

The General Tab has two setting in Inbound (Subscription) Management. One is the amount of time that should be yielded for non-receive processing. This is to prevent mcsMQTT taking up all the CPU time when the receive queue is large. The second is the number of received messages that will be processed at each time after the CPU use has been regained by mcsMQTT. For example, 5 for depth and 50 for interval means that mcsMQTT will process 5 messages in the receive queue and then go to sleep for 50 milliseconds. It will continue this until the queue is empty.

Another setting in inbound management has been added that relates to processor utilization consideration. This is an option to decode JSON (default as in prior versions) or to only put JSON-encoded payloads into the HS DeviceString where the user would then need to deal with extracting information of interest.

Another prior setting that affects CPU use is the Discovery (default continues to be to enable discovery). When discovery occurs then mcsMQTT receives all messages from the broker. If only a few of a large set are of interest then the discovery can be disabled and the subscription will be to only those accepted. It is possible to enable discovery when adding new mappings into HS and after this has been done the discovery can be disabled. It may have an effect on message latency because greater burden is then being placed to pick messages to send to mcsMQTT.

As a point of reference I did testing on two computers using a two-level JSON encoded message of 8 items. This is a typical Tasmota payload. On old Windows laptop it was 300 ms average processing time. On Odroid C1 is was 70 milliseconds. When JSON decoding was turned off the time was reduced to 90 ms on the laptop. I did not evaluate this mode on Odroid.

I did my testing by flooding messages at a high rate. The queue built up a depth of well over 1000 messages and when the flooding and normal message rates started the queue depth steadily reduced to 0. In my case the General Tab queue management parameters had little effect unless I made the interval large and in that case the incoming rate was higher than could be processed.

[Michael McSharry]

Version 3.2.3.0 has modified the JSON options available on the General Tab. Previously the options were to to decode JSON into individual Devices or place the entire Payload into a DeviceString. A third option was added to both put the entire Payload in a Device and the individual JSON keys in Devices too. With this option the full Payload device becomes the parent and the JSON items are children.

At this time the selection is global. I can see where there are situations where a mixture of both would be desired. Just give me feedback.

[Michael McSharry]

V3.2.3.1 added a new row color to association table to easily identify messages that map directly into existing HS devices. Normally a MQTT message that is Accepted creates a plugin device, but with 3.2 it is possible to associate non-plugin devices to MQTT messages. This class of association is colored purple. The plugin devices have green color and the non-plugin devices have a pink color.

[petez69]

Michael

I'm getting a heap of these now on startup

HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.

Suspect the number of error messages = accepted associations.

I disabled mcsmqtt and rebooted and there were no error messages. I enabled mcqmqtt and the bunch of errors popped up....

I've tried 2 events and confirm they are not working....


Mar-29 17:37:35 Device Control Device: Sonoff Cupboard Man Cave to ON (1) by/from: CAPI Control Handler
Mar-29 17:37:21 Device Control Device: Sonoff Balcony Christmas Tree to OFF (0) by/from: CAPI Control Handler
Mar-29 17:36:54 Device Control Device: Sonoff Balcony Christmas Tree to ON (1) by/from: CAPI Control Handler

Device sonoff balcony did not toggle nor did man-cave.....

Thoughts ?

Peter

[Michael McSharry]

V3.2.4.0 added one additional setting on General Tab mcsMQTT Management. It is a means to automatically show the Association table when the mcsMQTT setup page is initially displayed. The button still exists to do subsequent displays after filter selections are made.

The text box is the max number of rows that will be displayed upon initial display of the page. It remembers the last one rendered so on the very first time the Association table will not be shown automatically, but all subsequent will honor the setting.

A default of 30 is being used. This means that if the last time the Association table was displayed had fewer than 30 rows then the next time the page is requested the Association table will be shown without pressing the "Show Selected Associations" buttons.

If the last time had more that 30 rows then the "Show Selected Associations" will need to be used to show anything in the table. The 30 is default and can be changed to 0 to never automatically display, a very large number to always display or somewhere in between. This is being provided to balance page load times and user convenience.

[Michael McSharry]

I'm getting a heap of these now on startup

HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.

Suspect the number of error messages = accepted associations.

I disabled mcsmqtt and rebooted and there were no error messages. I enabled mcqmqtt and the bunch of errors popped up....

I've tried 2 events and confirm they are not working....


Mar-29 17:37:35 Device Control Device: Sonoff Cupboard Man Cave to ON (1) by/from: CAPI Control Handler
Mar-29 17:37:21 Device Control Device: Sonoff Balcony Christmas Tree to OFF (0) by/from: CAPI Control Handler
Mar-29 17:36:54 Device Control Device: Sonoff Balcony Christmas Tree to ON (1) by/from: CAPI Control Handler

HSTouch Server is generating the error messages. I do not know what HSTouch server is doing, but my guess is that the mcsMQTT device have been created since a prior setup involving HSTouch was done. The same is also the case for any events that may have been setup with mcsMQTT device. When a database is deleted or devices not longer accepted the Device Reference will change the next time they are created. Events need to be setup again.

I need some additional insight into what your events looks like. Are you looking for incoming topics? Are you publishing based upon some device change? I added trigger monitoring to the debug in 3.2.5.1.

I also added debug around HS event processing which is where HS informs mcsMQTT that a device has changed, and around API SetIOMulti where HS commands a mcsMQTT device to change. You should have the Debug enabled from General Tab and the file will be in \data\mcsMQTT_Debug.txt. This will be useful to help isolate any problems.

[petez69]

HSTouch Server is generating the error messages. I do not know what HSTouch server is doing, but my guess is that the mcsMQTT device have been created since a prior setup involving HSTouch was done. The same is also the case for any events that may have been setup with mcsMQTT device. When a database is deleted or devices not longer accepted the Device Reference will change the next time they are created. Events need to be setup again.

I need some additional insight into what your events looks like. Are you looking for incoming topics? Are you publishing based upon some device change? I added trigger monitoring to the debug in 3.2.5.1.

I also added debug around HS event processing which is where HS informs mcsMQTT that a device has changed, and around API SetIOMulti where HS commands a mcsMQTT device to change. You should have the Debug enabled from General Tab and the file will be in \data\mcsMQTT_Debug.txt. This will be useful to help isolate any problems.

Hi Michael

HStouch is part of Homeseer and I've done absolutely nothing with it, it seems to be enabled by default and cannot be changed. As mentioned 3.1.4.0 worked absolutely fine, moving to the later 3.2.xx version all of the mqtt devices have stopped responding and I'm getting absoltuely no updates nor can I control anything.

Your debug shows:

------------------------------------------------------------------------------------
30/03/2018 12:29:21 PM 7 | mcsMQTT Running at C:\Program Files (x86)\HomeSeer HS3
30/03/2018 12:29:21 PM 12 | mcsMQTT InitHW ComputerName= homeseer-PC, IOEnabled=False
30/03/2018 12:29:21 PM 124 | mcsMQTT Debug InitHW Database Ready
30/03/2018 12:29:21 PM 151 | mcsMQTT Debug Receive Ready
30/03/2018 12:29:21 PM 152 | mcsMQTT Debug Trigger Ready
30/03/2018 12:29:21 PM 637 | Spawning MQTT Threads
30/03/2018 12:29:21 PM 638 | mcsMQTT Debug MQTT Ready
30/03/2018 12:29:21 PM 638 | HW Init Complete
30/03/2018 12:29:21 PM 639 | MQTT Thread Started with broker 192.168.11.63, Shutdown=False, Disconnect=False, Client=False, Connected=False
30/03/2018 12:29:21 PM 639 | MQTT Thread Not Connected Yet
30/03/2018 12:29:21 PM 640 | Background Init Started
30/03/2018 12:29:21 PM 642 | MQTT Thread Client Created
30/03/2018 12:29:21 PM 642 | MQTT Thread Client ID=36fcbe93-de7c-4a39-bbcb-a41b65941025
30/03/2018 12:29:22 PM 668 | Background Init Received
30/03/2018 12:29:22 PM 675 | MQTT Thread Broker Connect Response=0
30/03/2018 12:29:22 PM 675 | MQTT Broker Connection Accepted, Connected=True
30/03/2018 12:29:22 PM 748 | MQTT Thread Subscribing
30/03/2018 12:29:22 PM 750 | MQTT Subscription Start
30/03/2018 12:29:22 PM 750 | MQTT Subscription Topics being selected
30/03/2018 12:29:22 PM 750 | MQTT Subscription Topics selected
30/03/2018 12:29:22 PM 750 | MQTTSubscribe List / NoDiscovery=False to
30/03/2018 12:29:22 PM 750 | #
30/03/2018 12:29:22 PM 1155 | Background Send
30/03/2018 12:29:22 PM 1158 | Background Init Filters - Background Complete

-------------------------------------------------------------------------------------------------------------------------------------------

Homeseer debug:


Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:22 Plug-In Finished initializing plug-in mcsMQTT
Mar-30 12:29:21 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:21 HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.
Mar-30 12:29:21 mcsMQTT Version 3.2.4.0 Registered with Homeseer
Mar-30 12:29:21 Info Plugin mcsMQTT has connected. IP:127.0.0.1:49812
Mar-30 12:29:08 Info Plugin mcsMQTT with instance: has disconnected

Making a change to an mqtt device:


Mar-30 12:32:38 Device Control Device: Sonoff Balcony Christmas Tree to OFF (0) by/from: CAPI Control Handler
Mar-30 12:32:36 Device Control Device: Sonoff Balcony Christmas Tree to ON (1) by/from: CAPI Control Handler


I see NO mqtt messages being sent to the broker.....

AS mentioned, 3.1.4.x worked fine, 3.2.x.x now generates these messages and I cannot operate any mqtt devices....

Hope this is insightful, clearly I'll provide whatever data you need as I believe I cannot go back to 3.1.4 due to teh database change....

Pete

[Michael McSharry]

Do you have the debug enabled? What you are showing looks to be the default debug with the debug not enabled. Make certain you have Debug enabled from the General Tab. With 3.2.5.1 it should be pretty chatty in the debug file. I see you are still back at 3.2.4.0.

You can also try to disconnect from broker from the General Tab and then reconnect. Likewise you can change the subscription from discovery to explicit accepted devices. The statistics tab provides information about last messages and timing associated with their processing.

You can update to 3.2.5.1, run for some time that you expect to see something happen, post your config\mcsMQTT.ini, data\mcsMQTT.db and data\mcsMQTT_Debug.txt

[petez69]

Michael

Hmm, the thing seems to be in a right mess. The previous devices that were created through the associations are still on the HS web page however in the plugin there are NO associations.

The plugin is seeing mqtt traffic and wanting to create new devices for those strings at a new device value....

Darn this is a total world of hurt if the devices are lost as there are a huge number of events around these older device ids....

Can I back this thing out ? Appears I have backed up databases over the previous days that were working with 3.1.4.0

[petez69]

OK, I put 3.1.4.0 back on and restored a database from a week ago and all my devices are back and I can control the ones I've tried.

When I upgrade to the latest version the associations are not being transferred. On the new version there are NO A's against anything....

I'm seeing some errors in the logs now, not sure why but its to do with one of the lights I have.....

Error Calling SetIOMulti in plugin mcsMQTT:Conversion from string "False" to type 'Byte' is not valid.

I'm now seeing a few of these still, its impossible to know what devices its refering to..

HSTouch Server Warning Exception on Value Change callback: Object reference not set to an instance of an object.

I can e-mail you the files you wanted, I have a bunch of backed up database files using version 3.1.4.0, yoy could use these to see what happens when you apply the latest verison of the plugin to them....

Pete