Bug report and feature request

[tfoutfou]

Hi mister , after a few days playing with the plugin and going deep in the rabbit hole , i found different issue and problems
Here is what i found that might be interesting to look at

(i am french so i would do my best to explain as best as i can , dont hesitate to ask back if something is not clear)


FACT : my Mqtt payload are always number with a "." in them , i.e. (float) , never more than 3 digit after the point , and i see them perfectly in the Associations tab
FACT : later on i switched to JSON payload to save bandwidth and CPU , payload always look like this {"V":230.2,"Hz"50.0,"A":0.056,"W":6.8,"Wh":507,"Pf":0.52}
FACT : Express mode is never enabled because i have been having some problem with it
FACT : all is running on local machine on windows 7 64


first let talk about my use case , updating a bunch of HS devices with value from power meters that are connected to an arduino then processed as Mqtt payload
for Voltage , Current , Power Factor and frequency , i only want to show them on individual Devices then no processing
for Watt and Watt Hour , i would like to be able to use the numbers in event or something like that (not sure yet)
So , "DeviceString" update is OK for the first part and then "DeviceValue" update for the rest to be able to use Events.
i plan to use it more in the future , but this kind of thing take time



BUG 1 : Whatever the setting for Express Mode is in the General tab.
When you clic on the "Ref' button that is automatically created in the Association tab , the Express Mode is automatically switched to whatever the mode is selected
in the General Tab
the "e" column change , but the device seem to behave like it is still in the previous Express Mode , a refresh of the Association Tab dont bring back the mark in the "e" box
the same behavior is observed if you access the device config via the Edit/Add tab

BUG 2 : if you delete a Device/Topic , the topic is not rediscovered (forget this one , i have to investigate , it didn't do it anymore)
BUG 3 : removing topic dont work on my system randomly , i may have to restart to plugin for it to work (and yes the broker is offline so the topic are not appearing by them self)
( i am aware of the command to remove full branch of JSON topic "topic#")
BUG 4 : removed

BUG 5 : if "Create both Parent full payload and Child JSON keys" is selected , the payload never update on the Association tab for the "parent" device
while it is updating on the HS device list ( refer to BUG 7)

BUG 6 : if "Decode Payload JSON into individual HS Devices" is selected then i understand the HS Parent device will not be created Nor updated
but the Topic in association Tab must still be updated for controlling if the payload is indeed delivered to the plugin
(except if express mode is selected for the unused parent then the lastdate and payload must not be updated , right ?)

BUG 7 : the "parent" homeseer device automatically created by JSON option is not the one that we can see in the association page
the "true parent" dont become a HS device and is always staying in the Association tab
the "false parent" (the one created by the plugin) is not listed in the plugin anywhere , but it can still be updated by the plugin because
we can see the JSON payload in HS Device list
since it is a plugin device and for consistency the "true" and "false" parent should be a unique device dont you think ?

Additionally the configuration for the "false" parent is not possible because it is not listed in the plugin tabs , so what is happening in the background ?

Looking at the Advanced Tab in HS device page , i can see a MAJOR issue there , because the "Value" is growing by 1 every time a payload is received
is it creating and storing VSP in the database every time ?? or is it just setting the value to +1 every time ?

BUG 8 : linked to bug 7
if Auto parent is created when adding a JSON device , and there is only one device associated with it , when you remove the device , the parent is still in the
HS device list but what surprise me is that the parent is not updated anymore !! why ?
what kind of link you have between devices ? you should really join the "false" and "true" parent on the same device for readability and simplicity

BUG 9 : This one i pretty strange , and it took quite a while to reproduce it ( NOT REALLY A BUG SINCE A FIND A WORKAROUND)
ONLY If in Express mode for a device ( a normal device with a normal topic , i didnt test this with JSON device)
AND IF "Number" is selected in the Control/Status UI , THEN if the Number contain a float
IT goes in the String of the device , while it is supposed to go in the device Value
( my float contain a point "." , the bug dont appear if i replace it with a "," BUT if not in Express mode the data is parsed as a Value if "." or "," , Why ?)


FEATURE Request : a better way to remove obsolete topic , with a Button on the Association tab with a button for each maybe ?

FEATURE Request : JSON decoding must be a choice per Topic (maybe not that important , because if you receive JSON you have to parse it somewhere anyway)

FEATURE Request : show statistic in HS device , can you add a checkbox , to populate or not the corresponding devices in Homeseer , and a checkbox to create or delete the devices , this is because it delete the devices every time and it may be nice to dont do that

FEATURE Request : a button in the device edit tab to remove all the VSP at once for a particular device

FEATURE Req / QUESTION : a button in the device edit tab to NOT create the VSP automatically
My problem is that i have a payload that can fluctuate quite a lot , and i get an infinity of VSP that i will never need
if i check "Unspecified" it switch back on its own to "Text"
Is the "Text" button doing just that ? i dont know because i still see random VSP popping in the list under


QUESTION:
i have a hard time finding detailed information about the other button in the manual as weel ?
can you tell more about colorpicker and colorXY , if activated i dont see color control popping in HS for the selected device
i also have a hard time understanding the button behavior , "number" is clear , but the other one seem confusing too
can you help me please ?


Thank you very much for reading my long post

Hope you will understand everything and be able to help

Bye

[tfoutfou]

after reading a few more pages in the documentation there is possibly a solution for a few things there


what a nice work you did there , amazing how many pages you wrote , never seen a plugin Pdf this big

great job

[Michael McSharry]

#1 I understand and will try to replicate
#3 Can you expand on process used to remove topic. A specific example would help
#5 I will revisit this based upon #7
#6 I do not understand expectation in this case. If parent device is not created then no full payload updates will be done by design. Only the individual JSON items. Please illustate your expectation.
#7 My intent with parent devices is to support the Associated Devices property in HS where a set of related items is always displayed together. I never intended that the Association tab be used to select parent vs. child devices so no need for a parent topic to be shown. Am I missing your point?
#8 I understand and will look into it. If no children exist then the parent should be deleted.\
#9 I understand and will investigate
F1 I understand
F2 I will leave it as-is. It becomes confusing to let the user select decode vs. not decode on a topic by topic basis. The discovery aspect where JSON is decoded before any user action whould result in an awkward user experience.
F3 This already exists as option on General tab
F4 I understand
F5 VSP discovery is capped at 20 (if I recall). Use of Control UI of Text will keep the discovery behind the scenes. It is done so a user can select List at some time and have VSPs created automatically.
Q1 I do not undertand the "other" button. Where does it show up?
Q2 Colorpicker is used for devices that expect a #RGB control and they report status in a similiar #RGB manner. The color control should only appear if a pub topic is entered for the device.
Q3 ColorXY is used for devices that use XY rather than RGB for color. I have only one use case for it with Sengled zigbee bulb. If you are need more I can provide more specifics.
Q4 Button and List use VSP to associate numbers with text payloads. The numbers go into DeviceValue and HS reports DeviceStatus with the text. Button generates indivisual buttons on the Device Management UI. List generates a pulldown.
Q4 Text goes into DeviceString so not very useful for event triggers

[tfoutfou]

#1 ok waiting feedback (if Default mode is Express , the checkbox will also check itself when accessing Edit tab or clicking "Ref")

#3 simply by going to the General tab and inputting the Topic name , then clicking on submit or enter , the topic isnt deleted from the Associations tab (topic not associated with any device) , some kind of database lock maybe ?? but not really possible since the plugin dont throw any errors ... ? Not a big issue since a simple plugin reboot fix the issue (will try more and find a way to reproduce the bug)
(trying this right now on non associated devices and it dont work)

#5 it is my OCD kicking back , the parent topic was shown in the list , but is not updated anymore because the plugin is doing its magic in the background when JSON is enabled .
i didnt know until a few moments ago that when JSON processing is enabled the parent topic is discarded from the list
but if you dont remove it manually or check "r" it stay there forever

#6 still my OCD lol . i wanted to know if the plugin is "optimized" for JSON processing
to be more precise , is it better to have a longer payload then divide it with JSON or is it better to have more small individual topic ?
I would say JSON because processing a single Mqtt packet seem more efficient , but hey just my felling !

#7 as in #5 the parent Topic is discarded from the list , so no "true" parent anymore.
i was more concerned about the DeviceValue growing by 1 at every payload ?!
is it ONLY updating the value with a function like hs.SetDeviceValueEx , or is it doing something else in the background (like updating a database with the full payload ) ?

#8 ok waiting feedback

#9 ok , dont waste your time on this , maybe we can fix this with "payload RegEx match/replace pattern" on our own

F1 ok nice

F2 agree with you pretty useless to have topic by topic for decoding , but parenting may be an issue
EDIT : or a Non-issue , since we can break auto-parenting afterward , or set to "Json into individual" before selecting devices

F3 let me explain better , i would like to be able to keep my Statistic devices in HS , but to save CPU i would like a checkbox to disable update of said devices . (if you disable and re-enable Statistic device it keep deleting and recreating them , pretty annoying)
Maybe CPU time is ridiculous to do such a task ? do you know ? if so this request is USELESS

F4 ok

F5 ok i see your point there , but if "text" is selected , to save ressources i really dont want some kind of magic in the background creating VSP and filling database

i have 20+ power sensors sending Mqtt every 3 to 5 seconds , it would be nice if i can send the fully formatted string to put directly in a HS DeviceString without being forced to run a script on HS side , because currently i use "number" then run a script to fill DeviceString.

Q1 answered in Q4

Q2 Q3 ok nice , i can implement that with my arduinos already controlling RGB but each color is a different channel on its own actually.
can you tell me more ?
what is the formatting of #RGB ? #128,212,3 ? for values going from 0 to 255 ?
what is the formatting of XY ?

Q4 ok nice





#10 NEW BUG / QUESTION

Express mode on JSON devices dont work , if i enable "express" on one device , the full JSON payload is putted as a string in the said device , and the associated parent and remaining child device stop updating at all.
If any child device is set to Express mode , we can NOT set any other one to express mode , we get this error message

"Topic PZEM04:A has already been accepted for Express mode. PZEM04:A must not be selected for Express mode to be able to accept PZEM04:Pf"

however are there any benefit at all to use express mode on JSON devices ? are they processed differently than direct Topic devices or is it all the same after they are created as "fake" Topic ?

[tfoutfou]

#3 retrying this right now on an random topic created for testing last night

JSON payload with 4 child , topic name "PZEM000" , subtopic name"V1" "V2" "Hz1" "Hz2" , only 2 associated child's
going in general , inputting "PZEM000#"

topic update has been disabled from the other side of the chain

result : associated child's are deleted , unused child's remain in the association tab

enabling topic update , deleted child's appear again in Association tab

rebooting plugin via HS management tab

disabling topic update

exactly the same steps

Result : all child's deleted

enabling topic update again , child's appear again in Association tab

result : impossible to delete those child's or any other topic again

if i associate a child to an HS device , then try to delete the topic , it work again only for the associated child

[Michael McSharry]

#1 fixed in http://mcsSprinklers.com/mcsMQTT_3_4_6_1.zip

#3 fixed. Note syntax needs to be MQTT-compliant for wildcard.

#7/#5 The Association tab does not show any topics that are associated with a HS parent device. It only shows the individual JSON items or non-JSON item.
I could add a row for parent and update the full parent payload on this tab, but It has downsides beyond just the CPU use.
If I added checkbox controls for parent then it could be confusing to know if a parent is unassociated from HS then will all children devices be deleted.
If the checkbox action was to only remove the parent then the children would no longer be associated to any parent.
If a user selects only a parent association checkbox then does this mean that only the full JSON payload goes into a HS device or does it mean that all children also get associated with HS device and then need to uncheck the children not desired.
if no checkbox controls were shown for parent then one needs to deal with how to sort when both parent and children rows are present. For example, sort by payload would result in parent and child no longer being in contiguous rows.
My thinking is the way to deal with these things is to make the Association row represent only potential child HS devices so the operation of every row is consistent
Can you explain the following. I do not understand. "i was more concerned about the DeviceValue growing by 1 at every payload". When a payload is received mcsMQTT updates its internal dictionary. If HS device is associated then the deviceValue or deviceString is updated. At shutdown the internal dictionary is copied to the mcsMQTT database. The raw (full) topic/payload goes into the History database on every message reception if the topic satisfies the history collection criteria. This is the database from which charts are generated.

#6 my benchmarking showed that decoding JSON was very small CPU use so your assumption is likley correct that one message with JSON encoding will be overall more efficient that multiple without JSON. This is especially true when considering the broker's involvement too.

#8 I have a Topic x/RESULT with JSON POWER1 and POWER2. Two rows exist on Association tab. I Associate POWER2 to HS. On HS page device 1068 and 1069 were created. 1068 is the parent. I unchecked the Association on Association tab for POWER2/1069. HS devices 1068 and 1069 removed from Device Management page. Can you explain your steps where the parent device remains in HS?

#9 I believe what you are seeing is the differences in regional use of the period and comma. Your OS, running in France, sees 1.234 as either number 1234 or text "1.234". You MQTT node is publishing "1.234" with intent of it being a number. Since this node is not compliant with France regional use of the period/comma you should use a regular expression in mcsMQTT to change the period to a comma for this device. There is example in the manual.

F3 my benchmarking discovered that doing anything with the HS devices is very expensive with CPU vs other activities within mcsMQTT. I will look into providing on/off control in the Statistics parent HS device.

F5 The background collection of VSP information occurs for Unspecified, Button and List types. If you define it for Text then no overhead and you can publish your HTML-formated text that will go directly into DeviceString

Q3 The RGB fromat is HEX RRGGBB.

Q3 Subscription to ColorXY type expects JSON that includes "color":{"x":<value>,"y":<value>},"brightness":<value>. mcsMQTT translates this into RRGGBB which is what the HS device expects for the Color Picker control.

Q3 Publish of ColorXY type will send JSON {"color":{"r":<decimal value>,"g":<decimal value>, "b":<decimal value>}}
I have not yet investigated #10

[tfoutfou]

#7 ok for the first part of the response , i understand the downsides

for the second part , i am talking only about JSON parent HS devices
if you go in the HS Device config , in the advanced tab you can see the Value , this value is growing by +1 every time a JSON payload
is received (not realtime you have to refresh the page) , this mean somewhere in the plugin there is a command to update this value .
the question is all about CPU , if updating HS devices is CPU intensive , why updating the parent device Value ?
maybe if will stay away from parent for Json devices to keep it simple

#8 by doing this your way it is working , i was using the "Ref" button in association tab , then clicking "Delete Sub and Ref" button
tested this right now i can confirm with the "delete sub and red" it is not working

#9 perfectly agreed on this

F3 nice

F5 ok but i still see some VSP in the list , even when "text" is selected , maybe created at the creation of the device ??

Q3 so the payload must be something like that for RGB ? : AABBCC
where AA = 170 for Red BB = 187 for Green CC = 204 for blue ?

#10 ok , but while waiting can you tell me if Express mode on JSON devices is supposed to have the same CPU saving effect than on normal topic devices ?
supposedly yes , because i imagine the JSON devices as virtual topics created by the plugin but then afterward it have to be processed the same way as a normal topic right ? or is it using other internal plugin logic ?

Thanks

[Michael McSharry]

#10 From Table 2 of the manual you can see that a JSON payload and a non-JSON payload take just about the same to process. Compare row 3 with rows 1 and two of the table. What takes the time is updating the HS device. Compare rows 3 and 4.

#8 I will be able to replicate now that I know your sequence.

#7 I agree there is an incrementing counter for DeviceValue. This was done so one could trigger an event on the receipt of a new payload. It was done before benchmarking so I will remove it now that we know the CPU penalty.

[tfoutfou]

#7 one can trigger and event with the condition "mqtt topic received" and inputing the name of the parent if needed , i have a feeling this approach is more efficient

#10 ok about the table , i already have seen it , but since i have an issue with express mode and JSON at the moment , i cant use it

[Michael McSharry]

What is your specific issue with JSON/Express?

[tfoutfou]

same as explained in post 4

#10 NEW BUG / QUESTION

Express mode on JSON devices dont work , if i enable "express" on one device , the full JSON payload is putted as a string in the said device , and the associated parent and remaining child device stop updating at all.
If any child device is set to Express mode , we can NOT set any other one to express mode , we get this error message

"Topic PZEM04:A has already been accepted for Express mode. PZEM04:A must not be selected for Express mode to be able to accept PZEM04:Pf"

[Michael McSharry]

I believe everything on your bug and feature list that we agreed to is at It is at http://mcsSprinklers.com/mcsMQTT_3_5_7_0.zip except for the Obsolete checkbox on the Association tab. I will do that next.

Let me know if I missed something or something should work differently to make it better. You should be able to do JSON in Express mode. I let the plugin handle the dependencies between the two features.

[tfoutfou]

thank you for the reactivity

edit : typo in your link , replaced 4 with 5 , now i got the last version

[tfoutfou]

10# Express mode still not working with JSON , strangest behavior is happening
when i check any child device , i get the same message as before
but this time all the other child are being checked expect the one i actually checked (if i check again i get the message again)

the same thing is happening with the JSON full payload being applied in DeviceString for the device i checked (but device appear not checked)
the same thing is happening with the childs (but this time they appear checked) , no update anymore of Value Or String
NEW bug : the parent device is deleted when i enable express for a JSON topic

[Michael McSharry]

Correct about version number.
Express mode does not support Parent/Child grouping. It is intentional that no grouping is available for express. If you remove express checkbox the grouping will be restored.
Can you give me screen shots and step by step description of what is not working. I don't see issue at my end so I must not be doing the same thing as you.