HS4 mcsMQTT skWare Device History

[syedamerali]

I have the following settings pertaining to data logging in mcsMQTT:


The value of the devices is not getting updated in Device History. Is this expected?

It leaves me in a bit of a quandry as I cannot graph non-asscociated devices in mcsMQTT and mcsMQTT associated devices in Device history.

[Michael McSharry]

I'm not clear on what you are actually asking. mcsMQTT keeps its own database for charting. Your setup looks fine for mcsMQTT charting of devices. I did a quick test of a device that changed from 0 to 255 and back to 0 with settings similar to yours. The charts shows when requested


skWare plug-in has its own setup and database for history. I have never used it myself. Whatever you do in mcsMQTT settings has no effect on this other plug-in.

[syedamerali]

Some devices I now have can ONLY be charted with Device History - let us call these devices group A devices or z-wave devices. Then I have a set of devices which can only be charted in mcsMQTT- let us call these devices group B devices or MQTT devices. If I want a device from group A to be charted with a device from group B on the same chart I cannot do it.
The workaround seems to be not to let mcsMQTT create its own devices by clicking the "a" column. Create a device manually then assosciate it with mcsMQTT by using the
"ref" column. This way Device History can track the changes and subsequently chart them. This way all the historical data is in one database and anything can be plotted against anything else.
This behaviour of the plugin mcsMQTT is at variance with other plugins. For instance, the Arduino plugin creates its own devices just like mcsMQTT. The difference is that that the devices created by the arduino plugin are normal HS Devices whose values can be tracked by any other plugin which stores historical data.
To request a fundamental change like this in a mature product like mcsMQTT would be unreasonable. I have found a workaround, which is a little more tedious but gets the job done.
Thank you once again for everything. I will try not to waste your time any more.
PS
I upgraded (?) to HS4 last evening. If you install mcsMQTT and/or mcsShelly from the updater the computer hangs till you change the DLL file. Also, the windows in mcsShelly have the title mcsMQTT so one is confused as to whether one is in mcsMQTT or mcsShelly.

[Michael McSharry]

I don't understand why you find mcsMQTT at variance with other plugins. When you Associate a MQTT topic with a HS device the the plugin will create a HS device for this association and all plugins and users have access to the HS device. This is the only mechanism provided by HS and is the method by which HS events are triggered.

mcsMQTT also provides the ability to view MQTT traffic without taking the step of building HS devices. It is not creating devices, but just providing a viewable page to see the traffic in a easy manner. You can collect the Zwave data in the mcsMQTT database without creating mcsMQTT devices, but the charting support was not intended for long term trend analysis, but for short term data analysis and charting is just basic trends of one or two signals per chart.

mcsMQTT also has the mechanism to automatically create HS devices based upon topic template and also supports the homesassistant discovery protocol to auto create devices in HS. If your associations follow a pattern then this will save you time. If they do not then the "A"ssociate checkbox is the best way. These settings are in the General tab, Inbound Management section.

Yesterday I ran down the mcsShelly/mcsMQTT conflict and made the changes in the plugin to all both to be run at the same time. The original design intent is to run one or the other. They run fine together now on HS3. HS4 is having some problems handling both HS3 plugins as it gives the same pending message I mentioned in the other thread and mixes up the Config menu with the Hubitat plugin menu.

There is no HS4 mcsMQTT plugin in the updater, but there soon will be. I still need to make certain the HS4 plugins can run together. There really is no rush to migrate to HS4 as it is still in beta and there are many known issues that are being worked through.

I am uploading version 5.3.0.0 of the HS plugins today. If you are interested in running both before they are available in the updater then the process in HS3 is manually make a copy of the mcsMQTT subfolders and rename the copy to mcsShelly. These are in \bin, \html, \data subfolders of HS. From the 5.3.0.0 zip files copy the HSPI*.exe to HS folder. Copy mcsMQTT_2020.dll to each \bin\mcsMQTT and \bin\mcsShelly. You will also want HSPI_MCSSHELLY.exe.config from the zip into the HS folder. It tells the plugin where to look for the .dlls.

The plugin update is at http://mcsSprinklers.com/mcsMQTT_5_3_0_0.zip and http://mcsSprinklers.com/mcsShelly_5_3_0_0.zip. I just started the uploads now so if all goes well if should be 15 minutes or so.

[syedamerali]

you are right about HS4 - I had made a backup on Acronis before downgrading to HS4. Currently going back to HS3.

[Michael McSharry]

I took a brief look at Device History and see that is uses SQLite as the database. This works fine for limited data. It was designed for use on embedded devices. In my other database intensive applications I use MySQL or SQL Server. They are more robust and can handle large volumes of data.

A popular charting tool is Grafana. InfluxDB is often used with it as it is designed for time-series data.. Since I have my own charting from a decade ago I have not ventured into Grafana and InfluxDB, but a natural addition to mcsMQTT would be to a data source for these.

[syedamerali]

back to HS3 - life is so familiar now.
About the devices, I was tracking the wrong devices which were not being updating. They had the same names as the mqtt devices but were on different floors. I can now confirm that the devices created by mcsMQTT are being tracked properly by skware device history. I am extremely sorry for wasting your time on this subject.
I will wait for the update to go through presumably the updater will inform me that ver 5.3.0.0 is now available - it has not so far.
My mosquitto server seems to have survived all the system reloads and is running happily as a service, at least that is what MQTTLens reports.
Thank you for helping me with journey of discovering MQTT with your plugin mcsMQTT.
PS
I made my first MQTT device yesterday by flashing a Sonoff with my software. Sonoff published topics were picked up by mcsMQTT. Assigned a device to it and it is working with MQTT topics with the relevant payloads. Doing the same thing through JSON URL's is kludjy and time consuming.

[syedamerali]

The updater did not come through so I installed it manually as per your instructions in post #4.
A few things I do not understand.
1. on the mcsShelly general page I get the windows title as mcsMQTT rather than mcsSHELLY which I was expecting - probably because the html files were copied and renamed to mcsSHELLY? See below:



2. In the associations tab, mcsSHELLY seems to have picked up messages from SERVERS I have not subscribed to (as you can see above - the broker name is blank). In fact the associations tab is a replica of the mcsMQTT tab.

3. as the broker name is blank in mcsSHELLY. I would therefore assume that mcsShelly is acting as a broker. This does not appear to be the case. The moment I switch off the mosquitto broker service, my devices which are publishing and receiving messages to the ip on which mcsShelly is stop responding. Also the counters remain at zero. mcsShelly is acting as a broker why is it saying "Client Null" is offline. There is no client.
If the broker is running on mcsSHELLY then MQTTLens (running on a different machine) would not report the broker is off line.




I do not know very much about these things. But in spite of task manager showing seperate instances of mcsShelly and mcsMQTT the Publish/Subscribe data and the servers for both the instances is the same. Am I correct in this assumption? Or has mcsShelly picked up the data when I copied and renamed the mcsMQTT folder in the HS3 data folder?



So what is the way ahead?

[Michael McSharry]

1. This is only cosmetic. I should be using MQTT rather than mcsMQTT in the lablels.
2 & 3. When the Broker IP is blank then the plug-in tries to setup its own Broker. The port 1883 can only be reserved for accepting connections by one program at a time. On one of the two plugins you will need to setup an external broker. If you also have a Mosquitto broker on the same computer running then only the first of the three that start will get access to port 1883 and be the Broker.

[syedamerali]

When I was doing all this the mosquitto broker was off. As mcsMQTT was subscribing to three brokers it could not have been a broker itself. Therefore, as no broker was defined in mcsSHELLY it should have taken ownership of port 1883 and starting behaving like a broker - it did not.
Not quite sure what you mean by this "On one of the two plugins you will need to setup an external broker." external broker? I was under the impression that if you define an external broker in both mcsMQTT/SHELLY they started behaving like clients. If you do not define a broker the plugin starts to behave like a broker. Is my understanding incorrect?
What is still unexplained is why mcsShelly is receiving topics from brokers to which it has not subscribed. The broker field is blank.

You know what, My current system is working fine. I have mcsMQTT defined with three brokers with one of them 127.0.0.1. On the same machine I have a mosquitto broker running as a service - everything is working fine. I think I will forget about the concept of having two instances of mcsMQTT the second one being in the form of mcsSHELLY.

I think in the future, you could perhaps somehow allow having multiple instances of just the one mcsMQTT or alternately you could make mcsMQTT as both a client and server at the same time. The plugin Face Recognition does just that.It has once instance for each camera.

Thank you once again for introducing me to MQTT - I would not have bothered to even look at MQTT without your plugin.
PS I use the words broker and server interchangeably.I read somewhere that the MQTT broker is now called a server.

[Michael McSharry]

mcsMQTT and mcsShelly are both MQTT clients.
if the Broker IP is left blank in either then that plugin will also become a Broker (or Server). Since both plugins are on the same computer you cannot leave both Broker IP fields blank because both will then try to take ownership of port 1883 on the computer and the OS does not allow it.
HS4 has depreciated the concept of multiple instances of a plugin. If the plugin author want to add multiple instances it needs to be done within the plugin. When I added support for connecting to multiple brokers I considered making it a multi-instance plugin, but that would be more CPU cycles overhead than what I elected to do with mcsMQTT.

From a functionality perspective in HS3 it makes no difference if you call a plugin mcsMQTT(0) and mcsMQTT(1) vs. calling the two instances mcsMQTT and mcsShelly.

I believe your current solution with Mosquitto as a broker is a good solution.

[syedamerali]

Thank you once again, I think I will stick to mosquitto broker and mcsMQTT as advised.
PS
Just for the record. When I was doing these things mcsMQTT had 3 brokers defined and mcsSHELLY none and the mosquitto broker was also off.
As I shift from JSON URL's to MQTT I am also having problems with updating existing devices - but this should really be the subject of a properly documented post.
MQTT is blindingly fast as compared to JSON URL's.