Announcement

Collapse
No announcement yet.

Excessive CPU and Excessive MQTT Topics

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Excessive CPU and Excessive MQTT Topics

    Hello,
    I started a nodeMCU project that communicates with HomeSeer via MQTT and noticed several issues with the mcsMQTT plugin. Maybe it's my setup? Is it not recommended to have mosquitto installed on the HS machine? I'm using Ubuntu Server x64 16.04.03 and Mono 5.0.1.1. System info and picures are attached below.

    Here's a list of what I've found.

    Excessive CPU load.
    The mcsMQTT process uses the most CPU resources of any process on my machine. Even the HomeSeer Console process. After I successfully configured my project last night in Arduino, I took a look at the HS machine and the CPU was pegged at 100%, when it's typically under 10%. I rebooted the PC and the load went down but it was still over 60%. As of this morning, it's down to about 25%, but that's still very high. This brings me to the next issue.

    Excessive MQTT Topics
    After I rebooted last night I found excessive topics listed in the mcsMQTT plugin page. There is a topic for every HomeSeer device on my system. This includes all Z-Wave and virtual devices. Is this by design? There are 300 of them and the page takes quite a while to load.

    Removing Topics Deletes the HS Device
    So after finding these extra topics, I started removing them and it actually deleted the device in HomeSeer! For example, when I removed the topic for a Z-Wave light, it actually removed the HS device for the light in HS device management and I lost all control of the device. I didn't realize it was doing this until all topics were removed and when I went to device management, I had only like 10 devices when there should be 300!!! I lost all control of my HS system and had to restore a backup.

    Publish Label / Number Radio Buttons do not always appear
    Of all the issues, this one cost me the most time because on the original topic I was using for testing, this did not appear. I was testing turning on and off a led on the nodeMCU and it worked fine sending a 1 or 0. So I then added On and Off buttons but the led wouldn't work with On and Off. In the nodeMCU serial console I noticed it was receiving the string On and Off. This was a total pain to program for and took several hours to figure out how to convert a Byte value to Char then to String so it could be parsed by the nodeMCU. After having the issue above with the excessive devices, I notice some of the devices have a radio button to publish either the label or the number. I then noticed that if I refreshed the mcsMQTT page, the devices that had the radio buttons would change. So I did this quite a few times until my test device had them and enabled number. I then tested this and sure enough, pressing the On button sends a 1 and Off sends a 0. This would have saved me hours!!!!

    Please see my attached pictures of the CPU load, excessive topics, and topics that do not have radio buttons. Is there something I did wrong? This was very frustrating and I'm super lucky to have had a working backup to revert to otherwise this would have been a disaster. I have since removed all traces of this plugin from my system but would like to use it again if the bugs can be worked out.

    Current Date/Time: 3/13/2018 6:41:39 AM
    HomeSeer Version: HS3 Pro Edition 3.0.0.420
    Linux version: Linux hometrollerSEL 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux System Uptime: 0 Days 4 Hours 40 Minutes 49 Seconds
    IP Address: 192.168.8.2
    Number of Devices: 300
    Number of Events: 138
    Available Threads: 199
    HSTouch Enabled: True
    Event Threads: 1
    Event Trigger Eval Queue: 0
    Event Trigger Priority Eval Queue: 0
    Device Exec Queue: 0
    HSTouch Event Queue: 0
    Email Send Queue: 0
    Anti Virus Installed:

    Enabled Plug-Ins
    3.3.0.0: APCUPSD
    3.1.1.24385: Blue-Iris
    1.2.2.0: Device History
    0.0.0.19: DoorBird
    3.0.0.43: EasyTrigger
    3.0.0.22: Ecobee
    3.0.9.2: mcsMQTT
    3.0.1.4: MeiHarmonyHub
    0.0.0.38: Pushover 3P
    30.0.0.36: RFXCOM
    3.0.5.6: SDJ-Health
    3.0.6542.35207: UltraWeatherWU3
    3.0.1.199: Z-Wave
    Attached Files

    #2
    No recommendation as to where mosquitto is running.

    Excessive CPU load. CPU load should be a function of the number of MQTT topics being managed by broker and the rate of HS3 device change. To isolate for your situation the first step I suggest is after setting up your desired topics then disable discovery (subscribe only to accepted topic) which is a radio button setting. Further investigation can be done with timestamp debug to see where the utilization is occurring. Do you want to work with me on this investigation?

    Excessive MQTT Topics. Remove the checkbox on showing non-plugin devices after you have identified the HS3 devices that you want to have publish. There is also a display filter that allows you to narrow down specific topics and payloads as well as a checkbox to display only Accepted topics. After I have done my "topic editing" I will use the Display only Accepted checkbox and then later when I want to change something I will use the filter and uncheck this checkbox.

    Removing Topics Deletes the HS Device. The intention is that it remove the devices created by mcsMQTT plugin. If it is also removing other plugin devices then I need to fix it.

    Publish Label / Number Radio Buttons do not always appear. 0/1, on/off, open/close are all treated as two-state devices. The labels setup as value/graphic/status pairs are based upon the payload at the time the topic is Accepted. These provide the option to publish either the 0/1 value or the vgp value. If the payload is different then publishing the label option is not provided and either DeviceValue or DeviceString will be published for numeric and non-numeric topic payloads. I tried to describe this behavior in the manual. The intention was to make the MQTT/HS3 bridge as natural as possible with the minimum amount of effort by the user.

    Comment


      #3
      I looked at the source code when an Accepted devices has the Accepted checkbox removed. For the devices that are considered SEND devices it removes the device from the list of HS3 devices that are published, but it does not delete the device. For the devices that are considered RECEIVE devices it will delete the device. The RECEIVE devices should only be the ones that were created by mcsMQTT as a result of Accepting a topic received from the broker.

      There is also a Remove checkbox that only affects the displayed topics. In essence it is a topic by topic filter while the pulldowns toward the top are like wildcards to reduce the number of topics displayed. The Remove checkbox should also not delete any HS3 devices.

      Can you provide more guidance on what you did to cause your zwave device to be deleted?

      Comment


        #4
        Hi Michael, thanks for the response. I see a lot of potential with this plugin and would like to help resolve some of these issues if you're interested. Your instruction manual is great but a lot of people would probably benefit if the plugin was a little more intuitive. Please see my recommendations below in blue.

        Auto Generated Topics
        All of the topics listed in the attached screenshots and 99% of the total topics were created by the plugin. Only a small number of the auto-generated topics corresponding to HS devices did not have the accept box checked. I would say about 10% didn't have the accept box automatically checked. Instead of having it automatically make a topic for every HS device, could a drop down be used to select the device and a topic is then generated, instead of having all devices auto-generated? This would help reduce confusion, clutter and CPU load.

        Deleted Z-Wave Devices
        So for the Z-Wave devices that were deleted, all that I did was uncheck the A box and check the R box, click the reload button and it removed the Z-Wave HS devices. Is there a way to have it delete the HS device ONLY if it was created by mcsMQTT?

        Label / Number Radios
        In my case, the MQTT device sent a "Hello World" payload to the topic on startup. So this is probably why the radios never showed. Is it possible to have the Label / Number radios always show? Would help reduce confusion.

        Buttons Above Topics
        What are the buttons directly above the topics for? For instance hat do the A and R buttons do? I assumed they were for sorting but couldn't find any info on them. Is it possible to have an infobox appear when hovering over the buttons?

        Initial Setup Observation
        This is a little difficult to explain and probably more difficult to follow but I'll try to explain what I saw and did while setting it up. Hopefully this helps.

        After installing the plugin, clicking the plugin setup page was already sluggish. It took probably 10 seconds to load. All other pages on HS load within one to two seconds. When it loaded there were no topics listed. Only after connecting to the broker and having an MQTT device publish a topic was a topic listed in the center portion of the plugin. The only topics displayed were created by received topics from the broker, there were no HS devices listed. I selected the A box for the new topic, clicked reload and it created an HS device. Keep in mind the payload was "Hello World" so it didn't show the radios.

        I then tested the sending of a payload by entering 1 in the text box for the HS device and clicking submit. The payload was received by the nodeMCU and it turned on and off the LED. I then created On and Off buttons with On = 1 and Off = 0 and noticed the device did not operate. I then went through the process of changing the code to have the device recognize text strings, which wasn't easy.

        After the buttons were working, I then noticed the CPU was pegged on the HS machine. I rebooted it and the load went down but was still high. I then went to the mcsMQTT setup page and it took about a minute to load. Once loaded all HS devices were listed as topics with many of them (probably ~300) having the A box checked. I then unchecked the A and checked the R on all of them (it took forever) then went to the main HS screen and all but about 20 devices were deleted.

        At this point, I restored a backup and rebooted which brought back all HS devices but the plugin was still enabled and it created all of the topics again, with many of the A boxes selected. I then disabled and deleted the plugin and performed another restore. Now my system is back to as it was before installing mcsMQTT and CPU load is under 10%.

        Comment


          #5
          I added some millisecond-level timing around HS3 events and MQTT messaging. The data below is running on Odroid C1 which is the same class as a RPi3. It is not exact for timing, but gives an idea. In my case my CPU utilization is in the 0 to 5% range when viewing with htop on DietPi/Jessie.

          I uploaded 3.0.9.3 which contains the timing output to \data\mcsMQTT_debug.txt when Debug is enabled.

          Here is a ballpark for the throughput demands for certain tasks. It seems the recursive JSON loads is the biggest hitter so if one has much of this data being published and not being used by HS3 then disabling discovery may have some cpu benefits.

          HS3 event change to non-accepted not published (160-149 = 11 milliseconds)
          HS3 event change to value published (910-881 = 29 milliseconds)
          MQTT topic received for non-acccepted topic (223-193 = 30 milliseconds)
          MQTT topic received to HS3 value updated (398-325 = 73 milliseconds)
          MQTT JSON received with 9 payload items (325-059 = 266 milliseconds)

          Comment


            #6
            Auto Generated Topics
            All of the topics listed in the attached screenshots and 99% of the total topics were created by the plugin. Only a small number of the auto-generated topics corresponding to HS devices did not have the accept box checked. I would say about 10% didn't have the accept box automatically checked. Instead of having it automatically make a topic for every HS device, could a drop down be used to select the device and a topic is then generated, instead of having all devices auto-generated? This would help reduce confusion, clutter and CPU load.
            Something is wrong if the Accept checkbox is being automatically selected. There is intentional redundancy in the retained setup information. This was done as an integrity check and to allow a user to take his mcsMQTT database and have the mcsMQTT environment moved to another HS3 instance.

            During plugin startup it enumerates all HS3 devices. It assures consistency between the database and the device's PED. It will create HS3 devices for Accepted ones in the database that were not found during enumeration. The only devices that should be in the database are the ones owned by mcsMQTT and those non-plugin devices that have been Accepted.

            If somehow a non-plugin devices finds its way into the mcsMQTT database then it will become accepted during initialization. I don't yet understand how this could happen or if this is the mechanism by which they were automatically accepted in your case. I will have to polk around to try to recreated the scenario.

            Deleted Z-Wave Devices
            So for the Z-Wave devices that were deleted, all that I did was uncheck the A box and check the R box, click the reload button and it removed the Z-Wave HS devices. Is there a way to have it delete the HS device ONLY if it was created by mcsMQTT?
            This is a logical extension of a device thought to be a RECEIVE (mcsMQTT) device and then properly responding to a user action to unAccept it. In the back of my head an explicit check on the Interface property of the device would prevent this situation, but would rather address the root cause than patch the symptom.


            Label / Number Radios
            In my case, the MQTT device sent a "Hello World" payload to the topic on startup. So this is probably why the radios never showed. Is it possible to have the Label / Number radios always show? Would help reduce confusion.
            The labels are associated with the CAPI control type property of the device being "HomeSeerAPI.Enums.CAPIControlType.Button". If it is not a button then there is no label to publish so the option to publish label vs. value does not exist. This property is set when the device is created and it is based upon the payload containing open, closed, on, or off.

            I am not totally clear on what you would like to have seen to avoid your original issue that caused you additional work. There is logic in the HTML on mcsMQTT setup with <div> tags to dynamically build cells based upon user selections. For example, when Accept is selected then the additional options of send topic and label/number are shown. I have not have had success on making it work and was not important enough for the time to investigate.

            Buttons Above Topics
            What are the buttons directly above the topics for? For instance hat do the A and R buttons do? I assumed they were for sorting but couldn't find any info on them. Is it possible to have an infobox appear when hovering over the buttons?
            They are intended to be sort buttons as you suspected. I will research to see what needs to be done to add tool tips.

            Initial Setup Observation
            This is a little difficult to explain and probably more difficult to follow but I'll try to explain what I saw and did while setting it up. Hopefully this helps.
            After installing the plugin, clicking the plugin setup page was already sluggish. It took probably 10 seconds to load.
            In my case the setup page with only show accepted option loads in about the same time as the HS3 Device Management page.... a few seconds. A full page is about 10 seconds and likely is a function of the number of devices. I can do some timing debug to better understand the time. From the early HS days it was known that VB string concatenation is very slow so StringBuilder is used for that function. I know that when accessing the HS API I always call in the current hs object which can be slow, but the tradeoff is assuring the data being referenced is always current. I consider this most important during the setup selections.

            Comment


              #7
              In the \data folder each version of mcsMQTT.db is backed up. Do you still have them so I can observe some of history that might help explain why the auto acceptance was happening?

              Comment


                #8
                Yes, what's the best way to get it to you? Does it contain sensitive info?

                I fired up a virtual machine with HS and loaded mcsMQTT, rebooted and it created accepted topics for each HS device. Below is a screenshot.

                Sure would be nice to have some way to selectively create topics for HS devices.
                Attached Files

                Comment


                  #9

                  Comment


                    #10
                    Not to change the topic, interested in not having so many unaccepted topics. Seems logical to only make the topics that will actually be used and reduce the clutter. Since they're not automatically created, this would also prevent them from being automatically accepted and make new topics a whole lot easier to find. I think it would also be much more intuitive for a new user.

                    Auto or not, either way is fine. Would prefer to not have them automatically created, that's all.

                    So checked my live HS system and all databases and logs were deleted. I have the databases from my virtual system and attached them. Let me know if you need them from a live system and I'll see about installing the plugin on my main system again.
                    Attached Files

                    Comment


                      #11
                      The intended design is to act as a firewall where rules allow penetration. If you do not want discovery and only manually enter the subscription topics then set the radio button to disable discovery. You can then enter the desired subscriptions one by one. The downside to this is that there will be no payload at time of Accept so it will show up as a status only device. This may be something I can let be edited or at this time you can get the desired effect by Accepting the manually entered topic, allow the broker to forward when it is published and then UnAccept followed by Accept to create a new device with payload available and then control options presented.

                      My personal preference is to see what is out there and use the pulldown filters to focus on subjects of interest at any given time. It helps with debugging and understanding system behavior when topics are close, but not exact. It also means I do not know what a node publishes apriori and may select one of multiple equivalent topics that best suit my need.

                      The two databases show no Accepted devices, but your screenshot showed them. The second clue is that the Device Reference should always be positive for existing HS3 devices. In one of the two almost all Ref values are -1.

                      Is your virtual machine also bogged down with mcsMQTT. Have you used the debug to collect some data to understand timing?

                      Comment


                        #12
                        Enabled debugging and rebooted several times. On all but one reboot new topics were shown and auto accepted. See screenshots and logs.
                        Attached Files

                        Comment


                          #13
                          I believe I ran down the cause of the auto accept. It is corrected in 3.9.0.4 from updater. I will look at you debug to double check and look at some of the timing.

                          p.s. the debug output is /data/mcsMQTT_Debug.txt

                          Comment


                            #14

                            Comment


                              #15
                              Tested 3.9.0.4 on the VM and it appeared to have resolved the auto accept and the CPU load was low. I then installed it on my live system and it didn't populate the HS device topics until a reboot.

                              Once it populated them the PI page load time went from about 3 seconds to about 30 seconds. When Show Only Accepted is checked, the page load time is about seven seconds. Attached the debug from the live system in case you want to look into the timing.

                              It appears all of the HS devices are listed but there several still have a ref of -1. Are the topics with ref -1 an issue?

                              Congrats! The auto accept has been resolved and the CPU load appears to be lower. Thanks for fixing this so quickly!
                              Attached Files

                              Comment

                              Working...
                              X