Announcement

Collapse
No announcement yet.

Excessive CPU and Excessive MQTT Topics

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    I use both Windows 10 and Debian Jessie and both provide similar results. I've also done mixed testing HS in Linux and the plugin on Windows 7.

    mcsMQTT is very multithreaded. I see no indication that the plugin has any delay in responding the HS. The transaction that has issue is
    User hits show button
    HS postback the button hit to plugin
    ...Plugin responds with 'Please Wait' text in DivToUpdate API call
    ...Plugin spawn thread to build Association table
    ...Plugin returns from HS postback
    HS posts 2 second timer event
    ...Plugin refreshes table size status where button was pressed
    ...Plugin returns
    HS posts 2 second timer event
    ...Plugin responds with DivToUpdate with HTML for Association table generated in other thread
    ...Plugin returns
    Two seconds later no timer postback from HS
    Lack of postback continues for 10 seconds, 20 seconds etc
    HS posts 2 second timer event (The browser now has table rendered)
    ...Plugin responds with DivToUpdate to restore the Show button
    ...Plugin returns

    If the table is too large then the 2 second postback will never occur

    I do not have socket disconnects between HS and the plugin, but your data shows that the socket connection status goes to disconnected in main loop that is being monitored by the plugin at 10 millisecond intervals.

    It looks to me as if HS got caught up in managing the HTML server and was unable to satisfy the criteria for keeping the main connection alive. I, however, have no visibility on what criteria exists to keep the connection alive. From the plugin side there is no management of the connection and its job is just to sense that the connection is lost, shut itself down so HS can then start it up again at its next monitoring interval.

    Something like wireshark could see what is happening at the TCP/IP level with the socket. Since I cannot create the main loop disconnect problem there is nothing for me to look at with Wireshark.

    Prior to 3.2.6.0 the transaction with the Show button was to remain in the same thread as called with the button push postback and not return until the table was constructed. There was no DivToUpdate calls made to change the text/button to provide progress feedback. This means that the CPU was dedicated to building the table during this time and the return back to HS would be a second or two rather than the milliseconds it is taking with the newer design. What it did do is added some additional burden for HS Server to communicate the progress status to the Browser at each of what turned out to be two updates.

    When comparing to other plugins behavior the question that needs to asked is to what degree are these other plugins asking the HS server to build tables of HTML the size that mcsMQTT is asking? The issues now are not with general plugin operation, but in Web Page update. I suspect that all your other plugins have very modest demands on the HS Server. If this is not the case then perhaps a dialog can be opened with the author of one that produces large tables of HTML that can be dynamically updated.

    Comment


      I did some further testing to understand which process is using CPU. My test setup was HS and plugin on a single core Windows 10 laptop and Chrome on a 4 core 4GHz I7. I requested a 5835 row Association table.
      For about 2 second mcsMQTT.exe was running at 25%
      For about 1 second or less HS3.exe was running at 15%
      Chrome has been running for 5 or more minutes with CPU jumping around between 13% and 60%. Memory use has gone from 35,000K up to 2,700,000K. The table did render.

      This shows that the HS Server should not be busy serving up the HTML and only the Chrome browser is burdened with rendering the HTML of the table.

      It does not explain where there is any relationship between the button press and the socket connection on the main thread.

      When I repeat the test with everything on the single core Windows 10 then similar timings exist for mcsMQTT.exe and HS3.exe. Chrome is using 50% and memory use continues to grow. It is growing at a much slower rate to reflect the relative CPU speed between the two computers. I suspect this test will fail with Chrome reporting something wrong as its memory consumption becomes too large for the more limited Windows 10 computer.

      Chrome is running 16 threads. HS3 is running 44 and mcsMQTT is running 23. Both HS3 and mcsMQTT hover around 0% utilization with no other activity happening.

      Comment


        I'm running Homeseer on an Lubuntu virtual machine (temporarily on a machine with few resources). Just loaded 3.2.7.1. Mosquitto is running on the Homeseer computer and I can subscribe to a topic from the command line using:
        mosquitto _sub -h localhost -t test -u "username" -P "password"
        I can then publish to the homeseer broker from another linux machine using:
        mosquitto_pub -h FQDN -t test -m "hello again" -p 8883 --capath /etc/ssl/certs/ -u "username" -P "password"

        With mcsmqtt disabled my CPU hovers around 20%. About a minute after enabling the mcsmqtt my cpu will go to 100% and stay there. I can still send messages fine using the command lines but mcsmqtt never sees them. Disable mcsmqtt and the cpu drops to normal.

        Current Date/Time: 4/3/2018 8:34:36 AM
        HomeSeer Version: HS3 Pro Edition 3.0.0.423
        Linux version: Linux HS3 4.13.0-37-generic #42-Ubuntu SMP Wed Mar 7 14:13:23 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux System Uptime: 0 Days 1 Hour 30 Minutes 45 Seconds
        Number of Devices: 414
        Number of Events: 90
        Available Threads: 199
        HSTouch Enabled: True
        Event Threads: 0
        Event Trigger Eval Queue: 0
        Event Trigger Priority Eval Queue: 0
        Device Exec Queue: 0
        HSTouch Event Queue: 0
        Email Send Queue: 0
        Anti Virus Installed:

        Enabled Plug-Ins
        3.0.2.0: AquaConnect
        3.0.0.56: DSC Security
        3.0.1.10: Kodi
        1.2.0.0: Monoprice Amp
        3.0.6629.22233: Ultra1Wire3
        3.0.0.83: weatherXML
        3.0.0.36: X10
        0.0.0.6: YAMAHA-RECEIVER
        3.0.1.190: Z-Wave
        https://forums.homeseer.com/forum/de...plifier-plugin

        Comment


          It looks like it keeps launching threads that are unhappy. Here is a screen capture of htop after about 2 minutes. After I took it thread count kept going up.
          Attached Files
          https://forums.homeseer.com/forum/de...plifier-plugin

          Comment


            Here is a zip of the debug:
            Attached Files
            https://forums.homeseer.com/forum/de...plifier-plugin

            Comment


              mcsMQTT is not seeing MQTT traffic because of the following in your log which means that the socket status shows not connected even though the socket connection to the Broker returned normal "Accepted=0" status.
              Code:
              MQTT Broker Connection Accepted, Connected=False
              My guess is that the dll managing the Broker connection is the one that is spawning new threads for a similar reason that the connection status remains false

              My htop view shows hsConsole.exe has about 40 children threads. One of these is mcsMQTT.exe which itself then has 18 children. They all hover around 0% CPU and thread count does not grow.

              mcsMQTT spawns three persistent threads and on events will spawn threads that run once and terminate. These are listed below. I suspect that most of the additional threads are related to the M2Mqtt.Net.dll that manages the socket level MQTT traffic.

              Persistent Threads
              ----------------------
              Main (inherited from HS)
              Connect to Broker
              Handle Received Messages
              Handle HS Event Callbacks


              Temporary Threads
              ---------------------
              Web Page postbacks (inherited from HS)
              Background Initialization
              Build Association Table
              Query Button Press

              My guess on your situation is that the Virtual Machine is not providing the socket emulation the M2Mqtt.Net.dll is expecting and becomes confused. What I did see in your startup debug is that you have sometime in the past received what looks like one receive MQTT message that has a JSON payload. This data was read from the mcsMQTT.db database during startup initialization.

              You should also be able to bring up the Statistics Tab to view what mcsMQTT see at the message level.

              Code:
              4/3/2018 9:13:42 AM	1097	| PopulateReceiveDict owntracks/happnatious1/BillPhone:_type, PluginDevice=True  
              4/3/2018 9:13:42 AM	1149	| PopulateReceiveDict owntracks/happnatious1/BillPhone:tid, PluginDevice=True  
              4/3/2018 9:13:42 AM	1151	| PopulateReceiveDict owntracks/happnatious1/BillPhone:acc, PluginDevice=True  
              4/3/2018 9:13:42 AM	1153	| PopulateReceiveDict owntracks/happnatious1/BillPhone:batt, PluginDevice=True  
              4/3/2018 9:13:42 AM	1154	| PopulateReceiveDict owntracks/happnatious1/BillPhone:conn, PluginDevice=True  
              4/3/2018 9:13:42 AM	1154	| PopulateReceiveDict owntracks/happnatious1/BillPhone:lat, PluginDevice=True  
              4/3/2018 9:13:42 AM	1155	| PopulateReceiveDict owntracks/happnatious1/BillPhone:lon, PluginDevice=True  
              4/3/2018 9:13:42 AM	1157	| PopulateReceiveDict owntracks/happnatious1/BillPhone:tst, PluginDevice=True  
              4/3/2018 9:13:42 AM	1097	| PopulateReceiveDict owntracks/happnatious1/BillPhone:_type, PluginDevice=True  
              4/3/2018 9:13:42 AM	1160	| PopulateReceiveDict owntracks/happnatious1/BillPhone:t, PluginDevice=True

              Comment


                I was able to connect at one time. I don't see a setting for port. What port is mcsmqtt using?
                https://forums.homeseer.com/forum/de...plifier-plugin

                Comment


                  1883 is the standard default and the plugin does not provide any option to use a different one

                  Comment


                    As for page rendering times I have experimented with changing the update resolution from a cell to a row of the table. I have also tried to render the table in a new tab like the history page is done. Neither approach has any measurable benefit. There is just too much HTML in 5000 rows for the browser to swallow at one time.

                    While the user should be selecting a reasonable number of rows, I think I will try to put a limit and perhaps a means to scroll through a large table 20 rows at a time.

                    Comment


                      Some type of list that prepopulates and has scroll buttons would probably be really good. Maybe trying to avoid ajax when pushing large amounts of data/fields. KISS, the last S is for silly
                      Last edited by mwolter; April 3, 2018, 03:15 PM.

                      Comment


                        3.2.8.0 is in the updater with the Association table rendering limited to 20 rows and buttons to select the row position. The construction logic is back to earlier versions where the table is fully built upon initial display of the URL, but only the first 20 are provided to the browser.

                        In my testing in both environment it worked well.

                        Good luck.

                        Comment


                          Thanks for working on this. Installed 3.2.9.2 and unfortunately, I'm receiving a blank screen on the MQTT Setup page and the CPU core is pegged at 100%.

                          Deleted the DBs and Configs, tried again, same thing. Deleted the PI through the PI manager and deleted all PI traces, then downloaded it again, same thing. No signs of any issues with the rest of the system, HS and all other plugins are working fine.

                          Also noticed the following file is made one level up from the HomeSeer root directory. In my case its /usr/local

                          HomeSeerDatamcsMQTTmcsMQTT Debug.txt

                          Screenshots and debug are attached.

                          Update: Manually edited the config file and enabled debug. Attached the updated debug file.
                          Attached Files

                          Comment


                            The debug file shows that you have over 500 non-plugin devices in the database, but otherwise very normal operation.

                            What rings a bell with me now based upon the location of the debug file is something that Rich posted on the MB with HS2. It may still apply with HS3. HS creates objects for the plugin initially just to call it and get its name then destroys those and creates new ones for the real startup.

                            The "/" are missing from the file path. The "/" vs. "\" is determined now in the Main procedure before anything is done. Before you reported this a day or two ago this logic was in a later procedure that was actually called by HS.

                            I bet we have two processes running and neither has a full picture. Let me move that logic back to a procedure called by HS.

                            Comment


                              I put 3.2.9.3 in the updater with the initial logic restored to InitIO API call rather than in Main. Also no longer tried to write debug from the Main process.

                              What is a odd is that my debug file today is located in the proper location and only the one created on 4/1 was in the HS parent folder. Something a little different between your and mine setup for interaction with HS.

                              Note also that you indicated that you deleted the database files. The data from the database files was showing in the debug log you posted. It kind of looks as if there is another instance of mcsMQTT.exe that is running based upon a different .ini file that is doing Device discovery and populating the mcsMQTT.db file. This would be happening before the instance that produced the debug data.

                              Comment


                                Thank you Michael.

                                Just an FYI ....typically when looking at the plugins under plugin manager the name is linked to the configuration menu and I noticed that your plugin is not linked.

                                [ATTACH]67881[/ATTACH]
                                - Pete

                                Auto mator
                                Homeseer 3 Pro - 3.0.0.548 (Linux) - Ubuntu 18.04/W7e 64 bit Intel Haswell CPU 16Gb

                                HS4 Pro - Ubuntu 22.04 / Lenova Tiny M900 / 32Gb Ram
                                HSTouch on Intel tabletop tablets (Jogglers) - Asus AIO - Windows 11

                                X10, UPB, Zigbee, ZWave and Wifi MQTT automation-Tasmota-Espurna. OmniPro 2, Russound zoned audio, Alexa, Cheaper RFID, W800 and Home Assistant

                                Comment

                                Working...
                                X