Announcement

Collapse
No announcement yet.

Huge Performance Hits with 3.30.1003.1

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Huge Performance Hits with 3.30.1003.1

    Ever since installing the latest HSBuddy plugin version on my HS3 system (3.30.1003.1), I have seen significant degradation in performance. I am running a stock Homeseer S6 HomeTroller Pro (w 8Gb memory). It sometimes takes 20-30 minutes for the system to restart after being fully rebooted, which is necessary every day or two now because it becomes so sluggish it can't run the house properly.

    The plug in frequently stalls on Pending Restart status in the Interfaces and Add Ons window. Sometimes, when I try to shutdown and restart the plug in, it remains running even when the Interfaces window shows otherwise. If you restart in this case, you get TWO separate HSBuddy processes running, which causes all kind of issues.

    I am already planning to replace the S6 with a much more significant machine. The 2-core Celeron S6 just doesn't have sufficient power to run my household. It is constantly pegged at 100% CPU usage no matter what I do with it.


    Current Date/Time: 10/10/2022 06:06:13
    HomeSeer Version: HS3 Pro Edition 3.0.0.548
    Operating System: Microsoft Windows 10 Enterprise 2016 LTSB - Work Station
    System Uptime: 0 Days 1 Hour 20 Minutes 51 Seconds
    Number of Devices: 1696
    Number of Events: 888
    Available Threads: 200
    HSTouch Enabled: True
    Event Threads: 7
    Event Trigger Eval Queue: 0
    Event Trigger Priority Eval Queue: 0
    Device Exec Queue: 0
    HSTouch Event Queue: 0
    Email Send Queue: 0
    Anti Virus Installed: Windows Defender
    In Virtual Machine: No MFG: shuttle inc.
    Enabled Plug-Ins
    3.0.1.9: AmbientWeather
    2.0.63.0: BLBackup
    1.0.6.0: BLShutdown
    3.0.0.3: DTSDeviceImages
    3.0.0.76: EasyTrigger
    3.30.1003.1: HSBuddy
    3.0.0.40: MeiUnifi
    3.0.0.14: NetCAM
    3.1.0.60: Sonos
    3.0.11.0: Z-Wave​

    #2
    Hi, thanks for reporting this. Do you know what version of the plugin you had prior to seeing the performance issues? 3.30 is a very minor update that changes the severity of a couple log lines, not something that would impact performance.

    Is there any chance that recently there may be new devices added that generate a lot of value changes? That could be a cause for higher CPU use from the plugin. I'll look up the issue with the plugin startup time. It shouldn't be doing much work on initial launch.

    Tracking in: https://trello.com/c/33J7DBsU

    Comment


      #3
      Just wanted to report that I have not seen any issue since the update. My system only has 4 gb and has been up for 24 days so far. However, I'm running Win2k16.
      HS3PRO 3.0.0.500 as a Fire Daemon service, Windows 2016 Server Std Intel Core i5 PC HTPC Slim SFF 4GB, 120GB SSD drive, WLG800, RFXCom, TI103,NetCam, UltraNetcam3, BLBackup, CurrentCost 3P Rain8Net, MCsSprinker, HSTouch, Ademco Security plugin/AD2USB, JowiHue, various Oregon Scientific temp/humidity sensors, Z-Net, Zsmoke, Aeron Labs micro switches, Amazon Echo Dots, WS+, WD+ ... on and on.

      Comment


        #4
        Originally posted by avargaskun View Post
        Hi, thanks for reporting this. Do you know what version of the plugin you had prior to seeing the performance issues? 3.30 is a very minor update that changes the severity of a couple log lines, not something that would impact performance.

        Is there any chance that recently there may be new devices added that generate a lot of value changes? That could be a cause for higher CPU use from the plugin. I'll look up the issue with the plugin startup time. It shouldn't be doing much work on initial launch.

        Tracking in: https://trello.com/c/33J7DBsU
        I haven't added any new devices now for a couple of months at least. I generally keep the plug-ins up to date as much as possible, so I am guessing I was on the one immediately preceding 3.30. I am not certain, however. HSBuddy has always started pretty slowly for me - often 5 minutes or more just to load - but this has been much, much slower since the upgrade.

        Comment


          #5
          I set up the TenHsEvents scripting on my machine, and I do run through a LOT of events every minute. I am wondering if that is causing slowness with HSB. I am showing consistently about 150 device events per minute. So far the queue has gotten as high as 50 events. I do have a lot of child devices, such as watts, amps, volts prevented from writing to the log, but I am still doing about 1000 events in less than 6 minutes.

          Comment


            #6
            So here is one thing I am noticing - HSBuddy places a very heavy burden on my HomeTroller every 10 minutes. This continues for around 2 1/2 minutes every 10 minutes, where the CPU is maxed at 100% (HSB is consuming as much as 50% of total resources at this time.

            Click image for larger version

Name:	image.png
Views:	233
Size:	90.3 KB
ID:	1570403

            I have two Android phones that run HSB full time with all battery optimizations disabled (I find this is necessary to run Geofencing properly or the app goes to sleep). I do have dashboards on each phone, but I can confirm that neither were running during these logs.

            The pattern is very predictable and repeatable. If I disable HSB plug-in, CPU usage immediately falls do to a much more reasonable 50-60% on average. HSB seems to be very heavy on CPU usage and this is happening more than 1/5 of the time.

            When this happens, queued events on the system increase, and it takes a while for the queue to fall back to zero. This I believe is what causes the overall sluggishness in commands being executed, schedules being run, etc. Despite being the flagship controller, the S6 is a very weak box performance wise, and the two-core Celeron simply cannot keep up with the demands of a large house (I have over 180 devices spread across 9 separate Z-Net controllers (zones). This is why I will be moving my HS3 to a Intel NUC with 8 I7 cores and 32Gb of memory.

            I have also noticed that if I disable HSB completely before shutdown and restart, my restart times drop from more than 20 minutes to about 3-4 minutes. What I am experimenting with now is combining the BLShutdown in an event that first disables HSBuddy, than 2 minutes later shuts down HS3 and restarts it. I then use Jon00PlugInSet in a separate event to restart HSBuddy 5 minutes after the system detects the PI is offline. This has greatly improved startup times for me, though being a little convoluted to execute.

            Comment


              #7
              Hi bebaldin thank you for the detailed post! I have an idea of what is going on. There's multiple improvements to be made based on your description:
              • When HSBuddy refreshes the cached information of devices it does so very aggressively, running up to 10 parallel tasks. I'll make a change to make this configurable and default to a number that is lower than the # of processors on the box.
              • The plugin does this every 10 minutes because I had noticed instances where the plugin was not receiving events on device admin changes (created/deleted or configuration changed). I the default of 10 minutes is rather aggressive - I am thinking I'll reduce it to hourly or even daily.
              • On plugin startup, it optimizes the history table, which on a system with 1696 devices (wow!) that may be changing frequently, it will likely take a considerable amount of time. I will make a change so that this process occurs on the background and does not block initialization of the plugin.
              Temporarily, a few tweaks we can try specific to your system:
              1. Disable the periodic refresh of devices. You can do that by changing the file under Config\hsbuddy.ini. Look for the setting RefreshPeriodInMinutes under the [Devices] section. Set it to 0 to disable this background process. A restart of the plugin is needed for this to take effect.
              2. If you do not use the device history feature in the app, you can disable the use of the history table on the plugin. You can do this from the Plugin configuration page, just turn off the History feature. A restart of the plugin is not needed for this change.

              Comment


                #8
                Originally posted by avargaskun View Post
                Hi bebaldin thank you for the detailed post! I have an idea of what is going on. There's multiple improvements to be made based on your description:
                • When HSBuddy refreshes the cached information of devices it does so very aggressively, running up to 10 parallel tasks. I'll make a change to make this configurable and default to a number that is lower than the # of processors on the box.
                • The plugin does this every 10 minutes because I had noticed instances where the plugin was not receiving events on device admin changes (created/deleted or configuration changed). I the default of 10 minutes is rather aggressive - I am thinking I'll reduce it to hourly or even daily.
                • On plugin startup, it optimizes the history table, which on a system with 1696 devices (wow!) that may be changing frequently, it will likely take a considerable amount of time. I will make a change so that this process occurs on the background and does not block initialization of the plugin.
                Temporarily, a few tweaks we can try specific to your system:
                1. Disable the periodic refresh of devices. You can do that by changing the file under Config\hsbuddy.ini. Look for the setting RefreshPeriodInMinutes under the [Devices] section. Set it to 0 to disable this background process. A restart of the plugin is needed for this to take effect.
                2. If you do not use the device history feature in the app, you can disable the use of the history table on the plugin. You can do this from the Plugin configuration page, just turn off the History feature. A restart of the plugin is not needed for this change.
                I disabled the background process, and as expected, this made a pretty significant difference in the CPU usage. After the PI startup, you get the list of processes seen below, none of which seem to hold the processor at 100% for long. Almost any activity at all will push the S6 to 100% CPU, so this is pretty significant. I also monitored the event cache using the tenHSevents script, and without the Refresh Period running, cache remained much more manageable, and did not build as before. I saw a high mark of 13 events cached; with HSBuddy set as before, this number would reach as high as 150 events and would not fall back until HSB released the processor. I already had Device History disabled before, in an attempt to reduce overhead. I think it helped a little bit, but the RefreshPeriod disablement was far more profound.

                I am betting that with a high-powered CPU, such as the NUC with 8 i7cores, a lot of these things would not be an issue. The S6 is too anemic, especially with a large and very complicated setup like mine, to deal with the overhead. I am hoping in the next few weeks to be able to upgrade my controller.

                I have been monitoring CPU usage of the HSB executable using the Jon00 Process Monitor graphing. The reduction since disabling the background process is significant. Yesterday, logs showed average CPU usage of about 13% overall. Now, after disablement of the background process, average CPU usage is about 0.8% for the HSB plug-in. I would assume this would increase somewhat if there was active geotracking going on, but this has definitely helped.

                Click image for larger version

Name:	image.png
Views:	239
Size:	39.5 KB
ID:	1570533

                Comment


                  #9
                  Here's an example of the HSB Plug-in CPU usage, as tracked over 24 hours:

                  Click image for larger version

Name:	image.png
Views:	145
Size:	21.9 KB
ID:	1570538
                  You can see when the disablement occurred, and the PI was restarted. ​

                  Comment


                    #10
                    Thanks for the update bebaldin - one thing to look out for, after you make changes to devices (add/remove/update - anything except for status/value, like location) it would be good to double check if HSBuddy reports the update or does it remain stale. I'm hoping it catches the update as it is subscribed to receive those events, in which case disabling the background refresh should have no functional impact.

                    Comment

                    Working...
                    X