Announcement

Collapse
No announcement yet.

100% Processor Occupancy & sluggish & unresponsive site. Any ideas?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    100% Processor Occupancy & sluggish & unresponsive site. Any ideas?

    A site I am responsible for is frequently experiencing 100% processor occupancy along with very slow response times. In the attached diagram you can see CPU usage is 133%, meaning it is overtaxed.

    In looking at Resource Monitor in Windows the "HSPI_ZWave.exe" is consuming 67% of the computer and "HSPI_MyQ.exe" is consuming 15% of the processor, totaling 82%.

    1) MyQ has a polling interval of 10,000 ms (10 Seconds) so I am surprised it is consuming so much CPU time (15%)

    2) The site has 6 Z-Net Z-Wave interfaces, which is a lot. But really? Consuming 67% of a 2Ghz machine with 16GB memory. I wouldn't think that talking to Z-Net interfaces consumed 11%~12% of the CPU, each.

    Any thoughts, suggestions, or ideas on how to get decent performance?

    System Stats:
    Current Date/Time: 2019-08-31 22:41:54
    HomeSeer Version: HS3 Pro Edition 3.0.0.548
    Operating System: Microsoft Windows 10 Pro - Work Station
    System Uptime: 4 Days 4 Hours 57 Minutes 59 Seconds
    IP Address: 10.10.10.30
    Number of Devices: 678
    Number of Events: 209
    Available Threads: 400
    HSTouch Enabled: True
    Event Threads: 0
    Event Trigger Eval Queue: 0
    Event Trigger Priority Eval Queue: 0
    Device Exec Queue: 0
    HSTouch Event Queue: 0
    Email Send Queue: 0
    Anti Virus Installed: Windows Defender
    In Virtual Machine: No MFG: intel corporation
    Enabled Plug-Ins
    3.0.27.0: BLLock
    3.9.8.1: BOSESoundTouch
    1.3.7.3: Device History
    3.0.0.65: EasyTrigger
    1.2019.211.1740: MyQ
    3.0.1.252: Z-Wave

    Click image for larger version

Name:	Pegged CPU.jpg
Views:	502
Size:	107.8 KB
ID:	1323713






    #2
    Do the HS3 logs give any insights? Maybe a device or Z-Net controller that is freaking out and HomeSeer workstation is struggling to keep up with all the events it causes to fire off.

    Had a slight logical error in a simple event myself that caused it to fire off in a constant loop with multiple actions per second that caused a spike in CPU usage. That was on a very small one with a few devices, so I can only imagine the impact on a large structure as you described.

    Jon00PerfMon plug-in should give you nice insights in system resource usage within HomeSeer so it makes it easier to debug.

    Otherwise trial and error maybe to power down all the Z-Net controllers, except one, and then add the others slowly while monitoring CPU until you find the controller that causes it to spike and then narrow things down further.

    Comment


      #3
      Try restarting HS3 and it should drop back down. Have you recently used Z-Tool by any chance?
      💁‍♂️ Support & Customer Service 🙋‍♂️ Sales Questions 🛒 Shop HomeSeer Products

      Comment


        #4
        1) How does "recently using z-tool" apply? Does z-tool mess up HS in some way? And, by "recently" do you mean within the past week?

        2) I have repeatedly power cycled the computer, and the problem returns.

        3) The logs show nothing unusual. No errors, no complaints, and a couple of entries every minute.

        4) I don't think it is one specific Z-Net that is the problem. Disabling them selectively doesn't fix the issue.

        Comment


          #5
          Z-Wave plugin losing it's mind and going CPU crazy has been an issue for at least a couple of years. Several have reported similar symptoms. This is what I've observed:

          1) After access of certain functions on the ZWaveControllers page (Plugins->Z-Wave-> Controller Management), after a delay of 15-20 minutes, CPU usage jumps to 50%. My system is a HomeSeer Hometroller S6 (Windows 7) with two processors, so I assume one CPU is tied up full time.

          2) The "certain functions" I've seen issues with are: add node (from web page or Z-Tool+), perform a backup. Possibly remove a node as well, not positive.

          It's difficult to get handle on this due to the delay from the action until the CPU shoots up. (Its pretty much guaranteed it will lose its mind after these functions, but the length of delay is difficult to quantify). Also of interest, for me I don't usually notice a performance hit, presumably since the CPU load doesn't go above 50%.

          3) Couple of ways to cure this: restarting the plugin of course, or (this one blows my mind) simply visiting the ZWaveControllers page from the browser ...

          Comment


            #6
            SeattleDavid I had a sluggish system years ago running a specific application, and could not pin-point source, because Task Manager would not show the culprit in the list of tasks using CPU cycles and causing interference with the respective app shooting up in CPU usage as well.

            SysInternals tools Process Monitor and Process Explorer (owned by Microsoft now, and available @ https://docs.microsoft.com/en-us/sys...cess-utilities) finally allowed me to narrow the root cause. In my case it turned out to be false-positives with Microsoft Windows Defender at the time. Excluding the respective folders was not enough by itself, but also excluding the respective process finally solved all the CPU spikes.

            Not sure if you can easily run those tools on a Hometroller S6 though, but it is a quick way to narrow things down as to what is causing the CPU spikes.

            Comment


              #7
              hi i have similar issues, but when i close down all plugins hs3 is eating the cxpu see: https://forums.homeseer.com/forum/ho...cpu-500-and-up
              Regards Bart
              ------------------------------------------
              Win7 64Bit on Intel NUCI7 with SSD
              HSPRO 3.
              Devices; 1370 Events; 691

              Jon00 Scripts, JowHue, HSTouch, Plugwise, Z-wave, Ultranetatmo, Ultracam, PHlocation, BLUSBUIRT, MeiHarmony, Buienradar, MEiUnifi Pushover 3P, Random, Nest HSPhone and Blueiris

              Visonic Powermax Alarm System (HS3) Interface: http://www.domoticaforum.eu/viewtopic.php?f=68&t=11129

              Comment


                #8
                I noticed over the last week or so that when I use Z-Tool to add/remove new devices that the Z-Wave plugin uses an inordinate amount of CPU. I restart HS3 after this and it's fine till I use Z-Tool again. I wasn't sure if it was just my system or if this could be your issue as well.
                💁‍♂️ Support & Customer Service 🙋‍♂️ Sales Questions 🛒 Shop HomeSeer Products

                Comment


                  #9
                  Originally posted by Rupp View Post
                  I noticed over the last week or so that when I use Z-Tool to add/remove new devices that the Z-Wave plugin uses an inordinate amount of CPU. I restart HS3 after this and it's fine till I use Z-Tool again. I wasn't sure if it was just my system or if this could be your issue as well.
                  This issue has been experienced and reported many times by others and it has not yet been addressed.
                  HS 4.2.8.0: 2134 Devices 1252 Events
                  Z-Wave 3.0.10.0: 133 Nodes on one Z-Net

                  Comment


                    #10
                    Unfortunately it needs to be reported more often or this may never get fixed.
                    💁‍♂️ Support & Customer Service 🙋‍♂️ Sales Questions 🛒 Shop HomeSeer Products

                    Comment


                      #11
                      Why is HS so full of these types of issues and bugs? It is as if the dev team is swamped, or just doesn't care about the last 10%. That attitude is fine when the customers are the hacker community. But it makes HS an inappropriate product for use in an enterprise application. And, it wastes a lot of people's time.

                      A little glitch like this can mean that a hydronic heating system doesn't shut off, and that can cost $1,000 in gas. But HS doesn't see this cost and so doesn't care. There is a disconnect when a company doesn't feel the pain of its users.

                      Comment


                        #12
                        Originally posted by SeattleDavid View Post
                        Why is HS so full of these types of issues and bugs?
                        Rupp just disclosed the reason above - HS must have a ranking system so a bug isn't a bug unless it is reported by a metric crapload of customers. That does align with comments all over this forum about endless customers being told they are the only one with identical bugs, or that a customer should simply rebuild because of a HS problem, heck let's be entirely honest here - when you scroll back through old posts here you find the same bugs being complained about YEARS ago and we all continue to run into them.

                        HomeSeer's approach to bugs remind me of a particular DeLorean vendor's approach to faults. Every customer that made a complaint about a faulty part would be told "first time we've heard of that" and "we have thousands of customers using that without problem". People aren't stupid, for years the topic was argued in a community forum and it was obvious hundreds or even thousands of customers were having the exact same "you're the only one" response to every problem - not to mention the community would get to the bottom of a problem (a third party non-legitimate part made of insanely thin plastic and thus easily snapping for example) but years later that particular vendor still selling that faulty part and still telling customers their fault is the first he's heard of it. Ultimately the community revolted and universarlly stopped using that particular vendor - he since closed up because of "family" reasons but it was obvious in the community nobody was buying from him anymore.

                        Comment


                          #13
                          Tillsy not saying you are wrong, but I often call power company to report an outage, when after speaking to multiple neighbors they all went "well somebody must have reported it", until talking to power company and being told "you are the first to call in an outage, we will send somebody out immediately". Of course they could be lying, but other times I've gotten "we are already aware, thank you for calling" response.

                          From a company/developer point of view I also understand it, because difficult choices have to be made in adding new exciting features, fix bugs, or work on next version. Deal with that for work as well with companies thousand times the size of HomeSeer and some bugs are unresolved for years because something else keeps taking precedence, and bugs have to be "rallied" by clients, which is ridiculous when high partner fees are paid, but you get used to it. That's the one benefit of open-source is that if you don't like something you can fix it yourself, or pay somebody to fix it for you. For closed source solutions such as HomeSeer you have to go through the process, but it is at least designed very openly allowing plugin developers to fill in gaps. HomeSeer development team probably runs a fine balance where some features might be a few hours of work with guarantees to make 80% of users happy, other bugs might take days/weeks to debug with no guarantee it will fix it and only satisfy 0.1% of users experiencing the bug, guess which work gets prioritized?

                          Even companies like Microsoft with tens of thousand of support staff members around the world relies on community forums that they do not always monitor themselves, because it becomes a daunting task. With HomeSeer users around the globe this forum is almost a 24/7 support system as often you will come across a user that has already fixed the problem you are experiencing and is willing to share with you how they did it. This can be a response within minutes, and it can take hours or even days.

                          The big thing missing is a bug tracker that allows proper up-voting on bugs, so that HomeSeer can quickly keep track of bugs with the highest impact, but the problem is for that to be effective it would require a large population of users to vote. Microsoft does the same, and the highest vote items are sometimes still not implemented, but each version other sparkly new features get added that nobody asked for.

                          Bugs will always happen, but it is how you deal with them that matters, and hopefully HomeSeer will not become the same as your DeLorean vendor

                          PS: For clarification, this is what Microsoft uses for a bunch of their apps: https://blogs.technet.microsoft.com/...gestion-boxes/

                          Comment


                            #14
                            I've worked for companies using both philosophies.

                            One company fixed the big bugs and just basically wrote off the outlier customers, arguing it was never cost effective to fix those issues. That company operated in fire fighting mode all the time, every fix was a haphazard bolt on, and there was always this rumbling from the 10% of the customers that experienced issues. Better yet, the 15% of customers that paid for and just quit using the product because of bugs was an incentive to have so many bugs. ("Customers who paid for our product and then abandoned it were our most profitable clients".)

                            I switched to a second company. They took the philosophy that every bug should be fixed, everything should be organized, and that having a stable codebase was actually less expensive in the end. It seemed like it was a much larger investment to address every bug or compatibility issue...having come from the first type of company I didn't see how it would be economically possible to achieve such high reliability.

                            After working for company #2 (the focus on reliability company) I saw the light. That company grew wildly and in unpredictable big spurts. The story would go like this: A customer would try our product, find it worked, would be amazed at the quality difference, and a year later they would order ten and then a hundred of our systems.

                            Reliability and works everywhere meant that:
                            1. It was an easy choice to make a purchase decision.
                            2. It was an obvious choice for future applications.
                            3. Word got around to other customers.
                            4. We were able to leverage growth because when sales scaled up the support didn't eat us alive.
                            5. Bit clients did big deals, and they don't want to mess around with quirky stuff.
                            It was almost a self-fulfilling prophesy that we succeeded because we succeeded.

                            It had a major influence on my career in a good way. Now I work for a family owned company with a simple philosophy: Do it right, clean, and with an eye on long-term maintainability. It's a fun job.

                            I think working for Company #1 (everything is a little broken, but not broken enough to die from) eventually leaves everybody feeling defeated, sloppy, careless, and wondering why they just never catch a break.

                            Its really just a mindset.

                            I think the way that HS operates is just fine for a "hacker" community of users. It's even fun for the hackers to know the quirks and to be able to navigate around them. But I also think this misses the professional and mainstream market.

                            Comment


                              #15
                              I've occasionally experienced this problem of high processor usage too and have wondered if it isn't HS written code but the third-party / open source libraries they use. Many of them seem a bit outdated. For example, it appears they are still using a 2016 version of SQL Lite (I noticed a 1.0.87.0 version number on the SQLite.Interop.dll library) and many fixes and performance updates have been added since then (see https://system.data.sqlite.org/index.../www/news.wiki). I have written a homekit plugin for Homebridge and also notice a big spike in processor usage when a JSON query is made to HomeSeer - I similarly wonder about their JSON parser library (Newtonsoft version 8.0.3.19514 from 2016 while current version is 12.X.X and also claims numerous performance improvements). Perhaps this could be as simple as using third-party libraries that are a bit out of sync with Windows code and no longer performs well under the current Windows (and Linux?) code. I am hoping to see a HS4 with some of these libraries brought up to date.

                              Comment

                              Working...
                              X