Announcement

Collapse
No announcement yet.

Increasing Handles and HS Server crashes

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    Handles are a way the os manages resources. An example would be when one opens a file the os provides a handle to track it. Handles are used all over. When an app forgets to tell the os it’s done with a handle, it’s called a handle leak. There are tools to show what handles represent. Running one should help you narrow down what’s causing it.

    Comment


    • #17
      Originally posted by jono View Post

      Thank you for commenting Jon. I am loading 4 files with the aspx loader. They are
      Jon00DeviceList.aspx
      jon00eventlist.aspx
      Jono00inieditor.aspx
      BLDSC_log.aspx

      The last one is the log for my alarm system (BLDSC plug in). It always takes a long time to load, so I thought the Jon00AspxLoader would help. Perhaps that is causing problems.
      Before seeing your note, I turned off the hourly Jon00AspxLoader and the slope of my handles graph flattened considerably. I will turn Jon00AspxLoader back on but remove BLDSC_log.aspx from the aspx loader and see if the graph still stays flat.
      Many thanks
      jono
      Let me explain. When you call an ASPX web page in HomeSeer, it is first compiled and then displays the page. This is cached in memory for a period and if not used will be released back into the memory pool by the .NET memory management process. This release time is not fixed and depends on memory requirements for other applications running. The code compilation is what takes the time for the page to display. Once run, this page will use memory and handles like any other plugin or script.As Bill states, if used handles are not released, you get a handle leak. It may be nothing to do with the programming within the ASPX code and could be a particular HS3 plugin/script command which is not releasing its handles when called. It could also be a bug with the HS3 Web server which is not releasing handles when running .ASPX web pages.

      By running the loader, you are guaranteed fast page loads, however any handle leak caused by running an ASPX page will be accelerated. If you run my loader without any pages defined, I expect you will see very little change in handle increase.



      Jon

      Comment


      • #18
        Update: My handles count appears to have leveled off with WeatherXML disabled, but the time span is too short to draw a firm conclusion. Will continue to monitor.
        Mike____________________________________________________________ __________________
        HS3 Pro Edition 3.0.0.548

        HW: Stargate | NX8e | CAV6.6 | Squeezebox | PCS | WGL 800RF, Rain8Net+ | RFXCOM | QSE100D | Vantage Pro | Green-Eye | X10: XTB-232, -IIR | Edgeport/8 | Way2Call | Ecobee3

        Comment


        • #19
          I was thinking of making a post about my "resource usage" findings over a relatively long HS3 uptime when I saw this thread so thought I would contribute.

          My live HS3 system specs on which the info in this post is based are:
          Hardware: Intel DN2800 solid state board with Samsung SSD, 2GB RAM
          OS: Windows 7 Pro SP1 (x86)
          .Net Framework 4.5.2
          HS3 v3.0.0.451
          Speaker.exe v3.0.0.10
          Plug-In Z-Wave v3.0.1.228
          Plug-In Sonos v3.1.0.24
          Plug-In RFXCOM v30.0.0.36
          Plug-In PHLocation v3.0.1.121
          Plug-In PHEvoHome v 3.0.0.37

          I have 3 other plug-ins which are my own creations but as they are not available for others to install they are irrelevant to this post.
          The machine is dedicated to HS3 so doesn't have anything other than the OS and HS3 + plug-ins running.

          The OS was re-booted on 15th January 2018. Yes, its a Windows machine and has been up for over a year!
          HS3 and all the above plug-ins was started on 7th June 2018 and have been running ever since. Yes, nearly 8 months of HS3 up time so far.

          Observations:
          There has been nothing strange WRT resource usage on any of the above plug-ins or Speaker.exe. All exe's have remained stable in terms of memory usage and handle counts etc for the whole 8 month up time.

          I do have a small "Key" handle leak in HS3.exe which has been there for years and is related to the Ivona voice that I use for HS speech. Every time HS3 speaks it generates a new Key handle which is never released. Not a big issue and I clean up the orphaned handles manually from time to time.

          The only other issue I see is a "Thread" handle leak in HS3.exe. Thread handles are increasing at a rate of approximately 350 per month. Again, not critical and the system could run for many months without this being an issue.
          I've not dug deeply into this leak but I suspect there is one or more threads somewhere in HS3 that are terminating and not fully cleaning up.

          I don't have any ASPX pages being called so can't comment if there may be an issue there.

          It's not an incredibly busy system with about 800 devices and 200 or so events.
          There are a handful of scripts being called quite regularly but these are all my own creations so I've tried to ensure they are well coded.

          Obviously every HS3 instance is different so my findings don't necessarily reflect what others may see however for my particular setup it is very stable.

          The plan is to let it run until something breaks.

          Paul..

          Comment


          • #20
            Try using this to narrow down what the handles are: https://docs.microsoft.com/en-us/sys...wnloads/handle

            Comment


            • #21
              Originally posted by bsobel View Post
              Try using this to narrow down what the handles are: https://docs.microsoft.com/en-us/sys...wnloads/handle
              I've tried that before. The handles are with the HS3 app and that tool does not provide any details as to what the handles are for except "threads" and "events".
              HS 3.0.0.548: 1990 Devices 1172 Events
              Z-Wave 3.0.1.262: 126 Nodes on one Z-Net

              Comment


              • #22
                Originally posted by sparkman View Post
                Just disabled X10, WeatherXML, ImperiHome and Restart. This is where I'm currently at:

                Click image for larger version Name:	Capture.PNG Views:	1 Size:	3.1 KB ID:	1281676
                Will post an update tomorrow morning. The handles dropped by about 1300 when I shut down those plugins.
                So 23 hours later and my handles are still going up at about the same rate as before:

                Click image for larger version

Name:	Capture.PNG
Views:	13
Size:	3.3 KB
ID:	1281928
                HS 3.0.0.548: 1990 Devices 1172 Events
                Z-Wave 3.0.1.262: 126 Nodes on one Z-Net

                Comment


                • #23
                  Originally posted by sparkman View Post

                  I've tried that before. The handles are with the HS3 app and that tool does not provide any details as to what the handles are for except "threads" and "events".
                  That’s actually a huge reduction in possible issues. Is it events or threads that is increasing?

                  Comment


                  • #24
                    Originally posted by bsobel View Post

                    That’s actually a huge reduction in possible issues. Is it events or threads that is increasing?
                    Not sure. I'll have to run it again a few times and see if it can give the counts just against HS3. The actual threads that jon00perfmon reports for HS3 are actually very low and stable:

                    Click image for larger version

Name:	Capture.PNG
Views:	13
Size:	4.3 KB
ID:	1281952
                    HS 3.0.0.548: 1990 Devices 1172 Events
                    Z-Wave 3.0.1.262: 126 Nodes on one Z-Net

                    Comment


                    • #25
                      Here are the counts for HS3 that handle.exe reports:

                      Nthandle v4.21 - Handle viewer
                      Copyright (C) 1997-2018 Mark Russinovich
                      Sysinternals - www.sysinternals.com

                      Handle type summary:
                      ALPC Port : 30
                      Desktop : 2
                      Directory : 8
                      EtwRegistration : 156
                      Event : 3338
                      File : 268
                      IoCompletion : 6
                      Key : 113
                      KeyedEvent : 2
                      Mutant : 57
                      Section : 75
                      Semaphore : 529
                      Thread : 9127
                      Timer : 7
                      Token : 103
                      TpWorkerFactory : 24
                      WindowStation : 4
                      Total handles: 13849


                      So Thread and Event are by far the largest component.
                      HS 3.0.0.548: 1990 Devices 1172 Events
                      Z-Wave 3.0.1.262: 126 Nodes on one Z-Net

                      Comment


                      • #26
                        When you look at the individual handles using the Sysinternal tool Process Explorer, what do you see? It should list the handles for each plugin as well as HS3. Might help isolate which one is running amok.

                        Z

                        Comment


                        • #27
                          Originally posted by vasrc View Post
                          When you look at the individual handles using the Sysinternal tool Process Explorer, what do you see? It should list the handles for each plugin as well as HS3. Might help isolate which one is running amok.

                          Z
                          These are all for the HS3 process itself. The handle leak is within HS3, the plugins that I'm monitoring don't seem to have a leak in their own processes, but may be causing a leak within HS3. The Sysinternal tools shows little to no information about each thread or event handle within HS3.
                          HS 3.0.0.548: 1990 Devices 1172 Events
                          Z-Wave 3.0.1.262: 126 Nodes on one Z-Net

                          Comment


                          • #28
                            That would make it more difficult then

                            So the HS3 handles are exceeding 3,000??
                            For what it's worth, I run many of the same plugins and my HS3 handles rarely exceed 1500.

                            Z

                            Comment


                            • #29
                              Originally posted by vasrc View Post
                              That would make it more difficult then

                              So the HS3 handles are exceeding 3,000??
                              For what it's worth, I run many of the same plugins and my HS3 handles rarely exceed 1500.

                              Z
                              Yup

                              I just shut down all my plugins and restarted HS. Here's the HS3 handle count, right after the restart:

                              Handle type summary:
                              ALPC Port : 23
                              Desktop : 2
                              Directory : 8
                              EtwRegistration : 136
                              Event : 785
                              File : 97
                              IoCompletion : 5
                              Key : 105
                              KeyedEvent : 2
                              Mutant : 55
                              Section : 53
                              Semaphore : 48
                              Thread : 182
                              Timer : 7
                              Token : 17
                              TpWorkerFactory : 24
                              WindowStation : 4
                              Total handles: 1553


                              I run many more plugins than what I posted above. That was just the ones that were the same as Uncle Michael. Here's my current list:

                              APCUPSD, BLLED, BLTVGuide, Chromecast ,DSC Security, EasyTrigger, Homekit bridge, IFTTT, ImperiHome, MQTT, MySensors, PHLocation2, Pushover 3P, Random, Restart, RFXCOM, SDJ-Health, SqueezeBox, Ultra1Wire3, UltraGCIR3, UltraLog3, UltraNetCam3, UltraPioneerAVR3, weatherXML, X10, Z-Wave, Jon00SpeedFanMonitor, Jon00ProcessMonHS3, Jon00DBChartingHS3, Jon00SNMPMonitor.

                              When HS3 starts with them all running, I exceed 3000 and they increase by about 1200-1500 per day.
                              HS 3.0.0.548: 1990 Devices 1172 Events
                              Z-Wave 3.0.1.262: 126 Nodes on one Z-Net

                              Comment


                              • #30
                                What arguments are you passing for handle64.exe so I see the same.

                                Z

                                Comment

                                Working...
                                X