Announcement

Collapse
No announcement yet.

3.0.0.22 constantly crashing HSConsole

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    3.0.0.22 constantly crashing HSConsole

    Spud,

    Looks like there is a fairly signifiant issue with the Autelis plugin version 3.0.0.22 on Linux. I have been fighting constant out of memory issue occurring in the HSConsole.exe application and have been turning off all my plugins and turning them on one by one. When I enable the current Autelis plugin my memory usage goes from about 9% to 20% pretty quickly and if I leave it running the system crashes and HSSentry restarts everything. This happens 2-6 times a day. The plugin also seems to go to very high CPU and takes quite a while to stop. Now ideally nothing a plugin can do should effect HSConsole like this (since they are separate processes), but perchance are you doing any work in any of the below calls (HSConsole loads the modules into its memory to call those, then unloads the app domain then launches the actual HSPI_plugin. So threads started (etc) in any of these calls wind up running inside of HSConsole on the first load, and again in your plugin on the real load).

    Name
    AccessLevel
    Capabilities
    comport
    supportsMultipleInstances
    supportsMultipleInstancesSingleEXE

    Rich is aware, I will ask him to reach out and see if the three of us can figure out why this is occurring and what we can do to ensure HSConsole at a minimum can't be crashed by a plugin.

    Help

    Best
    Bill
    Last edited by bsobel; September 14, 2018, 05:51 AM.

    #2
    Oh I forgot to mention the HSPI() constructor itself (it gets called as well during the load into HSConsole before the other API's are actually called)

    Comment


      #3
      I'm not doing any work in the calls you mentioned.
      I have no idea what the problem can be.

      Was it working better with a previous version of the Autelis plugin? a previous version of HS3?

      Comment


        #4
        Yes. Been running the plugin for over a year and 435 for awhile. Problem started recently. Not exactly sure when as hsdentry masked the issue until I realized just how often it was happening. Very very odd. Understood no work in those calls. How about the Hspi constructor or any static classes?

        Comment


          #5
          HSPI default constructor, and no static classes
          All my plugins use the same base code, and you're the first to report this problem so far, so it's really weird.

          Comment


            #6
            Originally posted by spud View Post
            HSPI default constructor, and no static classes
            All my plugins use the same base code, and you're the first to report this problem so far, so it's really weird.
            I suspect it is something environmental as well (no idea what yet), it is very similar to the issue reported with your Nest plugin (when I contacted HS support, they pointed me to that issue as an example). I know in that case it turned out to be a HS bug, but wasn't clear from the thread (like in this case) what in the plugin caused it (if anything). Happy to test/debug here, pretty unclear where to start on this however, thoughts?
            Last edited by bsobel; September 14, 2018, 12:53 PM.

            Comment


              #7
              Spud, forgot to mention I am also running EasyTrigger which is not showing this behavior so that (to me) suggests its not something from the shared code base, but something specific to the Autelis functionality.

              Comment


                #8
                In the plug-in config if you set the poll interval to 0 so that the plugin does almost nothing, do you still see this behavior?

                Comment


                  #9
                  Off to test, will report back.

                  Comment


                    #10
                    Hi Spud,

                    Below is the status from today. Text summary:

                    After a clean boot at 930am HSConsole.exe was using 5.8% of memory as reported by the linux top command. I enabled BLLock (one other plugin that was slowly being added back in, memory moved to 6.7%). After your post I started the Autelis plugin. Memory jumped to 14% (most plugins spike a few percentage points but not 7%). I set the poll interval to 0. Ran the system from 11:15am until 3:56 pm, memory floated in the 14-16% range.

                    Around 2pm since the system appeared stable with 0 interval, I wrote an event to call the Poll for Status action every 15 seconds (thinking if polling was related, perhaps this would quickly show it). That ran from 2pm until 3:55. System seemed normal. Deleted event and changed Autelis back to polling every minute. Memory has slow risen from 16% to 20.4% right now. HS crashes when it gets to the upper 20's or so. It does seem enabling polling is causing a small leak, but calling the polling action does not. I do have events tied to changes in some of your devices, so I also set a reoccurring timer to call the script it runs repeatedly, no memory leak detected. Also the poll for status would have resulted in those devices firing the same as if the Autelis was polled by your setting, so that currently doesn't seem a likely candidate to look into.

                    Rich did mention an issue with timers and mono, but AFAIK that was introduced in 5.4 (and later fixed) but I am on 5.0.1.1 like most on 5.x because its the only 5.x version where ASP pages in HS work. Also a timer bug (I would think) would effect your HSPI_Autelis process and not HSClient.

                    I do see two 404's and related exceptions in my log, but since forcing a poll didn't cause a leak I'm not seeing an obvious connection (also would it be possible to configure these errors to not report in the future?)

                    Click image for larger version

Name:	Screen Shot 2018-09-14 at 4.19.16 PM.png
Views:	69
Size:	60.8 KB
ID:	1247438

                    Curious, to you develop in VB or C#? Shouldn't matter, but just trying to find things to research to try and figure out whats going on.

                    Bill


                    Notes:
                    Booted around 9:30am

                    10:51 5.8
                    Starting BLLock
                    10:56 6.7
                    Starting Autelis to change poll interval to 0 for Spud
                    11:15 14.0
                    11:41 13.2
                    12:06 13.5
                    12:43 13.7
                    12:56 14.1
                    13:25 14.2
                    14:28 16.1
                    15:56 15.1

                    Ran with Autelis Poll command every 15 seconds, fine
                    15:58 Set poll for status back to 1 minute from 0
                    16:00 16.5
                    16:17 17.1
                    16:23 17.7
                    16:28 18.3
                    16:34 18.6
                    17:50 20.2

                    Comment


                      #11
                      Spud,

                      An update; I think your plugin actually is not at fault directly. I changed HS to go back to the original way of loading plugins and found that the issues attributed to the plugin stopped. I realized a bunch of plugins were crashing in different ways (or showing odd behavior) with the load plugins in background option set (another example, the Pushover plugin would have an exception in the log and wouldn't work). For some reason in this mode Autelis causes HS to chew them memory. Only theory I have is its related to the OSS thread (I debugged the code trying to figure this out), but since the problem stops when loading plugins traditionally, it feels like HS may be starting them in the background a bit too early in its own startup. I'll reach out to Rich about this and update.

                      Once I realized this and set plugins to load traditionally, the ongoing memory leak seems to be in one of Blade's plugins (BLradar).

                      Best,
                      Bill

                      Comment


                        #12
                        P.s. Nice Trailer Park Boys reference ,)

                        Comment


                          #13
                          One more issue, albeit unrelated. On my system Auetelis often spikes to over 100% cpu usage, overall its the second most cpu intensive plugin (after Zwave). None of my others are even close, any chance there is an issue causing higher than expected CPU? Given I'm polling once per minute, I'd expect a spike for a second or two during the xml processing, but nothing like the numbers I usually see...

                          Comment

                          Working...
                          X