Discussions related to HomeSeer Beta builds 3.0.0.369->3.0.0.423

[concordseer]

I'm thinking now there's gonna be so many different people on so many different versions. A case of stick to whatever works for ya.

As Pete says .318 was the last stable version I'm aware of but had HSTouch as a plugin and not integrated. I too wasn't paying too much attention to possible memory leaks in that version either.

Maybe a more productive exercise here would be to find out who is NOT having problems with HS3 and drill down from there.

[randy]

@Randy, Looking at your particulars it appears that your using v.379 and not v.383 is that correct?

Not that I've been using HS3 for years (Jun of this year), I would also say that I too never experienced a crash, until Beta v.382. I'm almost tempted to go v.368 at least I knew that it had a memory leak. Mike

I ran 382 for 3 days and 384 for 2. Neither was different in terms of stability and memory. I went back to 379 last night because my log was full of queue warnings due to a change Rich made. I haven’t had time to dig into the warnings, but they were for 4 plug-ins about every 2 minutes. The warnings were for BLStat, PHLocation, Z-Wave and one other that I don’t recall. Despite the warnings, the system still ran fine.

[randy]

@Randy ..what does your About page show relating to memory and plugin utilization?

Your 800 available threads indicates you are running W10 Pro 64 bit with 8 cores (VM?) - guesstimate from seeing 400 for 4 cores and 200 for 2 cores.

IE: here my about page only shows Homeseer memory used but not plugins in Linux...this is where I utilize htop.

Homeseer V.383
HomeSeer Memory Used: 120 Mbytes
Plug-In Memory Used: 89, 135, 48, 85, 60 Mb's
Total = 537 Mb used for Homeseer.




111 Mb is really nothing compared to total of 2Gb of RAM or 16Gb of RAM

Each plugin utilizes more memory.

Adding these up I am still under 1Gb of memory. That said with an RPi2-3 hitting the 1Gb mark could probably debend HS3.

It is a Xeon E3-1230 (4-core hyper-threading) physical server. It hosts Vivotec VAST, Ubiquiti UniFi controller, No-IP DUC (DDNS Dynamic Update Client) and HS3. Vivotec uses about 1-2% processor and about 750mb memory, UniFi only uses about 200mb of memory, virtually no processor and the No-IP client is all but nonexistent in its footprint.

[happnatious1]

It is a Xeon E3-1230 (4-core hyper-threading) physical server. It hosts Vivotec VAST, Ubiquiti UniFi controller, No-IP DUC (DDNS Dynamic Update Client) and HS3. Vivotec uses about 1-2% processor and about 750mb memory, UniFi only uses about 200mb of memory, virtually no processor and the No-IP client is all but nonexistent in its footprint.

[ATTACH]64562[/ATTACH]

Randy,

I'm running Homeseer on a virtual machine as well. ESXI 6.5, 8 virtual cores and 12gig of memory on Ubuntu 16. It doesn't crash, and I really haven't noticed any memory usage increase. But I do have a thread that will lockup, and I notice it on the overall CPU monitor screen for the Homeseer virtual machine. When I first start my VM average usage will be less that 5%. When the thread locks up it will jump to 19% and stay there indefinitely. It will take anywhere from 3 hours to one week to happen, so pretty random. When I first noticed the problem my CPU was over 30% so I believe the threads are cumulative however, that was several version ago. I also believe it started with the move to not run HStouch as a plugin but I have no data to substantiate that. I also believe that it has something to do with the API as Ive seen restarting plugins fixing it sometimes and sometimes not.

[MNB]

Question to the community in regards to HS Log, I was thinking wouldn't it be practical to have periodic (every X minutes), Metrics re. HS3 Memory used, Threads, handles and CPU % which would create a consistent rabbit trail as the HS log logs the various activities, warnings and cautions ect. one could look for trends and/or anomalies that could be quickly triaged. The format could be four extra columns that the values would be recorded along with the appropriate log traffic. Mike

[happnatious1]

Question to the community in regards to HS Log, I was thinking wouldn't it be practical to have periodic (every X minutes), Metrics re. HS3 Memory used, Threads, handles and CPU % which would create a consistent rabbit trail as the HS log logs the various activities, warnings and cautions ect. one could look for trends and/or anomalies that could be quickly triaged. The format could be four extra columns that the values would be recorded along with the appropriate log traffic. Mike

A simple script that runs every minute that logs as many details as it can based on which operating system it's running on.

Sounds like a fun project for mrHappy.

[randy]

I use Jon00’s PerfMon for current values and a 24 hour snapshot. I use Jon00’s database charting script to store the values from PerfMon for the longer term. Both the PerfMon devices and the database are updated every 5 minutes. It took all of 10 minutes to set both up.

[TechFan]

I looked into using that as well. . .but it looks to be windows only.

[fvhemert]

So I believe your problem is the plugin callback queue, it has a really high value (should be 0). For some reason callbacks are being put in the queue and are not being handled. I have to study the code, but it would appear that a plugin was called with an event and the plugin did not return. This is allowing the queue to fill and not empty so no plugins are getting called with event info. I guess I need to figure out a way to handle this, maybe thread the callbacks and stop calling a plugin that is not returning.

Maybe you could disable your plugins and run one at time and monitor that queue and see which plugin is causing this.

Because I am experiencing a very slow and unresponsive system I started investigating what could be the causing this.

I am running .379 and checked my Plugin Callback Queue, and noticed a very high number:

System Uptime: 0 Days 2 Hours 30 Minutes 10 Seconds
Plugin Callback Queue: 25823

I did a restart for HS3 and the number dropped to 39 right after HS3 was fully started:

System Uptime: 0 Days 0 Hours 0 Minutes 34 Seconds
Plugin Callback Queue: 39

After waiting another minute:

System Uptime: 0 Days 0 Hours 1 Minute 2 Seconds
Plugin Callback Queue: 0

But the number slowly increases every minute.

System Uptime: 0 Days 0 Hours 7 Minutes 10 Seconds
Plugin Callback Queue: 127

After 10 minutes the situation degraded even further:

System Uptime: 0 Days 0 Hours 10 Minutes 39 Seconds
Plugin Callback Queue: 1358

I then disabled ALL plugins and the queue was empty again:

System Uptime: 0 Days 0 Hours 15 Minutes 1 Second
Plugin Callback Queue: 0

Testing all plugins one by one:
11:18 Enabled the Arduino PI -->PI Callback Queue increase and decrease on frequent intervals. Seems to be returning to 0
11:21 Disabled the Arduino PI
11:21 Enabled the NetCam PI -->PI Callback Queue stays at 0
11:24 Disabled the NetCam PI
11:24 Enabled the PushOver PI -->PI Callback Queue stays at 0
11:26 Disabled the PushOver PI
11:26 Enabled the Restart PI -->PI Callback Queue stays at 0
11:29 Disabled the Restart PI
11:29 Enabled the SqueezeBox PI --> PI Callback Queue stays at 0
11:32 Disabled the SqueezeBox PI
11:32 Enabled the WeatherXML PI --> PI Callback Queue stays at 0
11:34 Disabled the Weather XML PI
11:34 Enabled the A10 PI --> PI Callback Queue stays at 0
11:39 Disabled the A10 PI
11:40 Enabled the Z-Wave PI --> PI Callback Queue stays at 0

Conclusion: behavior does not seem to be related to just 1 plug in

Adding plugins one by one
11:42 Enabled the Arduino PI -->PI Callback Queue increase and decrease on frequent intervals. Returning to 0
11:47 Enabled the WeatherXML PI
11:49 Enabled the NetCam PI
11:51 Enabled the A10 PI
11:53 Enabled the Restart PI
11:57 Enabled the SqueezeBox PI
12:01 Enabled the PushOver PI

I do no longer see the increase in the PI Callback queue. The interesting thing is all plugins are running just like before but now there is no permanent increase in the queue.

With all the plugins running I see the queue value go up to sometime 300+ but after a few seconds it will go down again and slowly reach 0.

I found there is a direct relation between the sluggish behavior I am experiencing and the number of items in the queue.

If the number is high:
- Clicking a button in HSTouch will work but the device change is executed after several seconds
- The status for the device in HSTouch is update with several seconds delays
- When changing a device in the web interface, the command is executed after several seconds
- The status in the web interface is updated after several seconds.

For now I believe we can conclude the following:
- A high number for the PI Callback Queue severely slows down the system
- Impact is noticeable in HSTouch, the web interface but also the execution of events. It's an overall system slowdown
- There is no direct relation between the individual plugins and the behavior observed
- This behavior is introduced in one of the latest (beta) releases since my hardware has not changed, additional plugins were not installed and the system had no dramatic configuration changes.

My questions:

• How has the internal processing changed to cause the overhead that is causing this?
• Is there a tool that can be used to view the items in the PI Callback Queue?

[Pete]

There are a bunch of opensource linux utilities which graph utilization. A few have been posted here on this thread.

For Linux boxes here have mentioned the use of Webmin. Webmin has a few monitoring plugins which do graphing et al.

You can go easy button here by just configuring a remote ssh session with HTop on a remote desktop and leave it running just to look at Homeseer 3 Mono processes / CPU utilization and total RAM utilization.

[rjh]

For Windows, there are 2 calls you can make in a script:

hs.HSMemoryUsed
hs.HSThreads

Question to the community in regards to HS Log, I was thinking wouldn't it be practical to have periodic (every X minutes), Metrics re. HS3 Memory used, Threads, handles and CPU % which would create a consistent rabbit trail as the HS log logs the various activities, warnings and cautions ect. one could look for trends and/or anomalies that could be quickly triaged. The format could be four extra columns that the values would be recorded along with the appropriate log traffic. Mike

[rjh]

Please try build 383. We moved each plugin callback to its own thread. This way if one plugin is not responding, all other plugins will continue to run. It will log a message with the plugin name of any plugin that is not responding to an event callback.

Because I am experiencing a very slow and unresponsive system I started investigating what could be the causing this.

I am running .379 and checked my Plugin Callback Queue, and noticed a very high number:

System Uptime: 0 Days 2 Hours 30 Minutes 10 Seconds
Plugin Callback Queue: 25823

I did a restart for HS3 and the number dropped to 39 right after HS3 was fully started:

System Uptime: 0 Days 0 Hours 0 Minutes 34 Seconds
Plugin Callback Queue: 39

After waiting another minute:

System Uptime: 0 Days 0 Hours 1 Minute 2 Seconds
Plugin Callback Queue: 0

But the number slowly increases every minute.

System Uptime: 0 Days 0 Hours 7 Minutes 10 Seconds
Plugin Callback Queue: 127

After 10 minutes the situation degraded even further:

System Uptime: 0 Days 0 Hours 10 Minutes 39 Seconds
Plugin Callback Queue: 1358

I then disabled ALL plugins and the queue was empty again:

System Uptime: 0 Days 0 Hours 15 Minutes 1 Second
Plugin Callback Queue: 0

Testing all plugins one by one:
11:18 Enabled the Arduino PI -->PI Callback Queue increase and decrease on frequent intervals. Seems to be returning to 0
11:21 Disabled the Arduino PI
11:21 Enabled the NetCam PI -->PI Callback Queue stays at 0
11:24 Disabled the NetCam PI
11:24 Enabled the PushOver PI -->PI Callback Queue stays at 0
11:26 Disabled the PushOver PI
11:26 Enabled the Restart PI -->PI Callback Queue stays at 0
11:29 Disabled the Restart PI
11:29 Enabled the SqueezeBox PI --> PI Callback Queue stays at 0
11:32 Disabled the SqueezeBox PI
11:32 Enabled the WeatherXML PI --> PI Callback Queue stays at 0
11:34 Disabled the Weather XML PI
11:34 Enabled the A10 PI --> PI Callback Queue stays at 0
11:39 Disabled the A10 PI
11:40 Enabled the Z-Wave PI --> PI Callback Queue stays at 0

Conclusion: behavior does not seem to be related to just 1 plug in

Adding plugins one by one
11:42 Enabled the Arduino PI -->PI Callback Queue increase and decrease on frequent intervals. Returning to 0
11:47 Enabled the WeatherXML PI
11:49 Enabled the NetCam PI
11:51 Enabled the A10 PI
11:53 Enabled the Restart PI
11:57 Enabled the SqueezeBox PI
12:01 Enabled the PushOver PI

I do no longer see the increase in the PI Callback queue. The interesting thing is all plugins are running just like before but now there is no permanent increase in the queue.

With all the plugins running I see the queue value go up to sometime 300+ but after a few seconds it will go down again and slowly reach 0.

I found there is a direct relation between the sluggish behavior I am experiencing and the number of items in the queue.

If the number is high:
- Clicking a button in HSTouch will work but the device change is executed after several seconds
- The status for the device in HSTouch is update with several seconds delays
- When changing a device in the web interface, the command is executed after several seconds
- The status in the web interface is updated after several seconds.

For now I believe we can conclude the following:
- A high number for the PI Callback Queue severely slows down the system
- Impact is noticeable in HSTouch, the web interface but also the execution of events. It's an overall system slowdown
- There is no direct relation between the individual plugins and the behavior observed
- This behavior is introduced in one of the latest (beta) releases since my hardware has not changed, additional plugins were not installed and the system had no dramatic configuration changes.

My questions:

• How has the internal processing changed to cause the overhead that is causing this?
• Is there a tool that can be used to view the items in the PI Callback Queue?

[MNB]

@fvhemert silly question I do not see where I can view my "plugin callback queue" the only items I see (Win 10) are the following in the "about" tab;

Event Threads: 1
Event Trigger Eval Queue: 0
Event Trigger Priority Eval Queue: 0
Device Exec Queue: 0
HSTouch Event Queue: 0
Email Send Queue: 0

@randy, I too use Jon's scripts but as you indicated they are typically 5 minute snapshots which is good for looking for long term trending but they are not tied to The HS Log activities which are continuously updating (near real time) where as the 5 minute snap shot can/do miss intermittent anomalies.

I'm finding that if I don't do anything with my system (just let it run basic scripts) and remove a series of scripts (Morning Announcement) my system seems to run, but like @Adam (mrHappy) if we do something that stress the system in some manner (to be determined how?) we get the dreaded "clr.dll crash" which is repeatable. I was thinking that a more robust HS Log should easily capture these metrics as their currently being captured by the system instead of trying to use workaround solutions (obviously one should be able to make them available for viewing by user selection so that if you don't want to see them you don't but they are being captured). Mike

[fvhemert]

For Windows, there are 2 calls you can make in a script:

hs.HSMemoryUsed
hs.HSThreads

Is it possible to create HS3 perf counters in Windows that receive their data from HS3?

Using perfmon it is really easy to monitor counters and correlate data and it comes pre-populated with relevant windows counters (CPU/Disk/Memory etc.).

[fvhemert]

@fvhemert silly question I do not see where I can view my "plugin callback queue" the only items I see (Win 10) are the following in the "about" tab;

This is funny

Following Rich' ask I have updated HS3 to .383 and now the "plugin callback queue" is no longer listed....it disappeared from the overview that I normally accessed through Tools -> About Homeseer --> Display Details for Support (button half way down the page)


@Rich: I am still testing and monitoring but believe the changes in the way the plugin call back's are handled is working. Allthough I can no longer check the queue, the system seems more responsive. I'll keep you updated. with more results.

Is there another way to check the queue or has this now disappeared with the changes to separate threads?