Out of Memory. Crashing HS3 - Solution!

[rjh]

I have yet to be able to recreate this so I need help. Can you try disabling all plugins, see if it fails and if not, then start enabling them? Also, are you running any scripts?

I may have to add some internal diagnostics that can detect when a thread is not sleeping or an external API is accessed too often.

Spud just sent me this way, I've been battling this for the last week until he sent me this thread. On my Odroid XU4, the HS3 process normally idles at 8-30%, but it will spike and stick at 160%+, and if I leave it alone, spike to 350%+, until the system eventually overheats and shuts off. I have reproduced this reliably all week since I moved from Windows to Linux. It can take anywhere from a few minutes to a few hours with the Nest Plugin enabled.

System:

• Odroid XU4
• Ubuntu 17.10 ARM
• Mono 5.4.1.6 (tarball Wed Nov 8 21:23:55 UTC 2017) (installed via apt repo)
• Observed on HS3 versions .368 and .398
• Nest Plugin version 3.0.0.28

I am willing to try whatever debugging you need.

[mloebl]

I have yet to be able to recreate this so I need help. Can you try disabling all plugins, see if it fails and if not, then start enabling them? Also, are you running any scripts?

I may have to add some internal diagnostics that can detect when a thread is not sleeping or an external API is accessed too often.

The first thing I did was disable my scripts thinking some of my scripts were at fault as I had to update a couple due to the differences with Mono. Tomorrow when events are at a lull, I can disable everything but Nest. Thank you!

[mloebl]

Running the test right now, see how it goes. Here's the total list of plugins I have besides Neset:

• 3.0.6.23: Insteon
• 1.5.0.0: MQTT
• 3.0.6551.16959: UltraGCIR3
• 3.0.6516.32486: UltraLighting3
• 3.0.6104.19146: UltraRachio3
• 3.0.1.179: Z-Wave

Right now everything is disabled, including scripts that run. I do recall in my testing, once I narrowed it down, the only plugins I had going were Insteon, MQTT, Zwave and Nest. Once I had eliminated them, one by one, I eventually left the "good" ones going, so it's possible there could be an interaction issue there.

[mloebl]

@rjh Just an update. I ran it for 7 hours with *just* the Nest plugin (including scripts enabled) and it was fine. I enabled the Zwave plugin, and within an hour it was sitting at 160%+ CPU.

[mloebl]

@rjh I created a clean Ubuntu 17.10 VM, HS3 .398 in (Trial mode), Zwave, Insteon, Nest (Trial mode) and MQTT plugins. I enabled Zwave (not connected to anything), MQTT (receiving data to give it some activity), and Nest (hooked up to my account.) In about 2 hours I had reproduced the problem.

I'll see if I can do some experimenting to find the bare minimum it takes to reproduce this. If so, I can upload this VM for you to experiment with.

Notes on how this vm was created:

1. Installed Ubuntu 17.10 server clean on a dual core 2gb RAM vm
2. Ran apt update to get latest OS
3. Rebooted post updates
4. Added the mono repo from these directions and installed mono-complete, http://www.mono-project.com/download/
5. Installed mono-vbnc and flite
6. Downloaded HS3 .398 and extracted to /opt/Homeseer
7. Set ownership recursive of /opt/Homeseer to the 'homeseer' user
8. Used these instructions to create a systemd service and /opt/HomeSeer/stop_homeseer.sh script for Homeseer, https://forums.homeseer.com/showthread.php?t=186864
9. Started HS3 from the go script to validate it worked, and installed Zwave (3.0.1.179), Insteon, MQTT and Nest Plugins
10. Stopped HS3, and started HS3 via the systemd service
11. Configured MQTT to point to local server and added 1 device
12. Configured Nest plugin to put to my account
13. Waited

EDIT 1: Zwave and Nest plugins only, and CPU at over 100% in <28minutes. Going to unregister Nest from account to see if that matters for the next test.
EDIT 2: Ran for several hours with Zwave + Nest, but with the Nest account deauthorized, and it did not fail

[mloebl]

Just an update, running the Nest plugin unconnected to the account doesn't seem to trigger the issue. Looks like account has to be connected to the account and the Zwave plugin active. Not entirely sure why.

[sbrunner]

I have two more days on the trial license, I am happy to provide SSH access to my SEL Pro to a developer. This was set up as a virgin system. Takes about 12 hours to lock up the box. I can reliably replicate with Nest plugin on and off. How do we get in contact?

[mloebl]

I have two more days on the trial license, I am happy to provide SSH access to my SEL Pro to a developer. This was set up as a virgin system. Takes about 12 hours to lock up the box. I can reliably replicate with Nest plugin on and off. How do we get in contact?

Are you running with the Zwave plugin enabled as well, or the only the Nest plugin?

[rjh]

I have an SEL running with the Nest plugin enabled. I have also enabled the Z-Wave plugin and am polling one node (just to keep it busy). Will monitor. I don't have any events configured.

[mloebl]

I have an SEL running with the Nest plugin enabled. I have also enabled the Z-Wave plugin and am polling one node (just to keep it busy). Will monitor. I don't have any events configured.

Should word if Nest is connected to a thermostat (at least in my testing.) I didn't need any events or anything connected to the Zwave plugin. I'm really hoping you're able to get to the bottom of it Thank you for trying!

[sbrunner]

Are you running with the Zwave plugin enabled as well, or the only the Nest plugin?

I have the Z-Wave plugin enabled, but nothing connected to it. The other was the Lutron Caseta plugin. Nest plugin polled two thermostats.

[mloebl]

I have the Z-Wave plugin enabled, but nothing connected to it. The other was the Lutron Caseta plugin. Nest plugin polled two thermostats.

Nice! Ok so our results were pretty consistent then I think.

[sbrunner]

I still think this is a HS3 issue and not a plugin issue. It may just be a vulnerability the plugin accidentally uncovered.

[rjh]

I agree that it is an HS3 issue, but most likely related to the plugin API. You are seeing this on Windows or Linux?

I still think this is a HS3 issue and not a plugin issue. It may just be a vulnerability the plugin accidentally uncovered.

[sbrunner]

On a SEL Pro - Linux.

Current Date/Time: 1/4/2018 3:12:25 PM
HomeSeer Version: HS3 Pro Edition 3.0.0.368
Linux version: Linux hometrollerSEL 3.16.0-031600-generic #201408031935 SMP Sun Aug 3 23:56:17 UTC 2014 i686 i686 i686 GNU/Linux System Uptime: 2 Days 1 Hour 50 Minutes 0 Seconds
IP Address: 192.168.x.x
Number of Devices: 94
Number of Events: 9
Available Threads: 199

Enabled Plug-Ins
2.0.1.6: JowiHue
3.0.0.102: LutronCaseta
1.1.17336.309: MyQ
3.0.6542.35207: UltraWeatherWU3
3.0.1.130: Z-Wave