Discussions related to HS3 build 3.0.0.368

[rjh]

Ok, that sounds like some progress, awaiting results.

Note that there is an issue with the 144 version of the Z-Wave plugin using CPU, so I recommend anyone running that to go back to 130 for now until I get that sorted out.

Rich,
I disabled my recurring events yesterday morning, and that didn't seem to make a difference. Last night I disabled all plug-ins but two, ZWave and JowiHue and restarted. The memory remained steady all night. So now I will start enabling plug-ins one by one.

[rjh]

There should only be one HSConsole, so that is really odd. If for some reason your system is launching more, each one is another instance of HS3!

Is this your own built system or one of our controllers?

Normally, HS is started from rc.local, not sure if your system is re-running that script for some reason?

I'm still having the same CPU issues with 368 as with anything after 326 -- within a few days the cpu loads go up enormously and I start running into thermal issues on my RPI3. With 4 days of uptime right now, htop is showing as per the attached image.

I believe I'm also starting to see kernel messages indication out-of-memory conditions, but the actual messages are unlike anything I've seen before so I need to do a little reading before I say for sure that this is true.

I know that the methodical approach to testing would be to disable all of my plugins and events and try to see what's causing the issue, but my wife will go nuts if I disable stuff for more than a few hours -- and this problem takes a couple of days to fully develop. I'm really unsure what to do.

Is it normal for there to be over 100 HSConsole.exe threads?

[dmurphy]

I discovered all my device codes got set to null, shortly after upgrade to 368 never seen this before?

See here for details, I thought this might be blradar ...https://forums.homeseer.com/showpost...65&postcount=5

[rjh]

Could be a plugin, I checked a bunch of systems here and the device codes are all intact and they are all upgraded to 368. So not really sure what caused it. HS never writes the codes (other than from the device properties). The only other possibility is that it could not read them from the HSD file.

Do you have an older backup of your HSD file that you could load just to see if they load ok?

I discovered all my device codes got set to null, shortly after upgrade to 368 never seen this before?

See here for details, I thought this might be blradar ...https://forums.homeseer.com/showpost...65&postcount=5

[Blade]

Rich

I want to revert mine back to .357
Is there a list of files that need to be restored. I have backups going back quite a way. I just need to know what files need to be reverted.

I want to verify that the system then runs OK with no memory issues, then move forward from there.

[rjh]

Normally you can go back with just replacing the HS3.exe and Scheduler.dll. You may need to also replace the HomeSeerAPI.dll file.

There were some changes to the event data, so any new events created may error when you go back. If you have an older HSD file, you can use that.

Rich

I want to revert mine back to .357
Is there a list of files that need to be restored. I have backups going back quite a way. I just need to know what files need to be reverted.

I want to verify that the system then runs OK with no memory issues, then move forward from there.

[Bob_Linux_User]

I turned off all plugins except Zwave to test memory leak. HSTouch was still enabled as was Energy logging and system logging. The memory usage still continues to grow (See Below). I have now disabled Energy Logging and will test. If I disable HSTouch I get continual login attempts as reported in post above. If I shutdown my HSTouch clients will that stop the login attempts?

I also noticed while testing I got an unusual error on the System Log.

Oct-02 19:45:00 Log Warning Log database maintenance may be needed. (217 >= 50)

Just FYI my log file is set for 50mb.

[rjh]

In regards to the log database warning, there may be a bad entry. Shut down HS and rename or delete the log file. That will fix that one.

The logins are odd, do you have any HSTouch clients attempting to connect to your system through MyHS? If not and you are not use MyHS, you can disable that totally for now in setup.

So maybe in your case its some event configuration that is causing the memory to climb?

I turned off all plugins except Zwave to test memory leak. HSTouch was still enabled as was Energy logging and system logging. The memory usage still continues to grow (See Below). I have now disabled Energy Logging and will test. If I disable HSTouch I get continual login attempts as reported in post above. If I shutdown my HSTouch clients will that stop the login attempts?

I also noticed while testing I got an unusual error on the System Log.

Oct-02 19:45:00 Log Warning Log database maintenance may be needed. (217 >= 50)

Just FYI my log file is set for 50mb.

[rjh]

Note that as a test, you can disable all event processing globally from setup. That would be a quick way to rule that out without messing with your events.

In regards to the log database warning, there may be a bad entry. Shut down HS and rename or delete the log file. That will fix that one.

The logins are odd, do you have any HSTouch clients attempting to connect to your system through MyHS? If not and you are not use MyHS, you can disable that totally for now in setup.

So maybe in your case its some event configuration that is causing the memory to climb?

[Bob_Linux_User]

This is very difficult to troubleshoot due to the time required to determine if your test action has a positive effect. My testing seems to indicate that the memory leak is caused by more than one issue.

I determined that the plugin BLLAN was causing a sizable part of the HS3 memory leak.

Disabling HsTouch did not reduce my memory usage. BTW I had to shutdown my HsTouch clients to stop the login messages since they were using MyHS login.

I am continuing to test but my leak has been slowed down enough that I will have to monitor overnight to determine if there is still a problem. It was climbing at a rate of 5kb increase every 5 minutes.

Bob

[MarkHargrove]

@Mark,

Curiosity question...

run

tcptrack -i eth0

What do you see?

Here ran it and see regular HS3 stuff and ports.

Except for one IP talking port 443 (SSL) and 8245 (dynamic DNS) to IP 40.76.43.114 which is:

NetName: MSFT
NetHandle: NET-40-74-0-0-1
Parent: NET40 (NET-40-0-0-0-0)
NetType: Direct Assignment
OriginAS:
Organization: Microsoft Corporation (MSFT)
RegDate: 2015-02-23
Updated: 2015-05-27
Ref: https://whois.arin.net/rest/net/NET-40-74-0-0-1

IP Address: 40.76.43.114
Organization: Microsoft Azure
ISP/Hosting: Microsoft Corporation
Updated: 10/02/2017 09:44 PM

See same on both HS3 lite and HS3 pro. Never paid attention. I do not see this on the non mono LAMP servers here.

Hmm. I've never looked at this before either. I have three unrecognized destinations on four ports:

40.76.43.114:443 -- both of which reverse to MSFT Azure
:8245
34.201.34.16:9553 -- which reverses to an AWS instance
45.76.21.192:10300 -- which doesn't reverse

That last one has been blackholed as of five minutes ago and I'm going to probably blackhole the other one if I can't figure out what it's for.

[MarkHargrove]

There should only be one HSConsole, so that is really odd. If for some reason your system is launching more, each one is another instance of HS3!

Is this your own built system or one of our controllers?

Normally, HS is started from rc.local, not sure if your system is re-running that script for some reason?

I'm running on a stock Raspberry Pi 3, and the only non-stock application running on the server is HS3 Pro. The application is started via systemd with the following service definition:

[Unit]
Description=HomeSeer
After=network.target

[Service]
WorkingDirectory=/usr/local/homeseer
ExecStart=/usr/bin/mono /usr/local/homeseer/HSConsole.exe --log
ExecStop=/usr/bin/curl -s -G 'http://192.168.101.77/JSON?request=runevent&group=Shutdown%20Events&name=Shutdown'
SuccessExitStatus=7
Restart=on-failure

[Install]
WantedBy=multi-user.target

The 'Restart=on-failure' may be the culprit. I've never seen an actual HS3 crash, and this may be the reason. I'm going to change that setting to 'no' to prevent any automatic restarts. This will tell me both whether HS3 is crashing AND whether this is the source of so many HSConsole.exe processes.

[rjh]

Port 10300 is HS3 connection to MyHS server. Port 8245 is HS connecting to our server to determine your WAN IP address which is used by find.homeseer.com and MyHS.

9553 is not used by HS so I am not sure what that is, the IP is not one of our servers.

Hmm. I've never looked at this before either. I have three unrecognized destinations on four ports:

40.76.43.114:443 -- both of which reverse to MSFT Azure
:8245
34.201.34.16:9553 -- which reverses to an AWS instance
45.76.21.192:10300 -- which doesn't reverse

That last one has been blackholed as of five minutes ago and I'm going to probably blackhole the other one if I can't figure out what it's for.

[Pete]

@Rich,

What is the MSFT (40.76.43.114) Azure stuff about relating to Dynamic DNS port (8245) and HTTPS port (443)?

I see this on both the Ubuntu 16.04 64 bit Homeseer 3 boxes here (lite and Pro) running with Mono 4.X.

I understand that Microsoft now owns Mono these days. Is this some mono calling home thing now?

It is not a constant connection and just connects every few minutes here.

It doesn't ding Homeseer 3 running here...just curious about it.

[waynehead99]

Rich, I am on .368, went to it about a week ago. For the most part, things seem to be smooth except one issue. I have a few android panels around the house that I have noticed in the past couple days keep randomly getting disconnected from HSTouch. I have a main touch screen/computer in the kitchen that runs windows and I don't see this issue with it. So I am not sure if its related to the HS update itself, or if a recent update to the android client happened that is causing the issue. All I see in the logs is the disconnects and reconnects. So far, this has been the only issue I have been able to see.

Oct-04 8:18:07 AM	 	HSTouch Server	Client Android named MasterBath has DISCONNECTED.
Oct-04 8:18:07 AM	 	ASR	Speaker Client speech recognition stopped
Oct-04 8:18:07 AM	 	Speaker	Speaker host disconnected, Name: AUTO IP address: 10.0.254.205
Oct-04 8:18:07 AM	 	HSTouch Server	Client Android:MasterBath (10.0.254.51/33547) has raised LineDisconnected or was forced closed.
Oct-04 8:18:07 AM	 	HSTouch Server Warning	Client dropped connection 10.0.254.51/33547
Oct-04 8:17:25 AM	 	Speaker	Speaker host added, Name: Auto Instance: Guest (10.0.254.47) IP address: 10.0.254.205
Oct-04 8:17:25 AM	 	HSTouch Server	Connecting speaker client to HomeSeer IP 10.0.254.205
Oct-04 8:17:25 AM	 	HSTouch Server	Client Android (10.0.254.47/39241) named Guest has CONNECTED.
Oct-04 8:17:25 AM	 	HSTouch Server	Opening connection from:10.0.254.47/39241
Oct-04 8:17:05 AM	 	Speaker	Speaker host added, Name: Auto Instance: MasterBath (10.0.254.51) IP address: 10.0.254.205
Oct-04 8:17:05 AM	 	HSTouch Server	Connecting speaker client to HomeSeer IP 10.0.254.205
Oct-04 8:17:05 AM	 	HSTouch Server	Client Android (10.0.254.51/33547) named MasterBath has CONNECTED.
Oct-04 8:17:05 AM	 	HSTouch Server	Opening connection from:10.0.254.51/33547
Oct-04 8:16:34 AM	 	Z-Wave	Device: Lights Master Bedroom Lamps controlled by Master Bedroom Poll Lights is being polled.
Oct-04 8:16:33 AM	 	Event	Event Trigger "Master Bedroom Poll Lights"
Oct-04 8:16:33 AM	 	Z-Wave	Device: Motion Master Bedroom Motion Sensor Set to Off/Closed/No Motion
Oct-04 8:16:17 AM	 	Event	Event Trigger "Network Monitoring Wayne iPhone Wifi Connected / Stop Timer"
Oct-04 8:16:03 AM	 	Z-Wave	Device: Motion Lower Hall Motion Sensor Set to NOTIFICATION for type Home Security, Event: No Motion
Oct-04 8:16:01 AM	 	Z-Wave	Device: Motion Guest Bedroom Motion Sensor Set to Off/Closed/No Motion
Oct-04 8:15:55 AM	 	Event	Deleting event after run: "Delayed Events Garage Lights Delayed"
Oct-04 8:15:55 AM	 	Event	Event Trigger "Delayed Events Garage Lights Delayed"
Oct-04 8:15:40 AM	 	HSTouch Server	Client Android named Guest has DISCONNECTED.
Oct-04 8:15:40 AM	 	ASR	Speaker Client speech recognition stopped
Oct-04 8:15:40 AM	 	Speaker	Speaker host disconnected, Name: AUTO IP address: 10.0.254.205
Oct-04 8:15:40 AM	 	HSTouch Server	Client Android:Guest (10.0.254.47/46045) has raised LineDisconnected or was forced closed.
Oct-04 8:15:40 AM	 	HSTouch Server Warning	Client dropped connection 10.0.254.47/46045
Oct-04 8:15:17 AM	 	Event	Event Trigger "Network Monitoring Wayne iPhone Wifi Disconnected / Restart Timer"
Oct-04 8:14:55 AM	 	Device Control	Device: Motion Garage Garage Motion to Off (0)
Oct-04 8:14:55 AM	 	Event	Event Trigger "Garage Garage No Motion"
Oct-04 8:14:55 AM	 	HSTouch Server	Client Android named MasterBath has DISCONNECTED.
Oct-04 8:14:55 AM	 	ASR	Speaker Client speech recognition stopped
Oct-04 8:14:55 AM	 	Speaker	Speaker host disconnected, Name: AUTO IP address: 10.0.254.205
Oct-04 8:14:55 AM	 	HSTouch Server	Client Android:MasterBath (10.0.254.51/49328) has raised LineDisconnected or was forced closed.
Oct-04 8:14:55 AM	 	HSTouch Server Warning	Client dropped connection 10.0.254.51/49328