HS3 ZEE S2 3.x update - too many open files after 3 or 4 days

[RichardZWave]

Ever since I updated to 3.0 Edition, after 3 or 4 days, I can no longer access any of the interactive interfaces. The system continues to run. Switches, dimmers and events work normally. Even Alexa skills work. But, I can't use HomeSeer Web Control - locally or remotely, nor any HSTouch Apps on IOS. When I'm at home, I've found I can login with SSH and reboot. Then things work for a few more days. The system was very stable until I went to 3.x. Since 3.x, it has not been stable. I keep hoping the next update will fix it. But no.

I've poked around enough to prove that there is a file handle resource leak. The number of open file handles builds until the homeseer service can't cope any longer and all of the interactive interfaces cease to operate. I can kill and restart the homeseer service and it works again. It certainly seems like it that service that's leaking the file handles.

I've bought in for 4.x, but don't know when to expect that and I'd like to get 3.x stable in the meantime. It's problematic when I'm away from home to be completely cut off from any to access the controller.

I'm wondering if others have seen this and fixed it. If not, I'll put in a help ticket.

Thanks in advance.

_/_/_/_/_/

Immediately after restart...

System Profile

MESSAGE BOARD

Current Date/Time: 12/4/2019 1:03:16 PM
HomeSeer Version: HS3 ZEE S2 Edition 3.0.0.548
Linux version: Linux HomeTrollerZeeS2 3.18.11-v7+ #781 SMP PREEMPT Tue Apr 21 18:07:59 BST 2015 armv7l GNU/Linux System Uptime: 0 Days 0 Hours 9 Minutes 21 Seconds
IP Address: 10.0.1.8
Number of Devices: 37
Number of Events: 20
Available Threads: 399
HSTouch Enabled: True
Event Threads: 0
Event Trigger Eval Queue: 0
Event Trigger Priority Eval Queue: 0
Device Exec Queue: 0
HSTouch Event Queue: 0
Email Send Queue: 0

Enabled Plug-Ins
3.0.1.252: Z-Wave

[rmasonjr]

Do any of your events run scripts of any kind?

[RichardZWave]

As I mentioned, the events and scripts do run - if triggered by switches, dimmers, Alexa IFTTT - but no access via interactive interfaces is possible. In fact the web interface literally says too many open files. It's been a few weeks, but I poked around with bash and was able to watch see that the number of open files going up and up and up - 10's of thousands

[RichardZWave]

Maybe I didn't quite answer correctly...

I did have a script I wrote to interact with stringify to work echo buttons - but that's all gone. The problems started with 3.0 - no two ways about that.

[drhtmal]

...I've poked around enough to prove that there is a file handle resource leak. The number of open file handles builds until the homeseer service can't cope any longer and all of the interactive interfaces cease to operate. I can kill and restart the homeseer service and it works again. It certainly seems like it that service that's leaking the file handles...

I would open a ticket. Since you can log into the Zee some trouble shooting can be done.

I am running 3.0.0.548 on a self-built Linux server. Not exactly what you are running on but at least the same software version. I would guess that there are literally thousands of people running Zee S2 with version 3.0.0.548. If there was a huge underlining problem others would likely be experiencing it as well.

Is your flash card full?

[RichardZWave]

I don't think think anything is full - memory-wise. I know it's a file handle leak.

I went to 3.x as soon as it was released and have updated with each release to .548. The instability came when I went from 2.x to 3.x. Nothing else changed at that point, and really nothing since. That's been months.

homeseer@HomeTrollerZeeS2 ~ $ df

Filesystem     1K-blocks    Used Available Use% Mounted on

rootfs           7338472 2310216   4632432  34% /

/dev/root        7338472 2310216   4632432  34% /

devtmpfs          470416       0    470416   0% /dev

tmpfs              94944     224     94720   1% /run

tmpfs               5120       0      5120   0% /run/lock

tmpfs             189880       8    189872   1% /run/shm

/dev/mmcblk0p5     60479   19118     41361  32% /boot

tmpfs             474704      68    474636   1% /tmp

tmpfs             474704     120    474584   1% /var/log

tmpfs             474704       0    474704   0% /var/tmp

[Madcodger]

I don’t know if it’s the same issue, but HS running on a zee S2 with only three plugins and no scripts locks up on me every 1-3 days. I now have an event that reboots it every morning, and I still need to hard restart it once or twice per week via a wifi switch. Rupp suggested that HST might have a beta version that would fix it but I have not heard back from HST since filing a support ticket about three days ago. This is not doing much to make me want to stay with HS, despite having purchased HS4 licenses. My ISY in another building has been chugging along this whole time, without a hiccup. It’s a bit harder to manage in terms of its interface, but in terms of reliability HS is (and since HS2 has been) far less reliable for me in multiple locations. Really has me thinking...

[Rupp]

I don’t know if it’s the same issue, but HS running on a zee S2 with only three plugins and no scripts locks up on me every 1-3 days. I now have an event that reboots it every morning, and I still need to hard restart it once or twice per week via a wifi switch. Rupp suggested that HST might have a beta version that would fix it but I have not heard back from HST since filing a support ticket about three days ago. This is not doing much to make me want to stay with HS, despite having purchased HS4 licenses. My ISY in another building has been chugging along this whole time, without a hiccup. It’s a bit harder to manage in terms of its interface, but in terms of reliability HS is (and since HS2 has been) far less reliable for me in multiple locations. Really has me thinking...

What is your ticket number and I'll check on why this hasn't been addressed.

[RichardZWave]

I submitted a ticket and got an immediate response. The issue with the open files was confirmed and there is a fix. Unfortunately, the fix requires re-imaging of the SD card. I've been provided with an image. I don't know if I'll go to the trouble. I can SSH into the controller, so I can set it up to shutdown and reboot every day or so.

I ordered HS4 - as soon as it was offered, hoping it might be a solution. Haven't heard anything back since my order. Is it shipping now?

[Madcodger]

I don’t know if it’s the same issue, but HS running on a zee S2 with only three plugins and no scripts locks up on me every 1-3 days. I now have an event that reboots it every morning, and I still need to hard restart it once or twice per week via a wifi switch. Rupp suggested that HST might have a beta version that would fix it but I have not heard back from HST since filing a support ticket about three days ago. This is not doing much to make me want to stay with HS, despite having purchased HS4 licenses. My ISY in another building has been chugging along this whole time, without a hiccup. It’s a bit harder to manage in terms of its interface, but in terms of reliability HS is (and since HS2 has been) far less reliable for me in multiple locations. Really has me thinking...

Thanks Rupp, but I didn’t have one. Given RichardZWave’s immediate response, I assume my ticket was not received despite the “form successfully submitted message popping up”. I have now sent in another and got ticket 68688 via an immediate email. We’re only here for two more days before being gone for about 8-9 weeks, so time is running short. And I am not thrilled with having to re-image the sd card. If there has been a fix, I have to wonder why it has not been made available yet?

[Madcodger]

Well, Rupp was nice enough to send me another version of HS and I installed it on two different SD cards (thanks again, Rupp). It turned out this was version 3.0.475, so an older version of HS. After trying each SD card I had trouble connecting to the network each time, although it is possible this was due to a need to reboot the closest access point (four inches away). Given that we had to leave in less than six hours, the uncertainty of this version in my limited testing, and the fact that it is actually an older, not newer, version, I went back to version .548. I figure it’s at least a supported release, and so I put the Zee behind a Leviton wifi on/off switch and turned off the only other plugin besides z-wave, BLLan. So now we have a “stock” Zee S2 running nothing other than that single, HST-supplied Z-wave plugin. HS reports 95 devices, but this is due to the many “devices” created for each z-wave device. There are only 14 of those (11 sensors, 2 on/off plugin modules, and a GoControl thermostat). It doesn’t get much simpler than that.

I joined this forum back in 2004, using v1 of HS (which I loved, and which was very stable for me running on Windows, from my recollection). Since then, I’ve been an on-and-off user as I tried other systems (mostly the ISY) due to their better reliability and lack of a need to babysit them. The next eight weeks are likely going to determine whether I remain a HS user. I figure that if a basic, stock, release version of HS, running on a Pi supplied by HST with no modifications (i.e., a Zee S2), on a UPS and sitting four inches from an access point can’t stay up and running, then the problem is not with me, but rather the product. I have another Pi running CumulsMX in exactly that situation, and an ISY with even more devices and programs (events) in another building, on the same network. It is connected via ethernet, but that’s the only difference and my issues have not been with internet connectivity except when trying v.475 last night. I also have a Blink sync module on the same UPS and access point. So, we have some pretty good controls against which to test HS. I’ll report back as I make observations from hundreds of miles away, returning in late Jan to Feb, hopefully. I am pulling for it, but am admittedly skeptical. I truly hope that proves to be unfounded.

[RichardZWave]

I must say, I'm disappointed that an update trashed my stable Zee. I'd benefited from past updates and thought there would be value in staying current. Had I known there was a real risk, I'd have gladly left it alone. Sadly too, the 'fix' - re-imaging the SD card and setting the whole thing up again - is something that I think would be beyond a great many HomeSeer customers. I have SW engineering experience that would give me a fighting chance, yet I'm reluctant to spend a bunch of time and potentially be back in the same place - or worse.

Over time, SW systems have a way of gaining complexity to the point of being effectively unsupportable. My Zee was as out of the box vanilla as could be and yet it reacted badly to what should have been a routine update. This was likely an isolated outcome, but still unfortunately negative for me and apparently others and seemingly not worthy of a more direct remedy - like a corrective update.

I've made a commitment to ZWave devices, so my avenues for new direction are limited.

[Wolfman3570]

I as well am having the same issues with my HS3 ZEE S2 Edition 3.0.0.548. It is the same problem as described at the beginning of this post, and I as well am looking for a solution. I will be trying to troubleshoot this via tech support, and will keep this post updated with any solutions I may have come up with.

Number of Devices: 95
Number of Events: 12
Available Threads: 399
HSTouch Enabled: True
Event Threads: 0
Event Trigger Eval Queue: 0
Event Trigger Priority Eval Queue: 0
Device Exec Queue: 0
HSTouch Event Queue: 0
Email Send Queue: 0

Enabled Plug-Ins
3.0.1.252: Z-Wave

[Rupp]

Well, Rupp was nice enough to send me another version of HS and I installed it on two different SD cards (thanks again, Rupp). It turned out this was version 3.0.475, so an older version of HS. After trying each SD card I had trouble connecting to the network each time, although it is possible this was due to a need to reboot the closest access point (four inches away). Given that we had to leave in less than six hours, the uncertainty of this version in my limited testing, and the fact that it is actually an older, not newer, version, I went back to version .548. I figure it’s at least a supported release, and so I put the Zee behind a Leviton wifi on/off switch and turned off the only other plugin besides z-wave, BLLan. So now we have a “stock” Zee S2 running nothing other than that single, HST-supplied Z-wave plugin. HS reports 95 devices, but this is due to the many “devices” created for each z-wave device. There are only 14 of those (11 sensors, 2 on/off plugin modules, and a GoControl thermostat). It doesn’t get much simpler than that.

I joined this forum back in 2004, using v1 of HS (which I loved, and which was very stable for me running on Windows, from my recollection). Since then, I’ve been an on-and-off user as I tried other systems (mostly the ISY) due to their better reliability and lack of a need to babysit them. The next eight weeks are likely going to determine whether I remain a HS user. I figure that if a basic, stock, release version of HS, running on a Pi supplied by HST with no modifications (i.e., a Zee S2), on a UPS and sitting four inches from an access point can’t stay up and running, then the problem is not with me, but rather the product. I have another Pi running CumulsMX in exactly that situation, and an ISY with even more devices and programs (events) in another building, on the same network. It is connected via ethernet, but that’s the only difference and my issues have not been with internet connectivity except when trying v.475 last night. I also have a Blink sync module on the same UPS and access point. So, we have some pretty good controls against which to test HS. I’ll report back as I make observations from hundreds of miles away, returning in late Jan to Feb, hopefully. I am pulling for it, but am admittedly skeptical. I truly hope that proves to be unfounded.

You can simply update to the latest version once you have the fixed installer installed.

[Madcodger]

So, does .548 have the fixed installer? I’m confused about how an earlier version (e.g., .475) would have a “fixed” installer, but a newer version would not. Canyou please explain/elaborate? Thanks!

5