HomeSeerData.last.hsd left open and exhausting open files

[Guest]

Been chasing a problem where HS3 becomes unresponsive and have traced it down to the following issue. Every 30 minutes a cron task runs to clean up php and additional open files are left attached to the "mono HSConsole.exe" process. After a recent reboot the list of file descriptors connected to the process shows:

$ sudo ls -ltr /proc/2223/fd
total 0
lrwx------ 1 root root 64 Sep 10 15:30 9 -> socket:[5849]
lrwx------ 1 root root 64 Sep 10 15:30 8 -> socket:[6991]
lrwx------ 1 root root 64 Sep 10 15:30 7 -> socket:[5844]
lrwx------ 1 root root 64 Sep 10 15:30 6 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Sep 10 15:30 5 -> socket:[5840]
lr-x------ 1 root root 64 Sep 10 15:30 4 -> /dev/urandom
lrwx------ 1 root root 64 Sep 10 15:30 3 -> /tmp/HomeSeerLog.hsd
lrwx------ 1 root root 64 Sep 10 15:30 25 -> socket:[7711]
lrwx------ 1 root root 64 Sep 10 15:30 2 -> /dev/pts/1
lrwx------ 1 root root 64 Sep 10 15:30 19 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 15:30 18 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 15:30 17 -> /usr/local/HomeSeer/Data/Energy/Energy.hsd
lrwx------ 1 root root 64 Sep 10 15:30 15 -> socket:[1557]
lrwx------ 1 root root 64 Sep 10 15:30 14 -> socket:[5853]
lrwx------ 1 root root 64 Sep 10 15:30 13 -> socket:[5852]
lrwx------ 1 root root 64 Sep 10 15:30 12 -> socket:[1556]
lrwx------ 1 root root 64 Sep 10 15:30 10 -> socket:[5850]
lrwx------ 1 root root 64 Sep 10 15:30 1 -> /dev/pts/1
lr-x------ 1 root root 64 Sep 10 15:30 0 -> /dev/null
lrwx------ 1 root root 64 Sep 10 15:36 23 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 15:36 22 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 15:36 21 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 15:36 20 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 16:09 24 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 16:39 28 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 16:39 27 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 16:39 26 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 17:09 30 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 17:39 33 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 17:39 32 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd
lrwx------ 1 root root 64 Sep 10 17:39 31 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 17:39 29 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 18:09 35 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 18:09 34 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 18:39 40 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 18:39 39 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 18:39 37 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 18:39 36 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 19:09 42 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)
lrwx------ 1 root root 64 Sep 10 19:09 41 -> socket:[24138]
lrwx------ 1 root root 64 Sep 10 19:09 38 -> /usr/local/HomeSeer/Data/HomeSeerData.last.hsd (deleted)

The list of (deleted) files continues to grow until the process runs out of file handles.

Using lsof tells a similar story:

$ sudo lsof | grep 2223 | grep delete | wc
864 9480 115776

The files almost always have a time tag of 09 or 39 which just so happens to match this cron.d entry:

$ cat /etc/cron.d/php5
# /etc/cron.d/php5: crontab fragment for php5
# This purges session files older than X, where X is defined in seconds
# as the largest value of session.gc_maxlifetime from all your php.ini
# files, or 24 minutes if not defined. See /usr/lib/php5/maxlifetime

# Look for and purge old sessions every 30 minutes
09,39 * * * * root [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime)

This problem has been occurring for several months over previous versions and requires a reboot after a week or so. There are no errors logged that I can find.

I'm running a Homeseer controller at the latest version.

Current version is: HS3 ZEE S2 Edition 3.0.0.548

[w1ntermute0]

I have this exact problem. I'm not that savvy with linux, but I saw some errors in the log related to too many open files, before the web client went unresponsive (Homeseer itself seems to keep working however). After I got the open files clue, I tried digging around with lsof. Running "sudo lsof | wc -l" returned over 49,000 prior to the web server dying. Now after a reboot it was about 9,000, and now 10,000 a few hours later. If I run a detailed lsof I see many entries for "HomeSeerData.last.hsd (deleted)" just like you show above. This only started happening within the past few months. My homeseer used to be rock solid and now it needs attention every few days, so I'd really like to get it solved. I'm about to try a new image on a new SD card, I think I read somewhere the newer images have a newer version of mono. Maybe that will help. Did you ever solve your problem?

I'm also on HS3 ZEE S2 Edition 3.0.0.548. I use Insteon and Insteon Thermostat plug-ins only. I also relatively recently connected my homeseer with Google Home, in case that might have anything to do with it.

-wm