HomeSeer crashes from Sonos plugin, memory leak?

[chubafelix]

Hi Dirk,


I am having a critical problem with Sonos plugin. It took a long time to capture but I have a log file now, attached.

Basically, HomeSeer just crashes after about 2 weeks of operation. I have experienced this now about 4 times. That's why it took so long to post anything about this. The log file says "out of system resources" as you will notice in the log. Then HomeSeer just crashes (quits). With Sonos plugin disabled and all the rest same, i do not have this problem.

Can you please suggest iether what the problem could be or a way to trap the issue for a fix? I would very much appreciate a resolution to this issue.

Thank you.
felix

[dcorsus]

Felix,

did you see this posting?

http://board.homeseer.com/showthread.php?t=151895

What OS do you have. With the help of Tenholde, we found an issue with Vista, we know it is not in Win7, we've asked others who are on Vista or XP to pay attention to it. A new beta (.81) was delivered where you can turn the auto-discovery off.

Dirk

[chubafelix]

Felix,

did you see this posting?

http://board.homeseer.com/showthread.php?t=151895

What OS do you have. With the help of Tenholde, we found an issue with Vista, we know it is not in Win7, we've asked others who are on Vista or XP to pay attention to it. A new beta (.81) was delivered where you can turn the auto-discovery off.

Dirk

Dirk,

All my customers are using either HS Pro100 or HomeTroller. All are embedded Windows XP and we really do NOT have options to change to other OS or modify much in these OS. Thus, I don't know what i could do here, please suggest.

Felix

[chubafelix]

Dirk,

All my customers are using either HS Pro100 or HomeTroller. All are embedded Windows XP and we really do NOT have options to change to other OS or modify much in these OS. Thus, I don't know what i could do here, please suggest.

Felix

PS, yes i did see the thread but not sure what the solution would be. Are you suggesting for me to upgrade to your version 81 and turn off discovery? How will that affect my normal operation?

f

[jon00]

Felix,

Which version of HS are you running. Did you upgrade recently?

[chubafelix]

Felix,

Which version of HS are you running. Did you upgrade recently?

Using PRO100

Current Date/Time:2/16/2012 8:32:57 AM Pacific Standard TimeHomeSeer Version:Professional Edition 2.5.0.23HomeSeer:Is Registered HomeSeer Localization:English (United States)Operating System:Microsoft Windows XP Professional - Work StationOS Version:5.1.2600 Service Pack: 3.0

[dcorsus]

PS, yes i did see the thread but not sure what the solution would be. Are you suggesting for me to upgrade to your version 81 and turn off discovery? How will that affect my normal operation?

f

First I suggest you monitor it, open the task manager, add a column so you can view the handles and see if it creeps up every 5 minutes (and never go down!). If that's what you see, upgrade to latest beta (which is now .83) and turn autodiscovery off (in the config page). What that means is that when a NEW zone is added you need to open up the Sonos Config webpage and click discover, so it will do it only once rather then every 5 minutes. We're talking here about discovery of new zones, once the zone has been discovered, it is a different mechanism to see whether they are on-line or not.

Has this always been this way? For the Vista case we traced it to a call to Windows UPnP and tried all things to do it differently or force windows to release the handles, no dice. I wonder whether Microsoft introduced some bug or we never noticed it. If the handles don't jump up every 5 minutes, than we are looking at another issue all together.


Dirk

[chubafelix]

First I suggest you monitor it, open the task manager, add a column so you can view the handles and see if it creeps up every 5 minutes (and never go down!). If that's what you see, upgrade to latest beta (which is now .83) and turn autodiscovery off (in the config page). What that means is that when a NEW zone is added you need to open up the Sonos Config webpage and click discover, so it will do it only once rather then every 5 minutes. We're talking here about discovery of new zones, once the zone has been discovered, it is a different mechanism to see whether they are on-line or not.

Has this always been this way? For the Vista case we traced it to a call to Windows UPnP and tried all things to do it differently or force windows to release the handles, no dice. I wonder whether Microsoft introduced some bug or we never noticed it. If the handles don't jump up every 5 minutes, than we are looking at another issue all together.


Dirk

OK, i will watch the handles and report back to you. Thx
f

[chubafelix]

OK, i will watch the handles and report back to you. Thx
f

Dirk,

I monitored Handles in task manager for 45 minutes. The value is around 13950 and did not increase continuesly. It stayed at about 5% of that value + or -

Also, if you look at my log, the following facts are clear.


The issue started in middle of the night (hs is not doing much)
2/16/2012 2:10:11 AM ~!~SONOSCONTROLLER~!~Update Time Reached

HS crashed at 2:32 which is only 22 minutes after sonos database update started.

last entry
2/16/2012 2:32:01 AM ~!~SONOSCONTROLLER~!

The first error occurs just a minute before HS crashes.

2/16/2012 2:31:56 AM ~!~SONOSCONTROLLER~!~ERROR in BuildTrackDB/Browse for zoneplayer = Enna with UPNP Error = UPNP_E_TRANSPORT_ERROR . Error = Exception from HRESULT: 0x80040211

followed with this error many times.
2/16/2012 2:32:01 AM ~!~SONOSCONTROLLER~!~BuildArtistDB unable write this record for zoneplayer = Enna with error - System resource exceeded.

So, this seems to be a different problem then the thread you pointed to.

The problem here is that this occurs only once in 2 weeks on its own. There was nothing going on in my house, on Sonos or HS at 2:30am so, i can not attribute this error to any changes or activities.

What do you think we can try here to catch the problem?

thanks
felix

[dcorsus]

Hi Felix,

the 13950 handles is huge, if I remember well, mine are around 1300. So it might actually be that the nightly build of the DB is pushes everything over the limit and not the cause.

So here's my suggestion, restart HS and take a base measurement of memory and handles. Now start monitoring and see if you can spot the offender. If it is the Sonos plugin, actions that require lot's of memory and handles are things like creating the DB but also using HSTouch screens, be it via HST or Apple or Android, especially if you use the "jukebox" implementation where you have created screens that allow you to browse, albums, tracks, artists. I'm not saying that's the problem, all I'm saying is these functions see a big hand-off of information from plugin to HS, to HS Server to ........ Do you have a large track collection? The larger the collection the more memory will be moving around when you browse music. I have a 30K track collection, I can see my memory use jumping up big time when I browse, but it all seem to come down to pre-levels.

Alternatively, it could be something else in HS or other plugins that erode handles over time. How is memory utilization looking?

Dirk

[chubafelix]

Hi Felix,

the 13950 handles is huge, if I remember well, mine are around 1300. So it might actually be that the nightly build of the DB is pushes everything over the limit and not the cause.

So here's my suggestion, restart HS and take a base measurement of memory and handles. Now start monitoring and see if you can spot the offender. If it is the Sonos plugin, actions that require lot's of memory and handles are things like creating the DB but also using HSTouch screens, be it via HST or Apple or Android, especially if you use the "jukebox" implementation where you have created screens that allow you to browse, albums, tracks, artists. I'm not saying that's the problem, all I'm saying is these functions see a big hand-off of information from plugin to HS, to HS Server to ........ Do you have a large track collection? The larger the collection the more memory will be moving around when you browse music. I have a 30K track collection, I can see my memory use jumping up big time when I browse, but it all seem to come down to pre-levels.

Alternatively, it could be something else in HS or other plugins that erode handles over time. How is memory utilization looking?

Dirk

Dirk, i'd like to chase this down but something does not make sense here. you are saying that 13950 is a lot of handles, i just don't know. The unit i am using is PRO100 with Embedded XP, just the way HomeSeer ships it. I just shut down everything on it. The only components still running were:
HSprotect (the flash memory protection)
Speech Modem driver
VNC server through which i connect and remotly manage the machine

my number of Handles without HS even running is about 9000. Please take a look at the two pictures i attached, showing memory usage and processes. Do you see anything abnormal here?

Thanks
Felix

[dcorsus]

What you need to do in the task manager is click on View, click on select columns and check-mark the Handles box. Now you can see the handles used by HS, that's what I was referring to with my 1300-ish. Why don't I see HS in that list of tasks? Is that HSProtect on a PRO100? Now monitor the memory and handles ONLY for HS unless other processes on that PRO100 are depleting resources.
Dirk

[chubafelix]

What you need to do in the task manager is click on View, click on select columns and check-mark the Handles box. Now you can see the handles used by HS, that's what I was referring to with my 1300-ish. Why don't I see HS in that list of tasks? Is that HSProtect on a PRO100? Now monitor the memory and handles ONLY for HS unless other processes on that PRO100 are depleting resources.
Dirk

Thanks, never new about these columns.
Handles for HomeSeer without Sonos is about 1400.
After i load Sonos and UltraMon (both use UPnP), goes up to 1500.

Now i am monitoring for an hour or so.

felix

[chubafelix]

Thanks, never new about these columns.
Handles for HomeSeer without Sonos is about 1400.
After i load Sonos and UltraMon (both use UPnP), goes up to 1500.

Now i am monitoring for an hour or so.

felix

OK, so i monitored the Handles on HomeSeer.exe and got this:

2:17 1504
2:20 1593
2:25 1597
2:31 1549
2:42 1603
2:49 1580

excluding the startup of 2:17, it looks like the number stays about same, right?

I can monitor some more but at this point the issue seems more related to something Sonos does when it updates the database. what do you think?
f

[tenholde]

I'd say your handles may be growing, based upon what you just posted. Keep monitoring the handles used by HS for several days.

tenholde