New Error on .460

[rjh]

There is no "fix". The issue is that one (or many) plugins are not returning from the event notification fast enough to allow the system to keep up. I would disable plugins until you find the one causing the issue, then we can discuss it with the plugin author. If its hard to find, I could enable a logging option that times all the callbacks, although that could make a mess of the log.

This issue has always been there, it just was not logged. I could simply remove the log I suppose.

bdickhaus I don't have any,never had, timer devices. I'm on .460 (gave been since it was released) but just past day or so seeing tons of these.

rjh are you working on a fix?

[bsobel]

There is no "fix". The issue is that one (or many) plugins are not returning from the event notification fast enough to allow the system to keep up. I would disable plugins until you find the one causing the issue, then we can discuss it with the plugin author. If its hard to find, I could enable a logging option that times all the callbacks, although that could make a mess of the log.

This issue has always been there, it just was not logged. I could simply remove the log I suppose.

Rich, I believe you said you need keep a per plugin queue. When that queue gets to large, can't you log out which plugin is backed up. With lots of plugins on a system, and interactions (one plugin causing events that may cause another plugin to receive changes) its very hard to just 'disable one by one'. The log is defiantly useful, I think once you start naming the plugin you will see a bunch of plugin updates from all of us fixing the issues for everyone, thus making the whole ecosystem faster.

[simonmason]

I second this request. This is becoming a major problem for me. I am seeing the queue full in my log repeatedly - there must be a way to add the offending plugin name to the error message? Disabling plugins and waiting is a really irritating way to solve this.

Let's play this out - disable plugin, don't see the error, can't confirm that it was the plugin, post in the plugin developers forum, plugin developer says he doesn't know what I am talking about as no one else is really seeing this issue, no conclusive proof, back to square one.

For some reason this error only seems to impact my z-wave activity when the queue is full? Of course, that is one of the most frustrating things about this - blinds not going up or down, lights not going on, etc.

[Mr_Resistor]

Agreed. We need to name the source in the logs so PI authors can fix. I'm not having it affect controlling devices, but it's playing havoc with my HSTouch screens. They're constantly "out of sync" with reality. Can we look at raising the limit too?

[simonmason]

ok, I have resorted to shutting down plugins and waiting. This is annoying because each plugin is woven into my whole house system and shutting them off naturally disables certain functionality. In the event that the error doesn't return after shutting down a particular plugin what do I tell the plugin developer? "I shut down your plugin and the error didn't occur again so it must be you?". Is there anything else I can tell them to help diagnose it, if they are even willing to look into it?

[bsobel]

ok, I have resorted to shutting down plugins and waiting. This is annoying because each plugin is woven into my whole house system and shutting them off naturally disables certain functionality. In the event that the error doesn't return after shutting down a particular plugin what do I tell the plugin developer? "I shut down your plugin and the error didn't occur again so it must be you?". Is there anything else I can tell them to help diagnose it, if they are even willing to look into it?

Can you post the list of plugins you're running?

[Mr_Resistor]

Number of Devices: 5227
Number of Events: 490
Available Threads: 3200
HSTouch Enabled: True

3.0.0.1: Alexa
3.3.1.2: APCUPSD
1.19.0.0: Big5
2.0.59.0: BLBackup
2.0.10.0: BLCpuAdvisor
2.0.11.0: BLEditor
2.0.10.0: BLEmail
2.0.1.0: BLFloorplan
2.0.6.0: BLGarbage
3.0.29.0: BLGData
2.0.9.0: BLIcon
2.0.43.0: BLLAN
3.0.19.0: BLLock
2.0.15.0: BLLogMonitor
2.0.2.0: BLMovies
2.0.27.0: BLOccupied
2.0.66.0: BLOnkyo
2.0.13.0: BLPlex *****
2.0.6.0: BLPlugins
2.0.142.0: BLRadar
3.0.16.0: BLReminders
2.0.21.0: BLRoombaWifi
2.0.36.0: BLSecurity
1.0.4.0: BLShutdown
2.0.76.0: BLSpeech
3.1.3.33206: Blue-Iris
2.0.5.0: BLVolume
3.0.0.26: Chromecast
1.0.8.0: Cps Flash
1.0.9.0: Cps Tags
0.0.0.3: Device Comparison 3P
1.3.4.3: Device History
1.0.0.0: drhsEventEnabler
0.0.0.31: drhsIpPlugIn
3.0.0.3: DTSDeviceImages
3.0.0.51: EasyTrigger
1.0.16302.20: EventGhost
3.0.1.0: FaceRecognition
0.2.0.10: FitbitSeer
3.7.2.0: Harmony Hub
3.9.610.5: HSBuddy
3.0.0.29: ImperiHome
3.0.1.8: ISYInsteon
3.0.0.6: KMS
3.0.1.12: Kodi
3.0.0.4: Log2Text
2.18.8.3: MCSSPRINKLERSP
3.0.0.37: MediaController *****
1.2018.717.1011: MyQ
3.0.0.14: NetCAM
3.0.1.109: PHLocation
4.0.3.0: PLEXMONITOR *****
0.0.0.44: Pushover 3P
1.0.0.7: Restart
30.0.0.37: RFXCOM
1.2.2.0: SceneMaster
5.0.2.0: SCREPOSITORY
3.0.6.5: SDJ-Health
3.3.5.17921: SMS-Gateway
3.2.0.1: SqueezeBox
3.0.4.6: Tasker Plugin
3.0.6551.16959: UltraGCIR3
3.0.6644.26753: UltraLog3
3.0.6554.33094: UltraMon3
3.0.6413.20219: UltraNetCam3
3.0.5902.19061: UltraSMTP3
3.0.6496.15709: UltraTivo3
3.0.6702.34494: UltraWeatherWU3
3.0.0.21: VWS
3.0.0.88: weatherXML
3.0.1.243: Z-Wave

Disabling the ones with ***** after the name reduced the number/frequency of errors for me.

[simonmason]

Here you go. As you can see I have disabled the Tasker plugin first. I can reproduce the error on demand - I have an event that turns off all lights in the house. Run that and it starts throwing the error. When I get a chance I am going to try running this with all of the plugins disabled and then turn them on one by one and see if I can find out when it happens. Presumably I need to restart homeseer between each test to make sure everything is truly getting cleaned out.

[bsobel]

Number of Devices: 5227
Number of Events: 490
Available Threads: 3200
HSTouch Enabled: True

3.0.0.1: Alexa
3.3.1.2: APCUPSD
1.19.0.0: Big5
2.0.59.0: BLBackup
2.0.10.0: BLCpuAdvisor
2.0.11.0: BLEditor
2.0.10.0: BLEmail
2.0.1.0: BLFloorplan
2.0.6.0: BLGarbage
3.0.29.0: BLGData
2.0.9.0: BLIcon
2.0.43.0: BLLAN
3.0.19.0: BLLock
2.0.15.0: BLLogMonitor
2.0.2.0: BLMovies
2.0.27.0: BLOccupied
2.0.66.0: BLOnkyo
2.0.13.0: BLPlex *****
2.0.6.0: BLPlugins
2.0.142.0: BLRadar
3.0.16.0: BLReminders
2.0.21.0: BLRoombaWifi
2.0.36.0: BLSecurity
1.0.4.0: BLShutdown
2.0.76.0: BLSpeech
3.1.3.33206: Blue-Iris
2.0.5.0: BLVolume
3.0.0.26: Chromecast
1.0.8.0: Cps Flash
1.0.9.0: Cps Tags
0.0.0.3: Device Comparison 3P
1.3.4.3: Device History
1.0.0.0: drhsEventEnabler
0.0.0.31: drhsIpPlugIn
3.0.0.3: DTSDeviceImages
3.0.0.51: EasyTrigger
1.0.16302.20: EventGhost
3.0.1.0: FaceRecognition
0.2.0.10: FitbitSeer
3.7.2.0: Harmony Hub
3.9.610.5: HSBuddy
3.0.0.29: ImperiHome
3.0.1.8: ISYInsteon
3.0.0.6: KMS
3.0.1.12: Kodi
3.0.0.4: Log2Text
2.18.8.3: MCSSPRINKLERSP
3.0.0.37: MediaController *****
1.2018.717.1011: MyQ
3.0.0.14: NetCAM
3.0.1.109: PHLocation
4.0.3.0: PLEXMONITOR *****
0.0.0.44: Pushover 3P
1.0.0.7: Restart
30.0.0.37: RFXCOM
1.2.2.0: SceneMaster
5.0.2.0: SCREPOSITORY
3.0.6.5: SDJ-Health
3.3.5.17921: SMS-Gateway
3.2.0.1: SqueezeBox
3.0.4.6: Tasker Plugin
3.0.6551.16959: UltraGCIR3
3.0.6644.26753: UltraLog3
3.0.6554.33094: UltraMon3
3.0.6413.20219: UltraNetCam3
3.0.5902.19061: UltraSMTP3
3.0.6496.15709: UltraTivo3
3.0.6702.34494: UltraWeatherWU3
3.0.0.21: VWS
3.0.0.88: weatherXML
3.0.1.243: Z-Wave

Disabling the ones with ***** after the name reduced the number/frequency of errors for me.

http://download.casapiedrasoftware.i...r_override.txt
http://download.casapiedrasoftware.i...r_override.txt

I have made changes to both to deal with the scenario where another plugin changed devices in bulk, I found they could slow down too much trying to update the device state. Not likely to be the sole cause on your system, but should help. The WebUI is also much more responsive in this version, was testing with a few, going to push out generally in a week or so. If you want to run them now, just download each file (one at a time) and put into your HS directory and click on the update listings, the beta update should then show up as an upgrade for you.

[Mr_Resistor]

Just loaded and will test and monitor. Thanks much for the pre-release.

[simonmason]

bsobel - I completely understand your response about the beta version. Is this something I can download to monitor for plugins getting queued up or is only for something using your plugin? Thanks.

[bsobel]

bsobel - I completely understand your response about the beta version. Is this something I can download to monitor for plugins getting queued up or is only for something using your plugin? Thanks.

Hi,

Those where just fixes for my plugins, they all use a common framework to track devices on the system, so I made changes there to be faster in the case of bulk changes. It didn't look like you were running any of those, so yours is a different issue. I think once HS starts logging who is the cause you'll see lots of plugins rev very quickly, I'm just trying to say a bit ahead of the curve...

Bill

[Mr_Resistor]

bsobel- Definitely appears to have made a difference. I'm seeing far fewer instances in my logs. Good work. Still seeing some, and still have some plugins disabled, but it's definitely better.

[sparkman]

rjh Hi Rich, can you provide clarity as to what the different Types mean in these warning messages? I'm trying to track down which of my plugins could be causing these issues as events and ImperiHome updates don't always seem to happen when they should.

The last three warnings I received are Type 64, 1024 and 16384, but I also see 32, and 2048. 16384 seems to be the most common one on my system. Can any more debugging info be provided to see what is in the queue? Having an adjustable queue size would be great as well. Is there a reason for the hard limit of 500?

Thanks
Al

Jan-04 11:46:47 AM

Warning

Dropping event callbacks due to full queue (Type: 64) (500 entries), system may be too busy, plugins and HSTouch may not receive all device updates

Jan-04 9:22:39 AM

Warning

Dropping event callbacks due to full queue (Type: 1024) (500 entries), system may be too busy, plugins and HSTouch may not receive all device updates

Jan-04 8:46:12 AM

Warning

Dropping event callbacks due to full queue (Type: 16384) (500 entries), system may be too busy, plugins and HSTouch may not receive all device updates

[simonmason]

Perhaps it would be possible to have Homeseer dump the event callbacks to a file when it reaches this amount so we could go in and diagnose what is causing this issue? This is crippling my homeseer system almost daily and so far I have not been able to find any specific plug-in causing the issue.