980 losing connection

[MNB]

paul BigBadBuzz, whilst working to try and getting my i7 up and running, one of my 980's had the same issue (whilst trying to get my blid and password using Dorida980 process it would give me the blid but I got an error message and no password), I found that if I did a iRobot reboot that it cured the issue and I was able to once again get both the blid and password with no issue. MIke

980 Rebooting Instructions:
Press and hold the CLEAN button on the robot for 10 seconds. When the button is released, Roomba® will play the reboot tone indicating a successful reboot.

[BigBadBuzz]

Mike:

Thanks for the suggestion. I did try the reboot, although I did not have time to try Dorita again;hopefully tonight. I can report that with the newest Roomba plugin, things do seem to be working pretty well, including being able to really change the options such as edge cleaning.

Regards,
Paul

[akurz]

Hi All.

I'm afraid this issue just came back - exactly as it was a few years ago. I've kept an eye on it for a couple weeks now, hoping it was caused by some other glitch, but it keeps coming back out of the blue about once a week. The only difference I can see (and that may be coincidence) is that the disconnect quite often (but not always) seems to happen while the Roomba is busy cleaning, with the result that HS thinks that the Roomba is still out and about while it's actually back on base.
As before, a stop/restart of the plugin solves the issue reliably.

Cheers,
Alex

[dzee]

You don't happen to using Unifi (Ubiquiti) gear any chance? There have been issues with Unifi access points blocking broadcast messages that Roomba (and Logitech Harmony Hubs) use. Just a thought.
Marty

This exactly describes what happens to me from time to time. I have Ubiquiti USG & AP Pros. Roomba is AWOL from HS3, but Alexa will start it and so does the iRobot Android App indicating the wifi connection is live. A restart of the BLRoombaWifi plugin, and all is well from HS3. The first few times it happened, I thought Roomba had updated so I checked the BLID and password but found them unchanged. Last time it happened was just a few weeks ago. Seems to work for 6-10 weeks, then recur.

I also have two Harmony Hubs (only one on HS3) and on rare occasions the one on HS3 disconnects and won't pause/unpause or mute the TV from the remote. This doesn't even involve HS3 since it's remote->Wifi->Hub->IR. Oddly, I've never had the Harmony disconnect while the system is off which is where it spends most of the hours. It never fails to turn on the stack of equipment or fail to turn it off after it had been running for hours. Generally when it happens we don't try to turn off the equipment, but I'm going to try that next time.

So, in the case of the Roomba, it seems to point to the HS3 plugin since Roomba responds through the Ubiquiti gear from other control points, but in the Harmony case, it points to the Ubiquiti gear because HS3 is not involved but could also be the Harmony gear. I have never had a disconnect between the second Harmony remote and hub in our bedroom, but we use that one relatively little compared to the one in the family room.

[dzee]

Hi All.

I'm afraid this issue just came back - exactly as it was a few years ago. I've kept an eye on it for a couple weeks now, hoping it was caused by some other glitch, but it keeps coming back out of the blue about once a week. The only difference I can see (and that may be coincidence) is that the disconnect quite often (but not always) seems to happen while the Roomba is busy cleaning, with the result that HS thinks that the Roomba is still out and about while it's actually back on base.
As before, a stop/restart of the plugin solves the issue reliably.

Cheers,
Alex

Alex, your symptoms don't match mine and mine have never been that frequent. Also, my Roomba 980 doesn't seem to ever lose a connection mid-run with HS3, but that could also relate to a wifi coverage issue. If it is, it's coming back every time before mid-run charging is complete because I have a "Roobma is continuing cleaning" announcement that it's never missed while I've been working from home this past year.

If you do have Ubiquiti equipment, I'd be happy to compare models and firmware, but I'm not convinced this is Ubiquiti's problem since it responds to the smartphone app and Alexa when it is unresponsive to the plug-in. How does yours respond to the app (and Alexa/Google if you have those)?

[akurz]

Alex, your symptoms don't match mine and mine have never been that frequent. Also, my Roomba 980 doesn't seem to ever lose a connection mid-run with HS3, but that could also relate to a wifi coverage issue. If it is, it's coming back every time before mid-run charging is complete because I have a "Roobma is continuing cleaning" announcement that it's never missed while I've been working from home this past year.

If you do have Ubiquiti equipment, I'd be happy to compare models and firmware, but I'm not convinced this is Ubiquiti's problem since it responds to the smartphone app and Alexa when it is unresponsive to the plug-in. How does yours respond to the app (and Alexa/Google if you have those)?

Hi dzee,

Yep, I'm using Ubiquiti as well with UAP-AC-Pro and UAP AC-LR all over the place. The Roomba never gets farther than about 10 meters from an UAP-AC-LR. It's literally line of sight at any time.
Nevertheless I also suspected the APs, so I did a firmware update a few weeks back after the disconnects started again. Made no difference as far as I can say.

The odd thing is that it was working rock solid for about 2 years now and suddenly it starts disconnecting again. and as before, a plugin restart fixes it right away. Very odd.

For now I'm automatically restarting the plugin once every night. Up to now things are ok, but I've only started doing that mid last week, so I'll give it a few more days before I crack the bubbly.

What would be brilliant would be a function like "Log Roomba Messages to HomeSeer Log" but less frequently. Right now when activated the log gets flooded with "still alive!" msgs every 6 seconds. If I had something like that every say 10 or 30 minutes, I could monitor the Roomba's online status and restart the plugin only when necessary. That would be useful not only for dirty hacks like the one I intend - a real "Online/Offline" status for the roomba would be generally useful, I believe.

Cheers,
Alex

[akurz]

...

For now I'm automatically restarting the plugin once every night. Up to now things are ok, but I've only started doing that mid last week, so I'll give it a few more days before I crack the bubbly.

...

And promptly last night the plugin restart didn't help. I also have to correct myself in terms of when the problem occurs: It DOESN'T happen while the Roomba is cleaning. That was a misconception because its status sits on "Cleaning", which is because I've set it to that state. In reality the Roomba stays on base because it never received the signal.

Long story short: the connection dies at some point. Just like that.

Cheers,
Alex

[akurz]

And promptly last night the plugin restart didn't help. I also have to correct myself in terms of when the problem occurs: It DOESN'T happen while the Roomba is cleaning. That was a misconception because its status sits on "Cleaning", which is because I've set it to that state. In reality the Roomba stays on base because it never received the signal.

Long story short: the connection dies at some point. Just like that.

Cheers,
Alex

As of now, my Roomba has been online for a little over 19 days with no disconnect. You may ask why it seems stable now and what I did to achieve that... To be absolutely honest: I'm not quite sure.
I was still curious what exactly causes the problem, whether its something in the plugin losing connectivity to an otherwise perfectly well connected Roomba, or if the connection drops completely on network level. To figure that out I've installed another BL plugin - BLLAN - and set up my Roomba to be pinged once every 10 min (Interval 600) and notify me when it misses 2 pings. Since this monitoring is in place the connection stays rock solid. Seems like regular pings to the robot keep things going smoothly.

Well, I suppose one could call that a fix or workaround for my problem. I'm happy with it.

However: since the plugin is able to monitor connectivity (Log Roomba Messages to HomeSeer Log), some kind of "Connected" status on the device would be super.

Cheers,
Alex

[dzee]

Interesting that a ping keeps it alive. Reduced power mode on wifi chips is common.

Two questions:

1. Is your HA system on wired or wireless LAN?

2. Have you tried reaching Roomba using the smartphone app when HA plugin doesnt see it? Mine responds to app when plugin says its disconnected. Also responds to Alexa. Both are immediate.

Both are coming in from outside servers, so I'm puzzled by pings doing keep alive on LAN since it seems to know when mothership is sending commands. Ignoring LAN addresses seems unlikely since the gateway is a LAN address. It also must poll mothership on 80 or 443 since I have no open ports inbound or port forwarding.

Sent from my SM-N975U1 using Tapatalk

[akurz]

As of now, my Roomba has been online for a little over 19 days with no disconnect. You may ask why it seems stable now and what I did to achieve that... To be absolutely honest: I'm not quite sure.
I was still curious what exactly causes the problem, whether its something in the plugin losing connectivity to an otherwise perfectly well connected Roomba, or if the connection drops completely on network level. To figure that out I've installed another BL plugin - BLLAN - and set up my Roomba to be pinged once every 10 min (Interval 600) and notify me when it misses 2 pings. Since this monitoring is in place the connection stays rock solid. Seems like regular pings to the robot keep things going smoothly.

Well, I suppose one could call that a fix or workaround for my problem. I'm happy with it.

However: since the plugin is able to monitor connectivity (Log Roomba Messages to HomeSeer Log), some kind of "Connected" status on the device would be super.

Cheers,
Alex

I swear this thing is jinxed:

Computer - - Monitoring - Roomba (192.168.121.160) is OFFLINE at 06.04.2021 18:40:18
The computer was Online for: 20 days, 18 hrs, 29 mins, 10 secs

Analyzing now.

[akurz]

Interesting that a ping keeps it alive. Reduced power mode on wifi chips is common.

Two questions:

1. Is your HA system on wired or wireless LAN?

2. Have you tried reaching Roomba using the smartphone app when HA plugin doesnt see it? Mine responds to app when plugin says its disconnected. Also responds to Alexa. Both are immediate.

Both are coming in from outside servers, so I'm puzzled by pings doing keep alive on LAN since it seems to know when mothership is sending commands. Ignoring LAN addresses seems unlikely since the gateway is a LAN address. It also must poll mothership on 80 or 443 since I have no open ports inbound or port forwarding.

Sent from my SM-N975U1 using Tapatalk

The HS box is running on a Windows 2019 Server in a KVM VM on a 3 node HA cluster. Connection is wired 4x1GbE Trunk. Plenty of other servers running on this host (~ a dozen) with no issues.
As to the app I have the same effect as you: the robot responds to the app just fine while the HS plugin is offline. It's pretty clear that its a local issue as a plugin restart brings it back to life quite reliably.

My bet is on ARP. Had this a couple times on Windows boxes. Seriously difficult to troubleshoot as the problem usually disappears with the next ARP cache refresh, but the connected plugin sometimes needs an extra kick anyways.

Now that I've been notified of the disconnect in a timely manner I have a better chance to find something. Stay tuned.

While I was writing this post, the connection came back online again - and the HS plugin also reconnected just fine. Connection is still stable. I'm literally pulling my hair out here

Alex

[dzee]

Hi Alex, Is your Roomba at a static IP? Are you thinking the ARP table entry is expiring on the VM but restarting the plugin refreshes it, but it is OK on the gateway or wherever app commands come from? Not sure if I follow.

[akurz]

Hi Alex, Is your Roomba at a static IP? Are you thinking the ARP table entry is expiring on the VM but restarting the plugin refreshes it, but it is OK on the gateway or wherever app commands come from? Not sure if I follow.

I'm not sure of anything anymore either. Yep, the Roomba is on static IP. And yes, I believe something causes the connection to go stale on the HS server. Other devices, like my mobile with the app on it seem to be OK.
Pinging with BLLAN still keeps things going as it seems. I had 2 "Roomba is offline" warnings in a day, which resolved within one ping interval. Didn't happen anymore since then. They outages didn't break the connection either, so the alert mail was a red herring. Everything is still going strong - as long as I keep pinging the roomba from the HS server every 10 min.

Alex

[dzee]

We're dealing with a device that happily understands how to converse with the mothership on the WAN, but seems to be deaf to a fellow LAN device.

But how does the command from the mothership reach the Roomba in the first place? It can't simply penetrate the firewall with an unsolicited packet or we'd have to set up port forwarding. Roomba HQ mothership knows no IP other than the WAN address of your network, and the router has no clue to send a packet from Roomba HQ to the Roomba if Roomba didn't start the conversation.

My Roomba runs twice a week, and shows 77MB downstream and 281MB upstream in 25 days. So, it's doing about 11MB per day of outbound chatter on average. On days it runs, I can see the upstream/downstream traffic is more balanced, but on days it's not running there is 10MB up, but under 3MB down and around 56K packets. Spreading those out over a day, that's roughly a packet every 1.5 seconds around the clock. I don't know how many packets it takes for the command protocol, but if it were not a big JSON payload (and why should it need to be?) a single packet round trip would be about the time it takes Roomba to respond to a command from the app. Or should I say, the time it takes for Roomba to ask what it should do and receive an answer.

Could the plugin be spoofing the Roomba's request for instructions to HQ using the BLID and password we have to enter, which would cause the Roomba HQ response to come back to the plugin, then the plug in uses information received from the Roomba HQ to authenticate back to the Roomba but embed the command it wants? I'm completely theorizing but if looked at this way, what impact might that have on keep-alive or packet routing that could result in the Roomba seeming offline to the plugin and how would restarting the plugin solve it?

I could be way out in the weeds here. I'm sure BLRoombaWifi's author (Blade?) is probably laughing right now if reading this. Just trying to reverse engineer a solution, though.

[akurz]

We're dealing with a device that happily understands how to converse with the mothership on the WAN, but seems to be deaf to a fellow LAN device.

But how does the command from the mothership reach the Roomba in the first place? It can't simply penetrate the firewall with an unsolicited packet or we'd have to set up port forwarding. Roomba HQ mothership knows no IP other than the WAN address of your network, and the router has no clue to send a packet from Roomba HQ to the Roomba if Roomba didn't start the conversation.

[...snip...]

I could be way out in the weeds here. I'm sure BLRoombaWifi's author (Blade?) is probably laughing right now if reading this. Just trying to reverse engineer a solution, though.

I'm starting to feel seriously bad for sending you down this rabbit hole... Sorry for that.

Alex