Ethernet connection problems: Can we summarize our understanding?

[DMcKnight]

Hello all,
I've got three boards with wired ethernet connections, two with "regular" plugin sketches, and onewith an "API" style sketch.
I upgraded everything to the most recent beta as I was bringing the API sketch to life (and am taking care to use the recommended version of the Arduino environment).
Unfortunately, I now have terrible connection problems on a system that was previously very stable. So, I'd like the understand what we all know about Ethernet connection problems, and to contribute where I can. Clearly different parts of a system can interact, so it's not as simple as xxx is "bad" and "yyy" is "good", but we can at least start to gather information.

So, here are some questions that we can perhaps answer (and there may be others):

1. Does plugin version 1.0.0.103 have more problems in this regard? If so which is the most recent solid version. I think it may, but I'll need to dig into backups to find out what I was running before yesterday (duh).

2. Does the Ethernet chipset and/or board manufacturer really make a difference? Does the 5500 chipset work well? (Randy?)

3. I expect the Uno and Mega boards might be most popular. Do people have more problems with one over the other? Does the Mega really benefit from a more recent ethernet shield due to reset problems?

4. Is there a problem running multiple boards on the plugin?

5. Is there a problem if communication to and from the plugin gets too busy? What problems are caused? I make sure my API sketch only sends once a second at most, but it would be easy to write a sketch that floods the plugin with data, I suspect.

6. I'm not using POE. It shouldn't affect things one way or another, but what are people's experiences?

7. What else?

Best regards
Doug

[petez69]

Doug, some feedback

1) I've only used the latest BETA because that's all I've ever gotten to work. I ran an early beta and my boards wouldn't connect. I used wireshark to see what was going on, the boards tried to connect but the plugin didnt like what it was receiving. I've always used the latest IDE and plugin version.

2) I'm not sure if there are clone W5100 chips out there however I have found ethernet board totally vary in behaviour. Boards with 15/10 on the ethernet jack just wont play, I've had some supposed original boards not work either. I have boards with 09/44 on the Ethernet jack and these seem to work fine after a power restart. I have just ordered a mini ethernet board with W5500 onboard to test. I also purchased a small footprint W5100 board with 16/02 on the ethernet jack and paired it with a nano and that seems to be very stable, though in saying that I was messing with power to the board and after resetting the board 5 times I had to restart HS to get it to reconnect, didnt follow any logic...

3) Etherten does NOT like to be powered off for 5 minutes, once the plugin detects a failure its very difficult to get back online. I will be trying the W5500 on the mega when it arrives.

4) Multiple boards dont seem to be an issue, I have 7 ethernet board connected currently. When I've had issues it doesnt seem dependent on the number of boards.

5) Volume of traffic I cant speak to as I dont send data often though I do have 5 DS1820B sensors on various devices and I plan to add more, this doesnt seem to have caused any impact.

6) I tried "poor mans POE" using a splitter cable on ethernet to the etherten, boy that was bad. Board lost connectivity every 15 minutes, I seperated out the power and it was good. I have not tried this method on the MEGAs as yet

7) If you research Greigs arduino code, lots of it are snippets out of standard arduino code examples with variations, I say this as he's not doing anything weird or even pushing the boards capabilities. I still believe there is a fundamental issue with handshaking, what it is I dont know. What I have found is once you get it stable, dont drop power to the boards to allow HS to detect a comms failure, you will get unpredictable outcomes. I'm looking to build my own UPS for my new installation that will use a car battery to power HS and the Arduino boards

One idea I was going to implement was put a UNO on USB on HS to drive a replay board and distribute power to the boards via a central source and then put some logic into HS to drop power to the boards once HS detects the board has lost connectivity. Also I've not researched the arduino "lost connectivity pin: and how I can use it to reset the board itself to force a reconnect. Again that suggests it need to be done at the board, it could be the plugin that is rejecting/ignoring the requests, I dont know.....

I plan to stay with the Arduinos as I think its a good solution for distributed I/O..

Greig, you've done a good job, your Arduino plugin got me back into HS.....!



Pete

[randy]

Hello all,
I've got three boards with wired ethernet connections, two with "regular" plugin sketches, and onewith an "API" style sketch.
I upgraded everything to the most recent beta as I was bringing the API sketch to life (and am taking care to use the recommended version of the Arduino environment).
Unfortunately, I now have terrible connection problems on a system that was previously very stable. So, I'd like the understand what we all know about Ethernet connection problems, and to contribute where I can. Clearly different parts of a system can interact, so it's not as simple as xxx is "bad" and "yyy" is "good", but we can at least start to gather information.

So, here are some questions that we can perhaps answer (and there may be others):

1. Does plugin version 1.0.0.103 have more problems in this regard? If so which is the most recent solid version. I think it may, but I'll need to dig into backups to find out what I was running before yesterday (duh).

2. Does the Ethernet chipset and/or board manufacturer really make a difference? Does the 5500 chipset work well? (Randy?)

3. I expect the Uno and Mega boards might be most popular. Do people have more problems with one over the other? Does the Mega really benefit from a more recent ethernet shield due to reset problems?

4. Is there a problem running multiple boards on the plugin?

5. Is there a problem if communication to and from the plugin gets too busy? What problems are caused? I make sure my API sketch only sends once a second at most, but it would be easy to write a sketch that floods the plugin with data, I suspect.

6. I'm not using POE. It shouldn't affect things one way or another, but what are people's experiences?

7. What else?

Best regards
Doug

I'll answer tomorrow, when I have time to go through all the questions.

[DMcKnight]

Thanks for the reply, Pete. I agree it's an excellent plug-in. I've been using it since the early days, generally with great success.
These recent ethernet troubles are a bit of a head-scratcher...

Today, boards 1 and 2 were connected fine, all day, and board 3 would not connect no matter what I did. (resets, power cycles, etc)

I disabled & enabled the plug-in this evening and board 3 now connects fine. Boards 1 and 2 won't connect, no matter what I do.

Repeating the disable/re-enable didn't change anything, but restarting HS3 did allow all 3 boards to connect after a few minutes. After a few more minutes, boards 1 & 2 are showing "connection Failed".

No changes were made to the hardware throughout.

Probably the next steps should include more extensive logging...
Doug

[petez69]

So interesting today, my new Wiegand reader turned up. I put the code together and debugging a few things, found a quirk with the NANO and small W5100 card.

When you program the NANO, you need to disconnect the reset line to program the device or the upload times out. Problem is if you want to reset the nano and ethernet board, you need to have the reset line connected...Without the reset line I just couldn't get the NANO to connect to HS. Just did some power restart testing and this configuration restarts fine..

During this testing the etherten went offline and I've given up on connecting it again, doesn't seem to follow any logic and its a reputable brand (Freetronics).

Anyway the wiegand reader is up and running, most happy

Pete

[randy]

Hello all,
I've got three boards with wired ethernet connections, two with "regular" plugin sketches, and onewith an "API" style sketch.
I upgraded everything to the most recent beta as I was bringing the API sketch to life (and am taking care to use the recommended version of the Arduino environment).
Unfortunately, I now have terrible connection problems on a system that was previously very stable. So, I'd like the understand what we all know about Ethernet connection problems, and to contribute where I can. Clearly different parts of a system can interact, so it's not as simple as xxx is "bad" and "yyy" is "good", but we can at least start to gather information.

So, here are some questions that we can perhaps answer (and there may be others):

First of all, I am not sure how much I can help as I have had very little connection problems in the last year. I have three production Mega boards with genuine Arduino POE Ethernet shields (r2) and one test board in the same configuration. I abandoned Uno boards early on due to the limited I/O, so I have little time using them. I carefully chose my IP addresses within my DHCP scheme and chose different ports than the defaults. I leave the MAC addresses as generated by the plug-in. I use ports in the 55000 region, mu board at IP 192.168.2.12 is at port 55120, 192.168.2.13 is at 55130 and so on. I started off with generic Mega boards and generic Ethernet shields. The boards were powered by volt converters. I was having all sorts of connection problems. I first went to 5 volt DC-DC converters because the analog voltage regulators on the Megas were running very hot. That calmed the connection issues a little, but they still existed. In about July of 2014, I switched to genuine Mega and Genuine Ethernet boards. My connection problems ended. I did have problems with the boards reconnecting when I changed the IP address of the HS server. Sometime in mid to late 2014 Greig fixed that by not having the boards look for a specific IP address of the server. From then on I had no more connection issues. In October 2014 I switched from the genuine Ethernet (r3) shields to genuine POE (r2) shields. I have not had even a single hiccup since them. The three production shields show to have been online for 145 days. That would go back to last October, when I did a bunch of redundancy checking by purposely disconnecting Ethernet cables, unplugging network switches and killing the HS server. During all of that testing, the boards reliably reconnected, my fail over for heating when a board became disconnected or lost its power supply passed all of the tests. Since then they haven't been touched other than to upload sketches for new versions. I think the testing was under v.94

1. Does plugin version 1.0.0.103 have more problems in this regard? If so which is the most recent solid version. I think it may, but I'll need to dig into backups to find out what I was running before yesterday (duh).

As I said above I have had no disconnect problems at all for over a year.

2. Does the Ethernet chipset and/or board manufacturer really make a difference? Does the 5500 chipset work well? (Randy?)

I received my Ethernet Shield 2 boards a few weeks ago. I tested them on my Mega test board using an external supply and was never able to create a conection problem. With the original (r3) boards, I could get them to fail to connect by pulling the power and restoring it. It wouldn't always fail, only occasionally. Without messing with it and trying to break the connection, it stayed connected for a week or more. The 5V POE modules came from China last week. I am running the new Ethernet shield with POE module on my test board. I cannot make that connection fail either. I cannot switch my production boards because the POE modules stick up too high above the shields to work in my current configuration. I am going to try to remove the POE connectors from the boards and solder the modules directly.

3. I expect the Uno and Mega boards might be most popular. Do people have more problems with one over the other? Does the Mega really benefit from a more recent ethernet shield due to reset problems?

I only read that on the Arduino website. They stated that the r3 boards lacked the ability to properly reset when attached to a Mega, but work fine with a Uno.

4. Is there a problem running multiple boards on the plugin?

I currently am running 4, have run as many as six. The plug-in is limited to 9.

5. Is there a problem if communication to and from the plugin gets too busy? What problems are caused? I make sure my API sketch only sends once a second at most, but it would be easy to write a sketch that floods the plugin with data, I suspect.

I don't know the answer to that. I am using my boards primarily as I/O and PWM output. I don't have counter inputs, nor do I toggle outputs frequently. PWM outputs are generated in the board so only the change in channel values is set to the boards. I don't know how they would deal with a lot of data, but I also have not thought of an instance where I might want to send it. I have a single OneWire temperature probe on each of the 4 boards. My "busiest" board has about 20 inputs and 10 outputs, 3 PWM pins, a OneWire probe and an LCD display. I've never had a problem and latency seems to run consistently at about 250 ms on every board when tested.[quote]

6. I'm not using POE. It shouldn't affect things one way or another, but what are people's experiences?{/quote]POE gives the board consistent and reliable power that is mirrored by its connection to the POE switch. With a separate power supply you could lose power while the board remains connected or could remain powered when the Ethernet is disconnected. I chose POE so I could be sure my Arduinos remain powered despite any mains power fluctuations. I have a 48-port POE switch supplied by a UPS that can run it for 30 minutes or so. If we loose power (which has happened twice in the last 5 months) a generator is automatically started within a couple of minutes. The server, my router and modem are also on UPS power. All of the automation and network connectivity survives the power interruptions.

7. What else?

Best regards
Doug

I think that covers it from my perspective. I don't know why some of you are having connection problems, while I have none. The only way I was ever able to create a connection issue was by removing power from the board(s) while the Ethernet was connected. That all ended with POE in late 2014. I have not used USB connections - ever.

[DMcKnight]

Hi Randy,
Thanks for the detailed reply. Are you running plugin version 1.0.0.3?

I'll do some more experiments with swapping hardware, and perhaps also looking at the power supply integrity. The stability of your system is encouraging, for sure. I should probably look into trying out some different ethernet hardware. My Uno and Mega boards are "genuine" but the ethernet board on my Mega is a clone with the 5100 chipset. When it does connect it seems solid, but that's not much use if I can't connect...

Doug

[randy]

Hi Randy,
Thanks for the detailed reply. Are you running plugin version 1.0.0.3?

I'll do some more experiments with swapping hardware, and perhaps also looking at the power supply integrity. The stability of your system is encouraging, for sure. I should probably look into trying out some different ethernet hardware. My Uno and Mega boards are "genuine" but the ethernet board on my Mega is a clone with the 5100 chipset. When it does connect it seems solid, but that's not much use if I can't connect...

Doug

Yes - 1.0.0.103. The only problem I had with a clone mega board was an output pin that would change state on its own. That I one of 3 boards. With my clone Ethernet boards, I purchased 5. One was DOA and 2 of the remaining 4 are definitely intermittent in connecting. They're still in my parts bin and I have tried them on my test board subsequent to my retiring them and they will still introduce intermittent connection problems.

With regard to power supply there is not a lot of conditioning in the Arduino boards so I believe it is very important for the supply to be clean and stable. I ended up with a "master" 12V supply in my basement with some 5V DC-DC converters in the two closets where two of my Arduinos are mounted. I also believe that if they are powered by 5V it is more critical that the supply is clean, since the Arduino boards use analog 5V regulators.

[DMcKnight]

Right, it ought to be a rather clean 5V if you are bypassing the regulator. The regulator on the Arduino drops a volt, so to use it to regulate you need at least 6 (and a bit) volts on that power input.
I don't very much like using these cheapo 9V supplies that seem commonly suggested for Arduinos. The regulator gets pretty toasty.

If I could find a reliable, quiet, 6.5V or 7V wall-wart I'd buy a bunch of them. The "nice" solutions of POE or centralized power supply are annoyingly hard to implement in our house, but it may come to that. We've had three power outages in the last 12 hours. One of them still ongoing...

Doug

[petez69]

Doug

I'm planning to use centralised power...12v out to the devices with a 5v dc to dc regulator at each board. I've tried to use the onboard regulator on the mega and they get very hot..

Pete

[DMcKnight]

Well, I think I may have figured out the cause of my weird ethernet connection problems!

Here's what I think was going on. Recall that board number 3 is the board running the API sketch, and I was having trouble getting it to connect. When it did connect, the other two boards tended to eventually lose their connections. If I powered off this board *and* restarted the plugin, then boards 1 and 2 could connect normally.

It seemed that board number 3 was somehow causing problems. Recall also that I said I was taking care to make sure that the code was only updating HS every second, so as to not overwhelm the plugin with communications. Well, the good news is that that was a reasonable thing to worry about, the bad news was that I screwed up with a misplaced "}" in the code. Instead of sending data every second, it was sending data every loop... This, possibly combined with a slightly puny HS3 machine, seems to cause the plugin to get into a weird state. It doesn't exactly crash, but it seems to need a restart to get the connections working again.

So, I fixed the code to make it do what I meant it to do about 3 days ago, and it seems to work just fine! The debug logs from the plugin were a bit of a help, but this problem wasn't completely obvious to fix because the plugin kept almost sort of working even when I was accidentally abusing it like this.

It's only been working for a few hours, but the difference in behavior is so clear that I'm pretty confident that this was the problem. Hopefully this thread will help someone else with a problem like this. I'll update if something changes.

regards
Doug

[apluck]

I am using v101.

Board 1 is a genuine Uno with a genuine ethernet shield. This drops connection every couple of days but instantly reconnects. It has done this since I installed it 2 years ago.

Board 2 is a genuine ArduinoBoardEthernet with POE. This hard crashes every few days and the board needs to be power cycled to reset it. Last week I replaced this board with a new Genuino Uno with genuine ethernet shield which so far has worked fine.

Board 3 is a genuine ArduinoBoardEthernet with POE, bought at the same time as board 2 in May 2015. It has worked perfectly and not dropped connection once.

Board 4 is a genuine ArduinoBoardEthernet with POE, bought at the same time as board 2 in May 2015. It drops connection every couple of days but instantly reconnects.

Board 5 is a genuine Uno bought at the same time as board 1 with a genuine ethernet shield. It has a LCD connected to it, which updates once at powerup, and then disconnects from the plugin and doesn't update again. I tried replacing the Uno with a new Genuino Uno using the same ethernet shield and the LCD works fine.


In summary, my 2 genuine arduinos from May 2014 don't work very well. 1 of my 3 ArduinoBoardEthernets from May 2015 works perfectly, and the other 2 don't. The new Genuino bought last week is so far working ok.

I'm not able to trust the Arduinos, and have had to go back to my Netiom xAP boards for the more important I/O. The Netioms have been rock solid since 2007.

[petez69]

aPluck

Thanks for sharing, sadly a few of us are taking heart that its "not just me" that are experiencing issues.

My worst experience came with a "Freetronics Etherten", it kept dropping off and it just didn't make sense yet it cost a lot of money as its considered a reputable product.

I've currently got 4 Megas with clone ethernet boards and they seem to be rock solid and as a result I've got a pile of original and clone boards that I need to throw in the bin as they just do weird things.

I've got a Nano with a tiny W5100 board in API mode and that seems to be solid, still testing.

The arduino environment does need testing and tweaking to get good combinations..

Pete

[apluck]

To add another datapoint.

My HS3 is a Windows 10 KVM guest VM running under unRaid. HS3 is reasonably busy with 500 devices.

To eliminate the Network, unRaid, Server, and my HS3 install as problem sources, I did a clean install of Win10 on a separate physical PC attached to a separate network switch, with the only other network connection being an Arduino with the LCD attached. I forced the PC connection to downgrade to 100Mb. I did a clean install of HS3, then installed the plugin as a trial.

The LCD had the same behaviour as on my production install. It still only updated once on powerup, then the connection failed.

[petez69]

aPluck

You mentioned you've been testing on UNOs. Grab a Mega and try same, I've found the megas to play a little nicer.

Pete