Learning About Arduino Ethernet

Reducing amount of data transfer with official Arduino Ethernet Library

( 7 March 2009 )

As observed in the forum thread "Ethernet Shield sending byte by byte?" the current version of the Ethernet library sends data one byte at a time. This is because the Print class is implemented to operate in this manner and the Ethernet Client class sub-classes Print currently.

That the Print class is implemented in such a way doesn't really matter for character-orientated transport mechanisms such as Serial implements but it has a major impact for Ethernet due to overhead caused by the underlying transport.

I observed with WireShark when one byte is sent then around 60 bytes of packet overhead is sent with it plus more overhead is caused because the target must also acknowledge each packet.

As a first step to working around this I modified the print method in Client class to send a whole string if it is called with a character array, instead of calling the single character print method multiple times.

 Data Transfer with WebServer example 

           Packets    Bytes   Packets A->B    Bytes A->B   Packets A<-B	Bytes A<-B 
 Original      444    25829            223         12569            221       13260

 New           116     7201             59          3713             57        3488

There's still quite a bit of unnecessary overhead in this example as there's actually only 218 bytes of real data returned in the response. This is because each three digit number is returned as individual digits. But even with the one change it's still about 75% fewer bytes transferred.

The modified versions of Client.h and Client.cpp are here: Client.h Client.cpp. They need to be copied over the existing files in hardware/libraries/Ethernet and the library recompiled before use (re-select the board from the Tools > Board menu to recompile). Make a backup first. I have only tested it with the standard WebServer example. Usual disclaimers apply.

A better solution would perhaps be to offer the option to automatically buffer sent data on the WIZnet before sending. The question would then be how to decide when to actually send. There's a few options:

An alternative would be offer a "bulk send" method but that seems less attractive to me.

The current behaviour should probably be the default as it has the element of least surprise but given the overhead I think offering a flush option—manual or automatic—would be valuable.

Older notes

(The following notes refer to the ENC28J60, see more recent progress with the WIZ810MJ on Learning About Arduino WIZ810MJ.)

My notes about Arduino and Ethernet.

Parts sources

Parts suppliers (who sample) from xSmurf: @@ TODO: Tidy this up with links

"with that, all you need is a few resistors and caps and a 3.3v supply (which you have if you have a Diecimila)" -- "in the passive stuff, you'll want 2x~20pF caps for the xtal, a 10µF polarized caps, 2x 10µF, 2x 150ohm, 4x 50ohm, 2x 10k ohm, 1x 2.7K ohm, if possible a 0.5cm ferrite bead with about 6 turns of thin wire"

IP/ICMP Checksum-related resources

A valid reply packet (with correct checksum 0x4b9c—created using Python Impacket library) :

45 00 00 54 3c 88 00 00 ff 01 39 ce c0 a8 00 7e c0 a8 00 0c 00 00 4b 9c 90 66 00 00 50 52 ea 46 f0 60 0e 00 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37

The code:


import sys
sys.path.insert(0, 'Impacket-')

from impacket import ImpactPacket

icmp = ImpactPacket.ICMP()
ip = ImpactPacket.IP()
icmp.contains(ImpactPacket.Data('PR\xeaF\xf0`\x0e\x00\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./01234567'))
icmp.auto_checksum = 1
#print repr(ip.get_packet())

print " ".join(["%02x" % ord(c) for c in ip.get_packet()])


45 00 00 54 3c 88 00 00 40 01 bc 46 c0 a8 00 7e c0 a8 00 0c 00 00 4b 9c 90 66 00 00 50 52 ea 46 f0 60 0e 00 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37