Learning About Arduino Ethernet
Reducing amount of data transfer with official Arduino Ethernet Library
( 7 March 2009 )
As observed in the forum thread "Ethernet Shield sending byte by byte?" the current version of the Ethernet library sends data one byte at a time. This is because the Print class is implemented to operate in this manner and the Ethernet Client class sub-classes Print currently.
That the Print class is implemented in such a way doesn't really matter for character-orientated transport mechanisms such as Serial implements but it has a major impact for Ethernet due to overhead caused by the underlying transport.
I observed with WireShark when one byte is sent then around 60 bytes of packet overhead is sent with it plus more overhead is caused because the target must also acknowledge each packet.
As a first step to working around this I modified the print method in Client class to send a whole string if it is called with a character array, instead of calling the single character print method multiple times.
Data Transfer with WebServer example Packets Bytes Packets A->B Bytes A->B Packets A<-B Bytes A<-B Original 444 25829 223 12569 221 13260 New 116 7201 59 3713 57 3488
There's still quite a bit of unnecessary overhead in this example as there's actually only 218 bytes of real data returned in the response. This is because each three digit number is returned as individual digits. But even with the one change it's still about 75% fewer bytes transferred.
The modified versions of Client.h and Client.cpp are here: Client.h Client.cpp. They need to be copied over the existing files in hardware/libraries/Ethernet and the library recompiled before use (re-select the board from the Tools > Board menu to recompile). Make a backup first. I have only tested it with the standard WebServer example. Usual disclaimers apply.
A better solution would perhaps be to offer the option to automatically buffer sent data on the WIZnet before sending. The question would then be how to decide when to actually send. There's a few options:
- Send on a request to close the connection to the client which would be suitable for something like a web server interaction.
- Send on an end-of-line transmission.
- Send on a manual flush method call.
- Send when some arbitrary buffer size is reached.
An alternative would be offer a "bulk send" method but that seems less attractive to me.
The current behaviour should probably be the default as it has the element of least surprise but given the overhead I think offering a flush option—manual or automatic—would be valuable.
(The following notes refer to the ENC28J60, see more recent progress with the WIZ810MJ on Learning About Arduino WIZ810MJ.)
My notes about Arduino and Ethernet.
- Interfacing with the ENC28J60 -- Useful Arduino forum post started by xSmurf. (Initial code) (Up-to-date Trac instance)
- An AVR microcontroller based Ethernet device -- Comprehensive generic AVR ethernet design.
Parts suppliers (who sample) from xSmurf: @@ TODO: Tidy this up with links
- the ENC28J60 at microchip
- the crystal (you need a 25Mhz xtal for the enc) can be obtained at foxonline.com (or needasample.com) with SKU FOXSLF/250F-20.
- the MagJack comes from erni.com and is sku 203322 (Note: this maybe an outdated part number)
- luxtronic.com (might be luxtronics.com) have magjacks too
"with that, all you need is a few resistors and caps and a 3.3v supply (which you have if you have a Diecimila)" -- "in the passive stuff, you'll want 2x~20pF caps for the xtal, a 10µF polarized caps, 2x 10µF, 2x 150ohm, 4x 50ohm, 2x 10k ohm, 1x 2.7K ohm, if possible a 0.5cm ferrite bead with about 6 turns of thin wire"
IP/ICMP Checksum-related resources
- IP CHECKSUM CALCULATION and Useful IP Checksum Explanation
- IPv4 Packet Header
- IP Header Checksum Code
A valid reply packet (with correct checksum 0x4b9c—created using Python Impacket library) :
45 00 00 54 3c 88 00 00 ff 01 39 ce c0 a8 00 7e c0 a8 00 0c 00 00 4b 9c 90 66 00 00 50 52 ea 46 f0 60 0e 00 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37
#!/usr/bin/python import sys sys.path.insert(0, 'Impacket-0.9.6.0') from impacket import ImpactPacket icmp = ImpactPacket.ICMP() ip = ImpactPacket.IP() ip.set_ip_src("192.168.0.126") ip.set_ip_dst("192.168.0.12") icmp.set_icmp_type(icmp.ICMP_ECHOREPLY) icmp.contains(ImpactPacket.Data('PR\xeaF\xf0`\x0e\x00\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,-./01234567')) ip.set_ip_id(0x3c88) ip.set_ip_ttl(64) ip.contains(icmp) icmp.set_icmp_id(0x9066) icmp.set_icmp_cksum(0) icmp.auto_checksum = 1 #print repr(ip.get_packet()) print " ".join(["%02x" % ord(c) for c in ip.get_packet()])
45 00 00 54 3c 88 00 00 40 01 bc 46 c0 a8 00 7e c0 a8 00 0c 00 00 4b 9c 90 66 00 00 50 52 ea 46 f0 60 0e 00 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f 30 31 32 33 34 35 36 37
- Wake-on-Lan and ENC28J60 -- Don't you love it when people beat you to it... (See also: Includes semi-auto-discovery method.)