Mar 2009
9.9856Gbps It's the Law
17/03/09 07:52 Filed in: 10GbE
Someone said yesterday that we were wrong and that the speed limit for 10GbE was really 10.3125Gbps, so dumbstruck I needed to validate our claim of 9.95Gbps.
Most NICs talk XAUI to a PHY driver chip that then controls the media. XAUI has four pairs of receive/transmit lanes that each operating at 3.125Gbps and utilize 8/10b encoding. So after decoding you have 10.000Gbps of actual usable media bandwidth, hence 10G. Ethernet adds it's own shipping and handling fee though and so people who measure bandwidth using the operating system (OS) on their server will never see the full capacity of the pipe due to this overhead.
Ethernet is then traveling over XAUI and it requires an interframe gap (IFG) spacing of 96 bits, more precisely the time it would take to transmit 96 bits (actually 8 octets) which on 10GbE is 9.6ns. It turns out though that the 10GbE spec actually redefined the gap to 40 bits or 4.0ns. So between every packet there is 4ns of air.
Ethernet also requires a seven byte preamble and a one byte start of frame delimiter, that's 64 more bits or another 6.4ns of air. So between any two actual "packets" on a 10GbE wire this is a total 10.4ns of dead air. Well how does that impact the actual OS measurable bandwidth?
Let's look at the detailed best case, a jumbo frame. First there is the 7 octet preamble, 1 octet start-of-frame delimiter, then what would be called the "Jumbo Frame" which is 9,018 octets (it's really a 9000 octet payload with a 14 octet header and a 4 octet CRC at the end) then the IGF or 5 octets in the case of 10GbE. When people typically measure NIC bandwidth they only measure the "packet" or 9018 byte part, and often don't know about those other 13 required bytes. So a 10GbE NIC running jumbo frames can never achieve more than (9018/9031)*10Gbps or 9.9856Gbps and a good wire-rate NIC will demonstrate this.
Now in the worst case we have the dreaded "64 byte" frame which has the same 13 bytes of stuff between "frames" so the that calculation is (64/77)*10Gbps or 8.3117Gbps. So NEVER expect your 10GbE NIC to deliver 10Gbps using 64 byte frames (which really only contain 46 bytes of actual data) because Ethernet will shackle it to 8.3117Gbps. If your application just measures actual payload then it's 46/77*10Gbps or 5.97Gbps.
So if you drive a 9,000 byte tractor trailer you can speed along at 9.9856Gbps, but if you're tooling along on your little 64 byte rice rocket you'll never get above 5.97Gbps.
Most NICs talk XAUI to a PHY driver chip that then controls the media. XAUI has four pairs of receive/transmit lanes that each operating at 3.125Gbps and utilize 8/10b encoding. So after decoding you have 10.000Gbps of actual usable media bandwidth, hence 10G. Ethernet adds it's own shipping and handling fee though and so people who measure bandwidth using the operating system (OS) on their server will never see the full capacity of the pipe due to this overhead.
Ethernet is then traveling over XAUI and it requires an interframe gap (IFG) spacing of 96 bits, more precisely the time it would take to transmit 96 bits (actually 8 octets) which on 10GbE is 9.6ns. It turns out though that the 10GbE spec actually redefined the gap to 40 bits or 4.0ns. So between every packet there is 4ns of air.
Ethernet also requires a seven byte preamble and a one byte start of frame delimiter, that's 64 more bits or another 6.4ns of air. So between any two actual "packets" on a 10GbE wire this is a total 10.4ns of dead air. Well how does that impact the actual OS measurable bandwidth?
Let's look at the detailed best case, a jumbo frame. First there is the 7 octet preamble, 1 octet start-of-frame delimiter, then what would be called the "Jumbo Frame" which is 9,018 octets (it's really a 9000 octet payload with a 14 octet header and a 4 octet CRC at the end) then the IGF or 5 octets in the case of 10GbE. When people typically measure NIC bandwidth they only measure the "packet" or 9018 byte part, and often don't know about those other 13 required bytes. So a 10GbE NIC running jumbo frames can never achieve more than (9018/9031)*10Gbps or 9.9856Gbps and a good wire-rate NIC will demonstrate this.
Now in the worst case we have the dreaded "64 byte" frame which has the same 13 bytes of stuff between "frames" so the that calculation is (64/77)*10Gbps or 8.3117Gbps. So NEVER expect your 10GbE NIC to deliver 10Gbps using 64 byte frames (which really only contain 46 bytes of actual data) because Ethernet will shackle it to 8.3117Gbps. If your application just measures actual payload then it's 46/77*10Gbps or 5.97Gbps.
So if you drive a 9,000 byte tractor trailer you can speed along at 9.9856Gbps, but if you're tooling along on your little 64 byte rice rocket you'll never get above 5.97Gbps.