What are jumbo frames?

We could spend 30 days discussing that question and not reach a satisfying answer. The “Ethernet Alliance” published a position paper in November of 2009 that summarized things pretty well. Long-story short: They’re Ethernet frames, without the 1500-byte payload limit.

How do they work?

The Ethernet II specification defines a 1518 byte frame format. The most recent 802.3-2015 specifications define a mechanism for defining an MTU of up to 2000 bytes (adding multiple layers of VLAN/TOS tagging to the 18-byte Ethernet II headers). Modern implementations of Ethernet often include a feature commonly referred to as “jumbo frames”, which is not formalized within the IEEE 802.3 standards family.

9000-bytes is the generally used value for the number of payload bytes in an Ethernet jumbo frame (hence, the corresponding IP MTU), with corresponding Ethernet frame sizes of 9014, 9022, or as high as 9216 bytes. Vendor documentation is often inconsistent with regards to references to jumbo frame support as either the maximum Layer-2 frame size or the maximum Layer-2 payload size (the value to use for an associated IP interface’s MTU.)

But as for the actual mechanics of pushing a “jumbo frame” where an 802.3-conformant frame is expected:

  • There’s no “length” field in the Ethernet headers to worry about
    • The Ethertype field is canonically interpreted as the frame length only for values <=1500 bytes.
  • Since an Ethernet frame doesn’t have a fixed length or fixed header field, we have to wait for the interframe-gap to be sure we’ve seen the end of the frame.
  • There is the cyclic-redundancy-check (CRC) at the end of frame that we need to worry about.
    • The last four bytes of an Ethernet frame are defined as a CRC.
    • Ethernet uses the CRC-32 algorithm, which…
      • Is a one-way hash function.
      • And has roughly equivalent (and acceptable to the IEEE) probability of undetected errors for frame-sizes ranging from 376 to 11,455 bytes
        • With the undetected error probability increasing rapidly above frame lengths of 11,455 bytes

I couldn’t find any other sources for this assertion. It seems feasible, but I’m no cryptanlyst. As frame size increases, the number of possible messages increases, but the number of checksum values remains the same. There has to be some message-size past which false positives would become problematic, but whether that’s the case for specific types of false-positives (single-bit errors; transpositions; deletsions; etc.), I have no idea. For all I know, the folks at Alteon had some chipset issue that made <11,000 bytes way more attractive, and they cherry-picked transmission-error types to find one where the curve took off around 11K. Anyway, I’d take this with a grain of salt – and it makes no difference to this analysis anyway.

Advantages of Jumbo Frames

Larger MTUs have a positive correlation with increased TCP throughput (under certain conditions)

Table of Maximum TCP throughput for various latency

Show/hide table
TCP Segment Size* Latency 1460 2920 5840 8760
1ms 116.8gbps 233.7gbps 467.2gbps 700.8gbps
4ms 29.2gbps 58.4gbps 109.6gbps 175.2gbps
8ms 14.6gbps 29.2gbps 54.8gbps 87.6gbps
12ms 9.7gbps 19.5gbps 36.5gbps 58.4gbps
16ms 7.3gbps 14.6gbps 27.4gbps 43.8gbps
20ms 5.8gbps 11.7gbps 21.9gbps 35.0gbps
40ms 2.9gbps 5.8gbps 10.9gbps 17.5gbps
80ms 1.4gbps 2.9gbps 5.4gbps 8.8gbps

*TCP segment size is assumed to be IP MTU minus 40 bytes for TCP headers.

MTU sizes are inversely correlated with CPU utilization on sending/receiving hosts
  • A larger MTU means more data to be transported in fewer packets (per total payload data).
  • Fewer packets means less CPU cycles spent parsing packet headers (per total payload data).
    • Calculating Ethernet and TCP CRC values
    • Validating MAC and IP addressing
    • Etc…

Dis-advantages of Jumbo Frames

  • Lack of universal standard frame size makes integration difficult
  • Can increase latency of Ethernet switching
  • Can drastically increase queue-size requirements for Ethernet switches
  • Require end-to-end enablement (with no corresponding discovery mechanism)

Why ~9,000 bytes?

The functional upper boundary for jumbo-frame size is dictated by the 802.3 Ethernet frame format. Specifically, the 4-byte “frame check sequence” (FCS) that trails the payload section of each Ethernet frame. The FCS field is a “cyclical redundancy check” (CRC) which uses a one-way hashing function to validate that the data in the remainder of the Ethernet frame was received correctly. When a host receives an Ethernet frame, it runs the CRC calculation on the Ethernet frame (up to the FCS portion), and then compares its calculated value to that of the FCS value in the received frame. If the values match, the frame is assumed to have been transmitted correctly.

Apocryphally, the CRC has function used by Ethernet hits an inflection point in the probability of false positives when frame size exceeds 11,455 bytes. The only source I’ve found for this is a whitepaper published by NTT

Correspondingly, network transport of storage traffic does (or did, at one point) provide a lower threshold for achieving maximum benefit from jumbo frames. The preponderance of 8,192-byte block sizes on physical file systems (and their corresponding delivery as “blocks” by the NFS protocol) is significant here. An Ethernet frame with an entire NFS-on-TCP-on-IP payload carrying the contents of an 8,192 byte block would be approximately 8,500 bytes in size.

How the industry arrived at ~9,000 bytes as the happy medium between 8,500 and 11,455 unknown to this author. (But, it probably doesn’t hurt that it’s a multiple of 1500, is above the 8400-byte magic number for block-storage, and under the 11,455 CRC “limit”.)