I had the pleasure of attending the Ethernet Technology Summit last week in San Jose. In addition to presenting as part of the FCoE track, I was able to spend a day getting updated on 40Gb and 100Gb Ethernet from the people and vendors involved in creating the standards which are both expected to be ratified in June 2010. While most enterprise customers are only now starting to deploy 10Gb Ethernet, the completion of the higher speeds are very important developments. Traditionally, the higher speeds allow for greater utilization of the previous generation; we have seen that 10Gb is first deployed at the backbone for environments that have deployed 1Gb at the server. There are 2 significant changes that I’ll highlight about the next instantiation of Ethernet:
- There are new speeds – 40Gb and 100Gb
- There is expected to be a shift in cabling from a copper to optical
Why we need both 40Gb and 100Gb Ethernet
Ethernet has always moved in 10x increments from 10Mb > 100Mb > 1Gb > 1oGb. Below is a 2007 IEEE forecast of server adoption of Ethernet Connections. While the standard for 10Gb Ethernet was ratified in 2002, it wasn’t until 2007 until server adoption began and it was 2009/2010 before we saw significant customer deployments in their datacenters and servers.
IEEE April 2007 prediction from http://www.ieee802.org/3/hssg/public/apr07/hays_01_0407.pdf
The presenters at the conference made a compelling case that server IO doubled every 24 months, while core networking doubled every 18 months. Server bus architectures must also mature to take advantage of the high bandwidth interconnect. This led to the idea to create 100Gb for the core (between switches) and 40Gb for the distribution/aggregation (pedestal/rack/blade servers to switches). As for the uses for these speeds, it is the next generation of servers which are characterized by dense computing and high utilization through virtualization which will use 40Gb and 100Gb will enable the success of 10Gb servers.
Is it finally the end of copper?
When 10Gb Ethernet was first ratified as a standard, optical was the only option that was available. The idea of creating 10GBase-T (using RJ45 connected cables such as CAT6/6a) was around in 2002, became a standard in 2006 and just last month, Cisco announced their first product supporting this option. With the ratification of 40Gb and 100Gb coming in only a few months, we once again find that the options for cabling are optical and a short-distance copper (no UTP). A note on cabling:
10Gb Ethernet currently supports optical (300m support w/ OM3 multimode fibre), Twinax (SFP+DA copper w/ lengths varying by vendor, but < 10m) and 10GBase-T (CAT6 at 55m; CAT6a at 100m; note that today there are no FCoE solutions supporting 10GBase-T).
Both 40Gb and 100Gb Ethernet will have a copper option up to 7m (QSFP connector – this is what is used in InfiniBand today, not what is used for 10Gb) and multimode optical up to 100m (and singlemode up to 10km).
The official position of the Ethernet Alliance is that the adoption of 40/100 will see us shift from UTP to optical. With 1Gb Ethernet and earlier generations, over 99% of all cabling deployments were UTP/RJ45. The price, power requirements, distance limitations and other technical hurdles have been more and more difficult to overcome with each generation. Storage customers using Fibre Channel are already using optical cabling, so for some customers that are converging the SAN and LAN into a single network with 10Gb Ethernet, the migration to an all optical configuration is easy. For the legacy of customers with billions of ports of existing cabling infrastructure, it will be an analysis of whether they can reuse their existing environment. In new datacenter builds, there will need to be a determination as to how UTP and optical cabling options match the expected maturing of technology over the lifetime of a deployment (typically 5-10 years).
Outlook
Adoption of Ethernet speeds may take many years, but the availability of the next speeds provide investment protection and a path for continued growth. Ethernet may be ubiquitous, but there as practitioners roll-out 10Gb Ethernet, they should become familiar with the 40Gb and 100Gb to understand how decisions that they make today may allow for adoption of future technologies.
Comments and clarifications are welcome. If you’re new to this site, please consider subscribing to this blog.
Stuart Miniman
Are you on Google Buzz? Find me at http://www.google.com/profiles/stuminiman
Blog Stu - Stuart Miniman 
Nice post Stu. Thanks for the update.
Stu
Do we have any idea at this stage as to whether the 40GbE / 100GbE standards will use OM3 fibre or will a higher specification fibre be recommended ?
Greg
Greg – in my Converged Network presentation which you’ve seen, on slide 11 it shows that 40GbE and 100GbE will support 100m with OM3 fibre. OM4 will support 125m.
Thanks. And that presentation is great.
Hi Stu,
Any word on the progress of 802.3ba? It is due this month, correct? Looks like a plenary in July.
Regards,
Scott
Scott,
My understanding matches yours that we should be seeing this wrapping up at IEEE (at the Ethernet Technology Summit in Feb they said June) and the next 802.3 plenary is in the week of July 12th in San Diego. I’d expect to see some announcements at that time – the Ethernet Alliance is working on marketing: “planning is underway for a 40G and 100G presence at SC10, and a new set of collateral is being developed to help educate and increase public awareness of applications of higher speed Ethernet”.
Cheers,
Stu
Just saw this on Twitter http://www.ieee802.org/3/hssg/email/msg01822.html
June 17, 2010
“Dear Task Force Participants,
This morning the IEEE SASB approved IEEE P802.3ba.
I would like to extend my gratitude to all who participated in this effort. It was truly a team effort by the members of our industry. I would also like to thank Ilango, Mark, Pete, Chris, and the rest of the editorial team for all of their hard word. Finally, I would like to thank Mr. Grow, Mr. Law, Mr. Diab, and Mr. Carlson, for all of their help during the course of this project.
Regards,
John D’Ambrosia
Chair, IEEE P802.3ba Task Force “
100GB ethernet will give you some serious connection speed.
[…] different aspect of data centers, the network cabling. As you’ll see in the first picture, Ethernet cables can be a nightmare if they aren’t carefully organized. However, when they are organized, […]