40Gb and 100Gb Ethernet status and outlook

March 5, 2010

I had the pleasure of attending the Ethernet Technology Summit last week in San Jose.  In addition to presenting as part of the FCoE track, I was able to spend a day getting updated on 40Gb and 100Gb Ethernet from the people and vendors involved in creating the standards which are both expected to be ratified in June 2010.  While most enterprise customers are only now starting to deploy 10Gb Ethernet, the completion of the higher speeds are very important developments.  Traditionally, the higher speeds allow for greater utilization of the previous generation; we have seen that 10Gb is first deployed at the backbone for environments that have deployed 1Gb at the server.  There are 2 significant changes that I’ll highlight about the next instantiation of Ethernet:

  1. There are new speeds – 40Gb and 100Gb
  2. There is expected to be a shift in cabling from a copper to optical

Why we need both 40Gb and 100Gb Ethernet

Ethernet has always moved in 10x increments from 10Mb > 100Mb > 1Gb > 1oGb.  Below is a 2007 IEEE forecast of server adoption of Ethernet Connections.  While the standard for 10Gb Ethernet was ratified in 2002, it wasn’t until 2007 until server adoption began and it was 2009/2010 before we saw significant customer deployments in their datacenters and servers.

The presenters at the conference made a compelling case that server IO doubled every 24 months, while core networking doubled every 18 months.  Server bus architectures must also mature to take advantage of the high bandwidth interconnect.  This led to the idea to create 100Gb for the core (between switches) and 40Gb for the distribution/aggregation (pedestal/rack/blade servers to switches).  As for the uses for these speeds, it is the next generation of servers which are characterized by dense computing and high utilization through virtualization which will use 40Gb and 100Gb will enable the success of 10Gb servers.

Is it finally the end of copper?

When 10Gb Ethernet was first ratified as a standard, optical was the only option that was available.  The idea of creating 10GBase-T (using RJ45 connected cables such as CAT6/6a) was around in 2002, became a standard in 2006 and just last month, Cisco announced their first product supporting this option.  With the ratification of 40Gb and 100Gb coming in only a few months, we once again find that the options for cabling are optical and a short-distance copper (no UTP).  A note on cabling:

10Gb Ethernet currently supports optical (300m support w/ OM3 multimode fibre), Twinax (SFP+DA copper w/ lengths varying by vendor, but < 10m) and 10GBase-T (CAT6 at 55m; CAT6a at 100m; note that today there are no FCoE solutions supporting 10GBase-T).

Both 40Gb and 100Gb Ethernet will have a copper option up to 7m (QSFP connector – this is what is used in InfiniBand today, not what is used for 10Gb) and multimode optical up to 100m (and singlemode up to 10km).

The official position of the Ethernet Alliance is that the adoption of 40/100 will see us shift from UTP to optical.  With 1Gb Ethernet and earlier generations, over 99% of all cabling deployments were UTP/RJ45.  The price, power requirements, distance limitations and other technical hurdles have been more and more difficult to overcome with each generation.  Storage customers using Fibre Channel are already using optical cabling, so for some customers that are converging the SAN and LAN into a single network with 10Gb Ethernet, the migration to an all optical configuration is easy.  For the legacy of customers with billions of ports of existing cabling infrastructure, it will be an analysis of whether they can reuse their existing environment.  In new datacenter builds, there will need to be a determination as to how UTP and optical cabling options match the expected maturing of technology over the lifetime of a deployment (typically 5-10 years).

Outlook

Adoption of Ethernet speeds may take many years, but the availability of the next speeds provide investment protection and a path for continued growth.  Ethernet may be ubiquitous, but there as practitioners roll-out 10Gb Ethernet, they should become familiar with the 40Gb and 100Gb to understand how decisions that they make today may allow for adoption of future technologies.

Comments and clarifications are welcome.  If you’re new to this site, please consider subscribing to this blog.

Stuart Miniman

https://blogstu.wordpress.com

Are you on Google Buzz?  Find me at http://www.google.com/profiles/stuminiman

Share