40Gb and 100Gb Ethernet status and outlook

I had the pleasure of attending the Ethernet Technology Summit last week in San Jose.  In addition to presenting as part of the FCoE track, I was able to spend a day getting updated on 40Gb and 100Gb Ethernet from the people and vendors involved in creating the standards which are both expected to be ratified in June 2010.  While most enterprise customers are only now starting to deploy 10Gb Ethernet, the completion of the higher speeds are very important developments.  Traditionally, the higher speeds allow for greater utilization of the previous generation; we have seen that 10Gb is first deployed at the backbone for environments that have deployed 1Gb at the server.  There are 2 significant changes that I’ll highlight about the next instantiation of Ethernet:

  1. There are new speeds – 40Gb and 100Gb
  2. There is expected to be a shift in cabling from a copper to optical

Why we need both 40Gb and 100Gb Ethernet

Ethernet has always moved in 10x increments from 10Mb > 100Mb > 1Gb > 1oGb.  Below is a 2007 IEEE forecast of server adoption of Ethernet Connections.  While the standard for 10Gb Ethernet was ratified in 2002, it wasn’t until 2007 until server adoption began and it was 2009/2010 before we saw significant customer deployments in their datacenters and servers.

The presenters at the conference made a compelling case that server IO doubled every 24 months, while core networking doubled every 18 months.  Server bus architectures must also mature to take advantage of the high bandwidth interconnect.  This led to the idea to create 100Gb for the core (between switches) and 40Gb for the distribution/aggregation (pedestal/rack/blade servers to switches).  As for the uses for these speeds, it is the next generation of servers which are characterized by dense computing and high utilization through virtualization which will use 40Gb and 100Gb will enable the success of 10Gb servers.

Is it finally the end of copper?

When 10Gb Ethernet was first ratified as a standard, optical was the only option that was available.  The idea of creating 10GBase-T (using RJ45 connected cables such as CAT6/6a) was around in 2002, became a standard in 2006 and just last month, Cisco announced their first product supporting this option.  With the ratification of 40Gb and 100Gb coming in only a few months, we once again find that the options for cabling are optical and a short-distance copper (no UTP).  A note on cabling:

10Gb Ethernet currently supports optical (300m support w/ OM3 multimode fibre), Twinax (SFP+DA copper w/ lengths varying by vendor, but < 10m) and 10GBase-T (CAT6 at 55m; CAT6a at 100m; note that today there are no FCoE solutions supporting 10GBase-T).

Both 40Gb and 100Gb Ethernet will have a copper option up to 7m (QSFP connector – this is what is used in InfiniBand today, not what is used for 10Gb) and multimode optical up to 100m (and singlemode up to 10km).

The official position of the Ethernet Alliance is that the adoption of 40/100 will see us shift from UTP to optical.  With 1Gb Ethernet and earlier generations, over 99% of all cabling deployments were UTP/RJ45.  The price, power requirements, distance limitations and other technical hurdles have been more and more difficult to overcome with each generation.  Storage customers using Fibre Channel are already using optical cabling, so for some customers that are converging the SAN and LAN into a single network with 10Gb Ethernet, the migration to an all optical configuration is easy.  For the legacy of customers with billions of ports of existing cabling infrastructure, it will be an analysis of whether they can reuse their existing environment.  In new datacenter builds, there will need to be a determination as to how UTP and optical cabling options match the expected maturing of technology over the lifetime of a deployment (typically 5-10 years).

Outlook

Adoption of Ethernet speeds may take many years, but the availability of the next speeds provide investment protection and a path for continued growth.  Ethernet may be ubiquitous, but there as practitioners roll-out 10Gb Ethernet, they should become familiar with the 40Gb and 100Gb to understand how decisions that they make today may allow for adoption of future technologies.

Comments and clarifications are welcome.  If you’re new to this site, please consider subscribing to this blog.

Stuart Miniman

https://blogstu.wordpress.com

Are you on Google Buzz?  Find me at http://www.google.com/profiles/stuminiman

Share

9 Responses to 40Gb and 100Gb Ethernet status and outlook

  1. Brad Hedlund says:

    Nice post Stu. Thanks for the update.

  2. Etherealmind says:

    Stu

    Do we have any idea at this stage as to whether the 40GbE / 100GbE standards will use OM3 fibre or will a higher specification fibre be recommended ?

    Greg

  3. The Corfu says:

    Hi Stu,

    Any word on the progress of 802.3ba? It is due this month, correct? Looks like a plenary in July.

    Regards,

    Scott

    • Stuart Miniman says:

      Scott,
      My understanding matches yours that we should be seeing this wrapping up at IEEE (at the Ethernet Technology Summit in Feb they said June) and the next 802.3 plenary is in the week of July 12th in San Diego. I’d expect to see some announcements at that time – the Ethernet Alliance is working on marketing: “planning is underway for a 40G and 100G presence at SC10, and a new set of collateral is being developed to help educate and increase public awareness of applications of higher speed Ethernet”.
      Cheers,
      Stu

    • Stuart Miniman says:

      Just saw this on Twitter http://www.ieee802.org/3/hssg/email/msg01822.html
      June 17, 2010
      “Dear Task Force Participants,

      This morning the IEEE SASB approved IEEE P802.3ba.

      I would like to extend my gratitude to all who participated in this effort. It was truly a team effort by the members of our industry. I would also like to thank Ilango, Mark, Pete, Chris, and the rest of the editorial team for all of their hard word. Finally, I would like to thank Mr. Grow, Mr. Law, Mr. Diab, and Mr. Carlson, for all of their help during the course of this project.

      Regards,

      John D’Ambrosia

      Chair, IEEE P802.3ba Task Force “

  4. 100GB ethernet will give you some serious connection speed.

  5. […] different aspect of data centers, the network cabling.  As you’ll see in the first picture, Ethernet cables can be a nightmare if they aren’t carefully organized.  However, when they are organized, […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: