In February 2006 I helped coordinated Myricom's first shipment of 10GbE adapters to our largest OEM customer, so over the past five years I've picked up a bit about the industry.  Today most people use  fiber for 10GbE.  Recent advances in technology will usher in the emergence of 10GbE over standard Cat5 cable, the most popular network cable in the world, for runs up to 40 meters.  If you spring for the more pricey Cat6 cable you'll be able to reach 100 meters.  It will be at this point that 10GbE will be ready for mass adoption.  So why has it taken five years?  It's a combination of both physics and computing power.  Cat5 & 6 cable is four pairs of copper wire wrapped around each other.  At 10GbE speeds this is four 3.125Ghz antennas intertwined.  The ratio of signal to noise is extremely lopsided so at each end of the wire a sophisticated digital signal processor (DSP) is employed to locate the signal in the cacophony of noise.  Until recently these DSPs were very power hungry, not anymore.  This industry is approaching an inflection point, stay tuned...

Today for the umpteenth time I had to explain to someone that if you go optical to connect your server to your switch with 10GbE it could easily cost you twice as much.  There is a secret at the end of this entry that MIGHT allow you to save some big time cash, if you have enough muscle, but you have to read to the end of this entry.

For cable runs of seven meters or less you should always use Direct Attach (DA otherwise known as Twinax) cable if possible as it could easily save enough to basically connect a second server for free! Here are some actual numbers from earlier today.

First some basic end user costs assuming a five meter run, note these are rounded a little bit to keep the math simple:

10GbE Network adapters, roughly $400/port
10GbE Switches, roughly $500/port
10GbE SR SFP+ Optics from switch vendor $800/port
10GbE SR SFP+ Optics from NIC vendor $200/port
10GbE SR Optical 5M cable $80/ea
10GbE Direct Attach 5M cable roughly $140/ea

Now lets build a solution between the server and the switch using optics:

10GbE Network adapter $400
10GbE SR Optic from NIC vendor $200
10GbE SR Optical 5M cable $80
10GbE SR Optic from  Switch vendor $800
10GbE Switch port $500
Total $1,980 to connect a single server

Direct Attach (Twinax) Option:

10GbE Network adapter $400
10GbE Direct Attach 5M cable roughly $140/ea
10GbE Switch port $500
Total $1,040 to connect a single server

Let’s look more closely at the market dynamics going on here.  First only a handful of companies make 80% of the 10GbE Short Range (SR) optics that everyone uses today.  These companies are typically: JDSU, Finisar, Agilent, etc...  None of the switch companies or NIC companies make their own optics, we all source them from several of the above companies, and a few others, all of whom rebrand them for us and burn our company name and part number into what is essentially flash memory within the optic.

Here’s where it gets interesting.  Myricom, the company I work for, sells it’s SR SFP+ optics online via CDW’s website for $185.  Here are some of the more expensive SR SFP+ optics listed on CDW’s site:

HP Procurve: $1,498
Avaya: $1,350
Enterasys: $1,210
Cisco: $1,100
Juniper: $1,082
Brocade: $1,022
QLogic: $930
IBM: $920

Now remember under the covers we’re all sourcing these optics from the same competitive pool, so why the price spread?

First remember that we each buy our optics with our manufacturer name and part numbers already burned into them by the optics makers mentioned above.  Now here’s where it gets interesting the switch makers during switch initialization query the optic and if it does not return a valid company name and part number then it locks the optic out and reports the port as offline. 

A Cisco switch requires a Cisco optic.  If you were to use a Myricom optic it would see that the optic was made by “Myricom” with a part number “10G-SFP-SR” and it would lock that port out because it has an incompatible optic.  Never mind that a valid Cisco optic and the “failed” Myricom optics may very well have been made by JDSU on the same assembly line, perhaps even on the same day.

Network adapter vendors, like Myricom, are optic agnostic. You can shove in an Arista, Cisco, HP, or Gnodal, we won’t care.  We provide optics to offer a complete solution for our customers.  Finally, we are not “in the optic business” so we pick them up, mark them up fairly, then offer them for sale.  I can assure you we’re not buying them at the same discount that a Cisco or Juniper might be getting, yet our price is clearly so much more reasonable. 

Now here’s the secret I promised.  Most switch vendors have a patch for the switch operating system so that it will ignore the optic check and allow you to use anybody’s optics.  If you have the buying power and the cojones, then insist  that they provide the patch as a condition of buying their switch.  It will save you big time.  You can then take that savings, and buy a few more Myricom 10GbE adapters.

Last week Cisco jumped behind something called Twinax, Why? Three likely reasons:

• Flexibility - Cisco and Juniper both selected SFP+ as the PHY for their new line of 10GbE switches. Offering an SFP+ cable with a connector on the end that enables you to use a single SFP+ port for all your connection needs is a stroke of genius. Say you need a short run from one switch to a server, plug a Twinax cable with SFP+ connectors on each end in and your good to go, up to 10 meters. Suppose later you need to move that server another 50 meters away then pop in SR optics on both ends and use fiber. No changes to the servers or switches, just swap in optics.
• Cost - there has been a run up recently in the price for copper, while the cost of Twinax coax cable remains fairly fixed.
• Power - SFP+ is rated at 1W/port, the Twinax solution typically draws 1/4W. CX4 is similar but compared to 10GBase-T at 10W (current generation) or even 2W for the next generation (under 30M) this is a huge power savings.
• Latency over 10GBase-T - Current 10GBase-T uses a DSP at each end to separate the signal from the noise. This DSP adds roughly two micro seconds on each end of the connection, compared to under 200ns for the Twinax conversion.

We are closely watching how Twinax plays out over the next few months, and we'll let you know what we learn.

UPDATE JANUARY 2012 - Twinax, otherwise known as Direct Attach or DA has won. In the past year we've seen a significant drop off in requests for CX4 NICs since their height in 2008.

Today for short runs under 15 meters there are two common options: CX4 copper and SR (short range) fiber. The difference between them is essentially the cost of the fiber optic module. Today the most common module for 10GbE is the XFP, soon it will be SFP+. There are three sources for SR optics under $700 listed on our optics page. Optics are required on both ends so this makes fiber typically $1,400 more expensive than CX4 copper. Also copper adapters require less support logic and as such are often less expensive.

Single port copper NICs run in the $700-$1,000 range while similar fiber NICs are $800-$1,200. The expectation is that SFP+ fiber modules will be roughly 25% less expensive than XFPs so this will make fiber more affordable in the second half of 2008 as SFP+ gains traction. The real knee in the 10GbE adoption curve though will occur when the next generation of 10GBase-T products hit the market in early 2009. The current generation of 10GBase-T silicon requires far to much power to make it practical. This second generation of 10GBase-T will allow people to use cables and connectors they are familiar with, ex.Cat6E and RJ45, to attach servers and switches within 100M without the expensive CX4 cables or the optics required today for fiber.

Finally most of the 10GbE NIC vendors are on their second or third generation silicon. By early 2009 most will have trimmed and tuned things to the point that they will have, or soon support LAN on Motherboard solutions. When this happens we will see high end servers with 10Gbase-T support built in and 10G will then truly begin to replace GbE in the enterprise. We expect that this will likely begin to become common as we enter 2009.

No. In March and April several companies began marketing 10GBase-T NICs: Chelsio, Neterion, Tehuti, and even Mellanox (the Infiniband company). Only one switch company, SMC, has dipped their toe in 10GBase-T market, why? Power. All of these products are based on first generation 10GBase-T silicon which is very thirsty for power.

In the 10GbE world all the NIC vendors separate their 10GbE chip from the physical (PHY) interface chip so they can be more responsive and flexible in creating NIC products and easily support several PHYs with a single 10GbE NIC chip. Today only three companies make 10GBase-T PHY chips: SolarFlare, Teranetics and Aquantia. Teranetics is having the most success signing Chelsio, Tehuti and Mellanox while Solarflare picked up SMC. What most avoid telling you is how much power these 10GBase-T PHY chips require, 8-12W. The vast majority of this power is used for only one purpose, separating the signal from the noise, the needle from the haystack.

What does this mean to you? Below is a simple example with 50 servers, focused only on the PHY power and the total power utilization for each of the three currently available media formats. Below is the power budget for the PHY on each end (NIC or Switch), total to support a server (both NIC and Switch PHY power) and the total for a 50 server project:

• 10GBase-CX4 PHY 0.5W/end, 1W/server, 50W for the project
• 10GBase-R XFP PHY 3W/end, 6W/server, 300W for the project
• 10GBase-T PHY 10W/end, 20W/server, 1000W for the project

This is the power needed to support just the 10GBase-T cabling and it is consuming enough energy to power two of the servers in your project! This is a cost you will carry for the life of the project and we all know conditioned data center power and cooling is not cheap.

When GbE first came out the initial round of PHY chips were also power hungry, of course that is no longer the case. With 10GbE the physics are significantly more challenging and it may take another year or two before 10GBase-T solutions have power consumption similar to CX4. If you are doing a project today that can benefit from 10GbE use CX4 if possible, or fiber if you need more than 15M. For fiber consider using SFP+ or XFPs as they are the most current optics, are the least expensive and consume far less power then XENPACK or X2.

Note on April 14th Solarflare announced their new PHY silicon, 10Xpress SFT9001, which consumes between 2.2 and 6W depending on cable length. This brings 10GBase-T into parity with fiber. This PHY chip will be available in sample lots to 10GbE OEMs in May. Even more recently on April 21, Aquantia announce a 10GBase-T PHY chip that is sampling in May and which claims to bring the power down to 5.5W for Cat6A cable up to 100M long. The delay from samples to OEM NIC vendors and completed NIC samples for customers is often in the neighborhood of 3-6 months. So in our opinion 10GBase-T NICs with a reasonable power envelope, 10-15W for the entire NIC, should be something that is available for consideration in the fall of 2008, just in time for those with year end budgets.

For more information and another perspective consider checking out what the Linley Group has to say on 10GBase-T.