http://www.40gbe.net/index.htmlHot News!en2012-01-31T15:50:10-05:00 hourly 1 2000-01-01T12:00+00:00 Tue, 31 Jan 2012 15:58:27 -0500DBL2012-01-31T15:50:10-05:00http://www.40gbe.net/index_files/c7feb6bf11dc53351e10c8d438261256-23.html#unique-entry-id-23http://www.40gbe.net/index_files/c7feb6bf11dc53351e10c8d438261256-23.html#unique-entry-id-23
Myricom, the technology leader in extreme-performance 10-Gigabit Ethernet solutions specialized for vertical markets, today announces that Trading Physics, a producer of analytic software and custom market data delivery systems, has adopted Myricom's 10G-PCIE2-8C2-2S dual port SFP+ adapters along with Myricom's DBL 2.0(TM) low-latency software for High Frequency Trading (HFT) applications." rel="external">published yesterday:

Myricom Adds Design Win for High Frequency Trading with Wall Street Software and Infrastructure Specialist, Trading Physics

ARCADIA, Calif., Jan 30, 2012 (BUSINESS WIRE) -- --Achieves 4.0 microsecond latency normalizing US equity depth feeds
Myricom, the technology leader in extreme-performance 10-Gigabit Ethernet solutions specialized for vertical markets, today announces that Trading Physics, a producer of analytic software and custom market data delivery systems, has adopted Myricom's 10G-PCIE2-8C2-2S dual port SFP+ adapters along with Myricom's DBL 2.0(TM) low-latency software for High Frequency Trading (HFT) applications.

"The depth feeds contain all orders placed for equities listed on five exchanges: NYSE, NYSE/ARCA, NASDAQ, BATS, and Direct Edge -- an avalanche of data, and processing this order book data with the lowest possible latency is critical to the success of high frequency trading strategies. A customer asked us to achieve sub-five microsecond latency from the top of the switch to client server delivery. After extensive research we found Myricom's extreme performance low-latency solution to be very attractive. Unlike products from other vendors, Myricom's solution did not require the use of custom server hardware to process the feeds," said Artem Novikov, Founder of Trading Physics. "We ordered network adapters, set up a benchmarking test bed using Myricom native DBL libraries, and rapidly surpassed the target benchmark with production data. The customer is delighted with the results."...]]>Time2012-01-17T10:38:16-05:00http://www.40gbe.net/index_files/b67f0bd59016795c5a12061562252c7f-22.html#unique-entry-id-22http://www.40gbe.net/index_files/b67f0bd59016795c5a12061562252c7f-22.html#unique-entry-id-22
“With Myricom’s new SYNC network adapters, we can timestamp all incoming packets with 1us precision, allowing us to more accurately measure and monitor the performance of our high frequency trading platform,” said Oleg Chernyakhovskiy, CTO, Lynx Capital Partners, a leading provider of high volume, low latency automated trading applications.

“Myricom’s new SYNC adapter makes precise timestamping accessible for the broad 10-Gigabit Ethernet market,” said Dr. Nan Boden, CEO of Myricom. “Our solution, by using a CDMA or GPS timecode source, gives our customers the capability to accurately timestamp packets without having to sacrifice 10GbE throughput as is the case with competing solutions.”

*Note most of this text came from our press release.]]>DBLTime2012-01-17T10:37:11-05:00http://www.40gbe.net/index_files/f7384e973d1c53c52f3f296da46177b8-20.html#unique-entry-id-20http://www.40gbe.net/index_files/f7384e973d1c53c52f3f296da46177b8-20.html#unique-entry-id-20
DBL 2.0.2 provides low-latency UDP and TCP communication with transparent socket acceleration for high frequency trading (HFT) environments where cutting-edge latency performance and rapid time to deployment are required for business success. DBL 2.0.2 is supported on Linux and Windows platforms.

New features in DBL 2.0.2 include:

- Support for Myricom’s Sub-$1K Timecode-based SYNC
  10-Gigabit Ethernet Adapter.

- High accuracy packet timestamping for Cybersecurity
  and HFT applications.

- Support for large numbers of multicast groups (>32)
  in acceleration mode.

- Support for 32-bit libraries in transparent acceleration. 

*Note most of this text came from our press release.]]>Markets2011-09-01T10:33:48-04:00http://www.40gbe.net/index_files/7e65407831d6f95807c94a321727f818-19.html#unique-entry-id-19http://www.40gbe.net/index_files/7e65407831d6f95807c94a321727f818-19.html#unique-entry-id-19
One vendors uses Myri-10G network adapters to combine the performance benefits of iSCSI SAN protocol with NAS functionality and ease of use, all working over a single network infrastructure. The end result is a faster deployment that is easier to manage, and less expensive than an iSCSI SAN solution, and with even faster data access.  For color grading and special effects applications working with uncompressed video at 2K, 3K or even 4K resolution, client machines can achieve 800MB/s over a 10GbE connection. This throughput is faster than traditional Fiber Channel.

Another provider using MCS (Multiple Connections per Session), regularly sustains data rates of 1.2GB/s (10bit 35mm) for 4K video.]]>VideoPump2011-09-01T10:32:14-04:00http://www.40gbe.net/index_files/8dc9cc857b62b3887add10aadba200b0-18.html#unique-entry-id-18http://www.40gbe.net/index_files/8dc9cc857b62b3887add10aadba200b0-18.html#unique-entry-id-18DBL2011-07-01T10:00:29-04:00http://www.40gbe.net/index_files/36204ea10242a7cb8dda89cb94ecd2dd-17.html#unique-entry-id-17http://www.40gbe.net/index_files/36204ea10242a7cb8dda89cb94ecd2dd-17.html#unique-entry-id-17PHYs2011-07-01T09:59:34-04:00http://www.40gbe.net/index_files/8982c109c0a498ba54656437addf6d17-16.html#unique-entry-id-16http://www.40gbe.net/index_files/8982c109c0a498ba54656437addf6d17-16.html#unique-entry-id-16Sniffer2011-06-01T09:58:45-04:00http://www.40gbe.net/index_files/3c856463e9d01e0fd564205cf4312adb-15.html#unique-entry-id-15http://www.40gbe.net/index_files/3c856463e9d01e0fd564205cf4312adb-15.html#unique-entry-id-15DBL2011-06-01T09:57:39-04:00http://www.40gbe.net/index_files/931b45c840bf2c92456d2360f1da1451-14.html#unique-entry-id-14http://www.40gbe.net/index_files/931b45c840bf2c92456d2360f1da1451-14.html#unique-entry-id-14Cyber Warfare2011-05-30T08:23:03-04:00http://www.40gbe.net/index_files/d00f292985f18f145be8081eb2608563-13.html#unique-entry-id-13http://www.40gbe.net/index_files/d00f292985f18f145be8081eb2608563-13.html#unique-entry-id-13
Unlike the first two World Wars, or the final fictitious one is in the Matrix. WW III is being fought today exclusively in cyber space.  It is a silent war, rarely reported in mainstream media, primarily because there’s almost never a human casualty*, and most readers wouldn’t understand it.  Often the combatants have no idea who they're fighting. If we look at one of the early major battles, Stuxnet, no one has officially claimed to have launched the attack, but the target was laser focused on Iran's nuclear weapons program. The campaign was successful, and it delayed Iran's weapons program by several years.

Now I'm sure your wondering, "Why is this being covered in a blog about 10GbE?"  Simple, all battles require supple lines. Today the supply lines for cyber warfare are moving from GbE to 10GbE.  Sure a lone hacker can use a Low Orbit Ion Cannon (LOIC) to take down an unsophisticated opponent using little more than a laptop, and a decent Internet connection. Frankly this is synonymous in the real world to someone with a machine gun trying to shut down a store.

On the other hand governments, and corporations have substantially more resources.  As we saw with Stuxnet.  For those not familiar with Stuxnet, this was a very carefully engineered offensive cyber weapon delivered in virus form.  It had several different parallel attack vectors to ensure that it reached all the intended targets, which reports suggest that it did.  It contained a precise triggering system, and a lethal warhead.   This battle would not have been possible with only a handful of intelligent people.  It took a collaborative effort by several governments, at least one corporation, and several unique pieces of equipment for testing to ensure the weapon’s success.  This went way beyond your garden variety LOIC class assault, and defined a whole new category of offensive cyber weaponry.  While the LOIC is a machine gun obtainable by nearly anyone, Stuxnet by comparison is a state of the art cruise missile.

Now back to 10GbE.  Markets like HFT and HPC are breaking new ground in low latency, high packet rate & wire-rate bandwidth.  They are utilizing advanced 10GbE network adapters and switches, along with sophisticated algorithmic routines.  Today HFT’s are exploring lossless wire-rate 10GbE capture to do real time analysis and simulation.  They capture real trading data in buffers or spin up synthetic ones to emulate the market.  They then inject these buffers into an HFT trading engine to see how it responds.  Is this really any different then simulating 100s or 1,000’s of web surfers?  This technology is crossing over into cyber warfare.

All of the above was written over the holiday weekend.  This morning the Wall Street Journal reported that the US Government has announced that The Pentagon has adopted a new strategy that will classify major cyber attacks as acts of war, paving the way for possible real world military retaliation.  "If you shut down our power grid, maybe we will put a missile down one of your smokestacks," said a military official as quoted in the WSJ.  So if you don’t believe the ongoing war in cyber space is real, then just consider it a prelude to it.

* John P. Wheeler III may very well be the first person slain in this cyber war.  He is a former Pentagon official, and was a consultant for Mitre prior to his murder.  John was an outspoken proponent of Cyber defense.  In January 2011 he was found dead in a Maryland landfill, the investigation into is death is currently stalled.]]>PHYs2011-05-18T08:10:16-04:00http://www.40gbe.net/index_files/42bebf3dcb4959c752244ce6a4ce6b7d-12.html#unique-entry-id-12http://www.40gbe.net/index_files/42bebf3dcb4959c752244ce6a4ce6b7d-12.html#unique-entry-id-12
For cable runs of seven meters or less you should always use Direct Attach (DA otherwise known as Twinax) cable if possible as it could easily save enough to basically connect a second server for free! Here are some actual numbers from earlier today.

First some basic end user costs assuming a five meter run, note these are rounded a little bit to keep the math simple:

10GbE Network adapters, roughly $400/port
10GbE Switches, roughly $500/port
10GbE SR SFP+ Optics from switch vendor $800/port
10GbE SR SFP+ Optics from NIC vendor $200/port
10GbE SR Optical 5M cable $80/ea
10GbE Direct Attach 5M cable roughly $140/ea

Now lets build a solution between the server and the switch using optics:

10GbE Network adapter $400
10GbE SR Optic from NIC vendor $200
10GbE SR Optical 5M cable $80
10GbE SR Optic from  Switch vendor $800
10GbE Switch port $500
Total $1,980 to connect a single server

Direct Attach (Twinax) Option:

10GbE Network adapter $400
10GbE Direct Attach 5M cable roughly $140/ea
10GbE Switch port $500
Total $1,040 to connect a single server

Let’s look more closely at the market dynamics going on here.  First only a handful of companies make 80% of the 10GbE Short Range (SR) optics that everyone uses today.  These companies are typically: JDSU, Finisar, Agilent, etc...  None of the switch companies or NIC companies make their own optics, we all source them from several of the above companies, and a few others, all of whom rebrand them for us and burn our company name and part number into what is essentially flash memory within the optic.

Here’s where it gets interesting.  Myricom, the company I work for, sells it’s SR SFP+ optics online via CDW’s website for $185.  Here are some of the more expensive SR SFP+ optics listed on CDW’s site:

HP Procurve: $1,498
Avaya: $1,350
Enterasys: $1,210
Cisco: $1,100
Juniper: $1,082
Brocade: $1,022
QLogic: $930
IBM: $920

Now remember under the covers we’re all sourcing these optics from the same competitive pool, so why the price spread?

First remember that we each buy our optics with our manufacturer name and part numbers already burned into them by the optics makers mentioned above.  Now here’s where it gets interesting the switch makers during switch initialization query the optic and if it does not return a valid company name and part number then it locks the optic out and reports the port as offline. 

A Cisco switch requires a Cisco optic.  If you were to use a Myricom optic it would see that the optic was made by “Myricom” with a part number “10G-SFP-SR” and it would lock that port out because it has an incompatible optic.  Never mind that a valid Cisco optic and the “failed” Myricom optics may very well have been made by JDSU on the same assembly line, perhaps even on the same day.

Network adapter vendors, like Myricom, are optic agnostic. You can shove in an Arista, Cisco, HP, or Gnodal, we won’t care.  We provide optics to offer a complete solution for our customers.  Finally, we are not “in the optic business” so we pick them up, mark them up fairly, then offer them for sale.  I can assure you we’re not buying them at the same discount that a Cisco or Juniper might be getting, yet our price is clearly so much more reasonable. 

Now here’s the secret I promised.  Most switch vendors have a patch for the switch operating system so that it will ignore the optic check and allow you to use anybody’s optics.  If you have the buying power and the cojones, then insist  that they provide the patch as a condition of buying their switch.  It will save you big time.  You can then take that savings, and buy a few more Myricom 10GbE adapters. ]]>10GbE2011-05-15T08:05:46-04:00http://www.40gbe.net/index_files/2940a9edff16998611a3d0a078f51e14-11.html#unique-entry-id-11http://www.40gbe.net/index_files/2940a9edff16998611a3d0a078f51e14-11.html#unique-entry-id-11The Register even used 10GbE.net as an expert reference by mentioning & linking to us to validate a storyline. This resulted in 10GbE.net rocketing into the top four sites when searching for the string “10GbE”. 10GbE.net, and it’s sister 40gbe.net, were started in December 2007 as a one man stealth marketing effort to help get 10GbE off the ground more quickly.  Let’s face it my job is to sell 10GbE network adapters, so I created the site to drive traffic, and improve sales.  Both of which it did.

Initially 10GbE.net hosted several pages listing all the currently available network adapters by type, interface, performance & price.  A sort of bleeding edge Consumer Reports for both adapters, and later switches.  As the traffic, and attention grew it became more uncomfortable to operate in stealth.  Also it had become a huge sink for my spare time.  After speaking to one of my mentors it was decided to pull the plug on both sites.

Today the market is different, and more perspective is needed.  I’ve recast this site, and cross linked my 10GbE.net domain to 40GbE.net to form this new Extreme Performance Networking blog.  From this vantage point I can share more information, and my unique perspective while remaining in full view.]]>10GbE2009-03-17T07:52:19-04:00http://www.40gbe.net/index_files/e328398bc723e8120c5bd1121696441c-10.html#unique-entry-id-10http://www.40gbe.net/index_files/e328398bc723e8120c5bd1121696441c-10.html#unique-entry-id-10
Most NICs talk XAUI to a PHY driver chip that then controls the media. XAUI has four pairs of receive/transmit lanes that each operating at 3.125Gbps and utilize 8/10b encoding. So after decoding you have 10.000Gbps of actual usable media bandwidth, hence 10G. Ethernet adds it's own shipping and handling fee though and so people who measure bandwidth using the operating system (OS) on their server will never see the full capacity of the pipe due to this overhead.

Ethernet is then traveling over XAUI and it requires an interframe gap (IFG) spacing of 96 bits, more precisely the time it would take to transmit 96 bits (actually 8 octets) which on 10GbE is 9.6ns. It turns out though that the 10GbE spec actually redefined the gap to 40 bits or 4.0ns. So between every packet there is 4ns of air.

Ethernet also requires a seven byte preamble and a one byte start of frame delimiter, that's 64 more bits or another 6.4ns of air. So between any two actual "packets" on a 10GbE wire this is a total 10.4ns of dead air. Well how does that impact the actual OS measurable bandwidth?

Let's look at the detailed best case, a jumbo frame. First there is the 7 octet preamble, 1 octet start-of-frame delimiter, then what would be called the "Jumbo Frame" which is 9,018 octets (it's really a 9000 octet payload with a 14 octet header and a 4 octet CRC at the end) then the IGF or 5 octets in the case of 10GbE. When people typically measure NIC bandwidth they only measure the "packet" or 9018 byte part, and often don't know about those other 13 required bytes. So a 10GbE NIC running jumbo frames can never achieve more than (9018/9031)*10Gbps or 9.9856Gbps and a good wire-rate NIC will demonstrate this.

Now in the worst case we have the dreaded "64 byte" frame which has the same 13 bytes of stuff between "frames" so the that calculation is (64/77)*10Gbps or 8.3117Gbps. So NEVER expect your 10GbE NIC to deliver 10Gbps using 64 byte frames (which really only contain 46 bytes of actual data) because Ethernet will shackle it to 8.3117Gbps. If your application just measures actual payload then it's 46/77*10Gbps or 5.97Gbps.

So if you drive a 9,000 byte tractor trailer you can speed along at 9.9856Gbps, but if you're tooling along on your little 64 byte rice rocket you'll never get above 5.97Gbps.]]>10GbE2009-02-04T07:41:04-05:00http://www.40gbe.net/index_files/f1f82da3068ab75737353aaa82119b42-9.html#unique-entry-id-9http://www.40gbe.net/index_files/f1f82da3068ab75737353aaa82119b42-9.html#unique-entry-id-9
With 10GbE we've seen three main approaches for building dual-port NICs:

Active/Active: this is what most people expect, a single OS interface with a driver that sprays traffic fairly evenly across both network ports and if one port fails the other picks up the slack until it can handle no more:

• Chelsio's N320E for $790 is an example of this type of card.
• Intel's AF DA card for $799 appears to be another example of this class of card.

Dual-NIC: two OS interfaces are presented to the OS and both interfaces run independently. This typically affords the best performance and the most flexibility:

• Myricom's 10G-PCIE2-8B2-2S+E for $995 appears to be the only example of this approach. Myricom utilizes two unique 10GbE controllers on the same PCI Express Gen2 NIC and a PCI Express bridge chip to break the slot into two unique NIC devices.

Active/Passive or Active/Fail-over: a single OS interface with a driver that monitors connectivity on the active port and if the connection fails the driver migrates traffic rapidly over to the second port:

• Myricom's 10G-PCIE-8B-2S+E for $795 is an example of this type of card. The fail over time is under 10 microseconds.
• Chelsio's B320E Bypass adapter for $3,483 is similar but it can detect an OS/BIOS/System failure and make a hard switch over to the second port.

Do the above categories cover it, or do we need more lingo? When looking for a dual-port NIC, what features do you require, and what do you expect? Please let us know.

P.S. As I brought this page back online I left off the links as most no longer apply, but from a historical perspective it is interesting to see how things have progressed.]]>10GbE2009-01-10T07:33:44-05:00http://www.40gbe.net/index_files/927f2bd220f16aa237e143b2c1576129-8.html#unique-entry-id-8http://www.40gbe.net/index_files/927f2bd220f16aa237e143b2c1576129-8.html#unique-entry-id-8
Officially the first documented victim is NetEffect, the leader in iWarp (Infiniband for 10GbE) NICs. NetEffect rose from the ashes of a failed Infiniband company, Banderacom, earlier this decade to apply their silicon development skills and Infiniband algorithms to the more stable Ethernet market as a new feature called iWarp. NetEffect in-fact led the iWarp charge, it was the self proclaimed leader in low-latency iWarp 10GbE NICs. In August NetEffect filed for reorganization in US Bankruptcy court. With the failure of NetEffect the market has cast its vote and drove a steak through the heart of iWarp, hopefully terminating this feature.

Rumors have been swirling around Teak Technologies, a maker of 10GbE NICs and a switch, for some time. It appears that Teak has not weathered the storm and has since faded away, their domain name is no longer resolving to an IP address. The domain was never transferred from the founder, and the founder announced this spring on Linkedin that he had moved on some time ago. Is it conclusive evidence, no, but would you buy technology from a tech company whose URL won't resolve to a server?

It is a tough economic climate for startup NIC companies, particularly those in the bottom 20% as they have likely never had a quarter in the black. Now is a challenging time to be out there seeking another round of capital from ones VCs. Several have been without an injection of new funding for over two years and lack the sales volume required to sustain their own existence much beyond year end. As such we've directly questioned one firm to see if they are alive, and another that is widely rumored in the industry to be in trouble, but their marketing departments are still bailing. ]]>Infiniband2008-11-15T07:18:28-05:00http://www.40gbe.net/index_files/fc188c7e6d7f991b420f72dbb4bac2d4-7.html#unique-entry-id-7http://www.40gbe.net/index_files/fc188c7e6d7f991b420f72dbb4bac2d4-7.html#unique-entry-id-7
Once praised as the next generation networking technology, having conquered High Performance Computing, it continued it's battle for world networking domination by attacking storage and now the data center. It promises you 20Gbps, hinted that it would soon offer 40Gbps and shared with you it's plans for 160Gbps! It claimed full bi-section, the ability to use all the network capacity available, and low latency (the time it takes to actually move a packet of data around). It's democratic, the software stack was developed by an "open" committee of great technological leaders so it MUST be good for us. Everyone from HP to SGI has sung it's praises whenever they've come by to peddle the latest in server technology. A corpse wrapped in rags, a centuries old immortal Dragon Emperor or a black cable bandit, they all can be eradicated.

We will tear this black cable bandit down to size one claim at a time. First they assert that it's 20Gbps, how about 12Gbps on it's best day with all the electrons flowing in the same direction. Infiniband employs what is know as 8b/10b encoding to put the bits on the wire. For every 10 signal bits there are 8 useful data bits. Ethernet uses the same method, the difference is that Ethernet for the past 30 years has advertised the actual data rate, the 8, while Infiniband promotes the 25% larger and useless signal rate, the 10. Using Infiniband math Ethernet would then be 12.5Gbps instead of the 10Gbps it actually is. So using Ethernet math Infiniband's Double Data Rate (DDR) is actually only 16Gbps and not the 20Gbps they claim. But wait there's more! I said earlier that you will only get 12Gbps under ideal conditions, where did the other 4Gbps go? Today most servers use PCIe 1.1 8-lane I/O slots. Ideally these are 16Gbps slots, once you add in PCIe overhead though you only get about 12Gbps on the best of systems. So with a straight face they sell you 20Gbps knowing in their heart you'll never get more than 12Gbps.

Full bi-section, the ability for a network of servers to use all the network fabric available. Infiniband claims that using their architecture and switches you can leverage the ENTIRE network fabric under the right circumstances. On slides this might be true, but in the real world it's impossible. Infiniband is statically routed, meaning that packets from server A to server X have only one fixed predetermined path they can travel. One of the nations largest labs proved that on an 1,152 server Infiniband network that static routing was only 21% efficient and delivered on average 263MB/sec (2.1Gbps of the theoretical 10Gbps possible). So when they tell you full bisection, ask them why LLNL only saw 21%? In an IEEE paper presented last week it was proven that statically routed system can not achieve greater than 38% efficiency. Now some of the really savvy Mummy supporters will say that the latest incantation of Infiniband has adaptive routing, they do this by using yet another shell game, they redefine the term adaptive routing to mean more than one static route. Real adaptive routing and using a pair of static routes are vastly different things. Real Adaptive routing can deliver 77% efficiency on 512 nodes and nearly 100% efficiency on clusters smaller than 512 nodes. If you want full bisection for more than a 16 node cluster talk with Myricom or Quadrics, they do real adaptive routing.

Latency is the time it takes to move a packet from one application on network server to another application on a different server on the same network. Infiniband has always positioned itself as being low latency. Typically Infiniband advertises a latency of roughly three microseconds between two nics, using zero-byte packets. Well in the past year 10GbE NICs and switches have come onto the market that can achieve similar performance. If we look at Arista's switches they measure latency in a few hundred nanoseconds while Cisco's latest 10GbE switches are sub four microseconds, compared to the prior generations that were measured in the 10's of microseconds or more. Now when the Infiniband crowd crows about using low latency switching ask them about Arista or BLADE Network technologies 10GbE switches.

Infiniband claims 20Gbps and delivers less than 12Gbps. Infiniband claims full bisection yet beyond a small network they can't exceed 38% efficiency. Infiniband claims low latency and now 10GbE can match it. Where is their value proposition in the data center?]]>PHYs2008-10-15T07:11:52-04:00http://www.40gbe.net/index_files/823c433d4edc7c4c2bd7413d267e311f-6.html#unique-entry-id-6http://www.40gbe.net/index_files/823c433d4edc7c4c2bd7413d267e311f-6.html#unique-entry-id-6

• Flexibility - Cisco and Juniper both selected SFP+ as the PHY for their new line of 10GbE switches. Offering an SFP+ cable with a connector on the end that enables you to use a single SFP+ port for all your connection needs is a stroke of genius. Say you need a short run from one switch to a server, plug a Twinax cable with SFP+ connectors on each end in and your good to go, up to 10 meters. Suppose later you need to move that server another 50 meters away then pop in SR optics on both ends and use fiber. No changes to the servers or switches, just swap in optics.
• Cost - there has been a run up recently in the price for copper, while the cost of Twinax coax cable remains fairly fixed.
• Power - SFP+ is rated at 1W/port, the Twinax solution typically draws 1/4W. CX4 is similar but compared to 10GBase-T at 10W (current generation) or even 2W for the next generation (under 30M) this is a huge power savings.
• Latency over 10GBase-T - Current 10GBase-T uses a DSP at each end to separate the signal from the noise. This DSP adds roughly two micro seconds on each end of the connection, compared to under 200ns for the Twinax conversion.

We are closely watching how Twinax plays out over the next few months, and we'll let you know what we learn.

UPDATE JANUARY 2012 - Twinax, otherwise known as Direct Attach or DA has won. In the past year we've seen a significant drop off in requests for CX4 NICs since their height in 2008.]]>10GbE2008-09-20T07:06:26-04:00http://www.40gbe.net/index_files/a6bccf4a487a2105c27c5194cc572091-5.html#unique-entry-id-5http://www.40gbe.net/index_files/a6bccf4a487a2105c27c5194cc572091-5.html#unique-entry-id-5
Few vendors post actual price and performance data on the web, let alone the methodology they used to arrive at those performance numbers. If only there were an independent third party that actually ran Netperf, Iperf, ntttcps, ntttcpr and other tools on all the available 10GbE NICs using the same test systems then posted the results for everyone to see. Some companies would never recover. For legal reasons the vendors won't, and in most cases do not want to, do it because the results would only help one or two companies and likely not theirs. Today all most consumers have to go on is the cost of the adapter, wouldn't it be great if you knew the cost/Mbps of the adapter prior to buying it so you could easily compare between adapters. Some would argue that features like iWARP and TOE should be factored in, but today they are just marketing fluff and rarely delivery any significant end user value.

So how do you determine which NIC will perform the best and deliver the most value for your company, do a bake-off! If you can make the time and the project is big enough the cost to conduct the back-off should easily be offset by the savings, education, and performance gains you reap over time. Also a well constructed and executed bake-off will demonstrate not only to you, but your management, that you're an effective individual and a good steward of the companies resources.

Finally, share the full set of results with the vendors that participated, some will moan and groan, while others will kindly thank you for the opportunity to compete and move on. If the race was close their reactions at this point might be your deciding factor. So pull on your oven mitts and start baking...]]>10GbE2008-08-13T18:58:47-04:00http://www.40gbe.net/index_files/fac3fcd606a9cd852877d691f8d15696-4.html#unique-entry-id-4http://www.40gbe.net/index_files/fac3fcd606a9cd852877d691f8d15696-4.html#unique-entry-id-4
One time I assumed responsibility for improving the performance of an MS SQL server that was vital to our business. The primary job this server ran took 75 minutes and it was scheduled to run, how many of you see this coming, every hour! This server was tracking and reporting on $10's of millions in new business every month.At first glance I noticed several back to back to back bottle necks. The system was memory starved, the drives were in a near constant state of thrashing and all SQL I/O from the system went through a $10/NIC card. Although the NIC functioned it was forcing the switch to drop far too many packets. At lunch that day we picked up a newer server class NIC card for $40 and immediately recorded a substantial performance improvement. The job would finish in just under the 60 minutes allowed. We could have spent the next week chasing performance curves, instead we installed a new server, a dual processor single core box and the job now completed in well under a minute. So a $40 NIC improved performance by 20% while replacing the whole server for roughly $5,000 improved performance by 98%. Clearly the NIC delivered the biggest bang for the buck, but it just brought the network performance curve in-line with that of the CPU, memory & disk.

How many dual-socket quad-core servers were installed today, August 13th 2008, with GbE? These servers have 4X the horse power of my $5,000 server from 2002, but they both share the same GbE. Furthermore, today we use VMWare and Xen to pack several logical servers into a single physical server in an effort to more efficiently utilize our hardware resources. We don't hesitate to add more memory or disk, but adding a 10GbE board requires substantially more effort and planning.

When making the jump from GbE to 10GbE one needs to not only select a NIC, but the media (CX4 or fiber) and a new switch infrastructure. High performance NICs run $700-$2,000/each. depending on the media and vendor. If you go fiber the optics run $500-3,000/each and you need one on each end of the cable. Finally there's the switch. Stack-able layer-2 switches run in the $400-$1,200/port range while enterprise layer-3 switches often run several thousand dollars/port.

If your server is I/O bound a good 10GbE NIC and switch can enable 5-10X the output of the "free" GbE port that comes with your server. Suppose you purchase a new server for $5,000, then you add a high performance 10GbE CX4 copper NIC and use a low cost layer two switch so the upgrade to 10GbE costs roughly $1,200 for this server. You need to only measure a 25% gain in overall performance for you to realize a positive return on your investment! There are a new breed of hybrid switches that now offer 24 GbE ports and four 10GbE ports so one can easily make the shift from GbE for servers to 10GbE. Consider giving 10GbE a try.]]>Site2012-01-26T08:59:58-05:00http://www.40gbe.net/index_files/afe2f0570edc042ea9763bfba0f94a15-3.html#unique-entry-id-3http://www.40gbe.net/index_files/afe2f0570edc042ea9763bfba0f94a15-3.html#unique-entry-id-3TOEs2008-06-13T20:21:51-04:00http://www.40gbe.net/index_files/573e490edb18f5299c5f282ef5635bf1-2.html#unique-entry-id-2http://www.40gbe.net/index_files/573e490edb18f5299c5f282ef5635bf1-2.html#unique-entry-id-2
In the spring of 2006 a good non-TOE 10GbE NIC consumed 40% of the host CPU in a dual-socket dual-core server and provided >6Gbps of performance, while a similar TOE did the same job using only 10% of the host CPU. So with a 30% savings in host CPU there was some value in using a TOE. With two years of improvements in silicon, stateless off-loads, and servers moving to dual-socket quad cores we now have 10GbE NICs capable of near-wire rate (>9.5Gbps) that consume only 10% of the host CPU. Similarly, TOE NICs in the same environment consume roughly 5% of the host CPU.

By most estimates servers are typically running at 20% CPU utilization, as a result of application load. So will a 5% savings in host CPU be noticed, let alone worth the added purchase price of a TOE? No. Add to that the Linux Foundation's 14-point argument against using TOES, written by the Linux Kernel developers themselves, and one would wonder why people still consider TOEs in style.

Here are the 14 reasons cited by the Linux Foundation on their

1. Security updates
2. Point-in-time solution
3. Different network behavior
4. Performance
5. Hardware-specific limits
6. Resource-based denial-of-service attacks
7. RFC compliance
8. Linux features
9. Requires vendor-specific tools
10. Poor user support
11. Short term kernel maintenance
12. Long term user support
13. Long term kernel maintenance
14. Eliminates global system view

If you are seriously interested in buying a TOE you should read their TOE page.]]>PHYs2008-04-12T20:14:48-04:00http://www.40gbe.net/index_files/399af375714c90882f80a2d69f5993a3-1.html#unique-entry-id-1http://www.40gbe.net/index_files/399af375714c90882f80a2d69f5993a3-1.html#unique-entry-id-1
Single port copper NICs run in the $700-$1,000 range while similar fiber NICs are $800-$1,200. The expectation is that SFP+ fiber modules will be roughly 25% less expensive than XFPs so this will make fiber more affordable in the second half of 2008 as SFP+ gains traction. The real knee in the 10GbE adoption curve though will occur when the next generation of 10GBase-T products hit the market in early 2009. The current generation of 10GBase-T silicon requires far to much power to make it practical. This second generation of 10GBase-T will allow people to use cables and connectors they are familiar with, ex.Cat6E and RJ45, to attach servers and switches within 100M without the expensive CX4 cables or the optics required today for fiber.

Finally most of the 10GbE NIC vendors are on their second or third generation silicon. By early 2009 most will have trimmed and tuned things to the point that they will have, or soon support LAN on Motherboard solutions. When this happens we will see high end servers with 10Gbase-T support built in and 10G will then truly begin to replace GbE in the enterprise. We expect that this will likely begin to become common as we enter 2009.]]>PHYs2008-03-08T21:00:30-05:00http://www.40gbe.net/index_files/5ff8334606fd5be430f4f9058f51ec21-0.html#unique-entry-id-0http://www.40gbe.net/index_files/5ff8334606fd5be430f4f9058f51ec21-0.html#unique-entry-id-0
In the 10GbE world all the NIC vendors separate their 10GbE chip from the physical (PHY) interface chip so they can be more responsive and flexible in creating NIC products and easily support several PHYs with a single 10GbE NIC chip. Today only three companies make 10GBase-T PHY chips: SolarFlare, Teranetics and Aquantia. Teranetics is having the most success signing Chelsio, Tehuti and Mellanox while Solarflare picked up SMC. What most avoid telling you is how much power these 10GBase-T PHY chips require, 8-12W. The vast majority of this power is used for only one purpose, separating the signal from the noise, the needle from the haystack.

What does this mean to you? Below is a simple example with 50 servers, focused only on the PHY power and the total power utilization for each of the three currently available media formats. Below is the power budget for the PHY on each end (NIC or Switch), total to support a server (both NIC and Switch PHY power) and the total for a 50 server project:

• 10GBase-CX4 PHY 0.5W/end, 1W/server, 50W for the project
• 10GBase-R XFP PHY 3W/end, 6W/server, 300W for the project
• 10GBase-T PHY 10W/end, 20W/server, 1000W for the project

This is the power needed to support just the 10GBase-T cabling and it is consuming enough energy to power two of the servers in your project! This is a cost you will carry for the life of the project and we all know conditioned data center power and cooling is not cheap.

When GbE first came out the initial round of PHY chips were also power hungry, of course that is no longer the case. With 10GbE the physics are significantly more challenging and it may take another year or two before 10GBase-T solutions have power consumption similar to CX4. If you are doing a project today that can benefit from 10GbE use CX4 if possible, or fiber if you need more than 15M. For fiber consider using SFP+ or XFPs as they are the most current optics, are the least expensive and consume far less power then XENPACK or X2.

Note on April 14th Solarflare announced their new PHY silicon, 10Xpress SFT9001, which consumes between 2.2 and 6W depending on cable length. This brings 10GBase-T into parity with fiber. This PHY chip will be available in sample lots to 10GbE OEMs in May. Even more recently on April 21, Aquantia announce a 10GBase-T PHY chip that is sampling in May and which claims to bring the power down to 5.5W for Cat6A cable up to 100M long. The delay from samples to OEM NIC vendors and completed NIC samples for customers is often in the neighborhood of 3-6 months. So in our opinion 10GBase-T NICs with a reasonable power envelope, 10-15W for the entire NIC, should be something that is available for consideration in the fall of 2008, just in time for those with year end budgets.

For more information and another perspective consider checking out what the Linley Group has to say on 10GBase-T.]]>