Data center bridging
Encyclopedia
Data center bridging refers to a set of enhancements to Ethernet
local area networks for use in data center
environments. Specifically, DCB goals are, for selected traffic, to eliminate loss due to queue overflow and to be able to allocate bandwidth on links. Essentially, DCB enables, to some extent, the treatment of different priorities as if they were different pipes. The primary motivation was the sensitivity of Fibre Channel over Ethernet
to frame loss. The higher level goal is to use a single set of Ethernet physical devices or adapters for computers to talk to a Storage Area Network
, Local Area network
and InfiniBand
fabric.
Traditional Ethernet is the primary network protocol in data centers for computer to computer communications. However, Ethernet is designed to be a best-effort network that may drop packets
when the network or devices are busy. In Internet Protocol
networks, transport reliability has traditionally been the responsibility of the transport protocols, such as the Transmission Control Protocol
(TCP), with the trade-off being higher complexity, greater processing overhead and the resulting impact on performance and throughput.
One area of evolution for Ethernet is to add extensions to the existing protocol suite to provide reliability without incurring the penalties of TCP. With the move to 10 Gbit/s and faster transmission rates, there is also a desire for higher granularity in control of bandwidth allocation and to ensure it is used more effectively. Beyond the benefits to traditional application traffic, these enhancements would make Ethernet a more viable transport for storage and server cluster traffic.
To meet these goals new standards are being developed that either extend the existing set of Ethernet protocols or emulate the connectivity offered by Ethernet protocols. They are being developed respectively by two separate standards bodies, the Internet Engineering Task Force
(IETF) and the Institute of Electrical and Electronics Engineers
(IEEE) Data Center Bridging Task Group of the IEEE 802.1
Working Group.
Different terms have been used to market products based on the underlying Data Center Bridging standards:
These new protocols will require new hardware and software in both the network and the server interconnect. These products are being developed by companies such as Avaya, Brocade, Cisco, Dell, EMC, Emulex, HP, Huawei, IBM, and Qlogic.
Ethernet
Ethernet is a family of computer networking technologies for local area networks commercially introduced in 1980. Standardized in IEEE 802.3, Ethernet has largely replaced competing wired LAN technologies....
local area networks for use in data center
Data center
A data center is a facility used to house computer systems and associated components, such as telecommunications and storage systems...
environments. Specifically, DCB goals are, for selected traffic, to eliminate loss due to queue overflow and to be able to allocate bandwidth on links. Essentially, DCB enables, to some extent, the treatment of different priorities as if they were different pipes. The primary motivation was the sensitivity of Fibre Channel over Ethernet
Fibre Channel over Ethernet
Fibre Channel over Ethernet is an encapsulation of Fibre Channel frames over Ethernet networks. This allows Fibre Channel to use 10 Gigabit Ethernet networks while preserving the Fibre Channel protocol...
to frame loss. The higher level goal is to use a single set of Ethernet physical devices or adapters for computers to talk to a Storage Area Network
Storage area network
A storage area network is a dedicated network that provides access to consolidated, block level data storage. SANs are primarily used to make storage devices, such as disk arrays, tape libraries, and optical jukeboxes, accessible to servers so that the devices appear like locally attached devices...
, Local Area network
Local area network
A local area network is a computer network that interconnects computers in a limited area such as a home, school, computer laboratory, or office building...
and InfiniBand
InfiniBand
InfiniBand is a switched fabric communications link used in high-performance computing and enterprise data centers. Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable...
fabric.
Traditional Ethernet is the primary network protocol in data centers for computer to computer communications. However, Ethernet is designed to be a best-effort network that may drop packets
Packet loss
Packet loss occurs when one or more packets of data travelling across a computer network fail to reach their destination. Packet loss is distinguished as one of the three main error types encountered in digital communications; the other two being bit error and spurious packets caused due to noise.-...
when the network or devices are busy. In Internet Protocol
Internet Protocol
The Internet Protocol is the principal communications protocol used for relaying datagrams across an internetwork using the Internet Protocol Suite...
networks, transport reliability has traditionally been the responsibility of the transport protocols, such as the Transmission Control Protocol
Transmission Control Protocol
The Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
(TCP), with the trade-off being higher complexity, greater processing overhead and the resulting impact on performance and throughput.
One area of evolution for Ethernet is to add extensions to the existing protocol suite to provide reliability without incurring the penalties of TCP. With the move to 10 Gbit/s and faster transmission rates, there is also a desire for higher granularity in control of bandwidth allocation and to ensure it is used more effectively. Beyond the benefits to traditional application traffic, these enhancements would make Ethernet a more viable transport for storage and server cluster traffic.
To meet these goals new standards are being developed that either extend the existing set of Ethernet protocols or emulate the connectivity offered by Ethernet protocols. They are being developed respectively by two separate standards bodies, the Internet Engineering Task Force
Internet Engineering Task Force
The Internet Engineering Task Force develops and promotes Internet standards, cooperating closely with the W3C and ISO/IEC standards bodies and dealing in particular with standards of the TCP/IP and Internet protocol suite...
(IETF) and the Institute of Electrical and Electronics Engineers
Institute of Electrical and Electronics Engineers
The Institute of Electrical and Electronics Engineers is a non-profit professional association headquartered in New York City that is dedicated to advancing technological innovation and excellence...
(IEEE) Data Center Bridging Task Group of the IEEE 802.1
IEEE 802.1
IEEE 802.1 is a working group of the IEEE 802 project of the IEEE Standards Association.It is concerned with:* 802 LAN/MAN architecture* internetworking among 802 LANs, MANs and other wide area networks* 802 Link Security* 802 overall network management...
Working Group.
Different terms have been used to market products based on the underlying Data Center Bridging standards:
- Data Center Ethernet (DCE) was a term originally coined and trademarked by Cisco SystemsCisco SystemsCisco Systems, Inc. is an American multinational corporation headquartered in San Jose, California, United States, that designs and sells consumer electronics, networking, voice, and communications technology and services. Cisco has more than 70,000 employees and annual revenue of US$...
. DCE referred to Ethernet enhancements for the Data Center Bridging standards, and also including a Layer 2 Multipathing implementation based on the IETF's Transparent Interconnection of Lots of LinksTRILL (computing)TRILL is a IETF Standard implemented by devices called RBridges or Routing Bridges. TRILL combines the advantages of bridges and routers and is the application of link state routing to the VLAN-aware customer-bridging problem. RBridges are compatible with and can incrementally replace previous...
(TRILL) standard.
- Convergence Enhanced Ethernet or Converged Enhanced Ethernet (CEE) was defined from 2008 through January 2009 by group of including BroadcomBroadcomBroadcom Corporation is a fabless semiconductor company in the wireless and broadband communication business. The company is headquartered in Irvine, California, USA. Broadcom was founded by a professor-student pair Henry Samueli and Henry T. Nicholas III from the University of California, Los...
, Brocade Communications SystemsBrocade Communications SystemsBrocade Communications Systems, Inc. , based in Silicon Valley , is a vendor of storage area network hardware and software. The company also designs, manufactures, and sells networking products and management applications for local, metro, and wide area networks...
, CiscoCiscoCisco may refer to:Companies:*Cisco Systems, a computer networking company* Certis CISCO, corporatised entity of the former Commercial and Industrial Security Corporation in Singapore...
, EmulexEmulexEmulex Corporation is a California based manufacturer of storage networking infrastructure solutions. The company's products include Fibre Channel host bus adapters , Fibre Channel over Ethernet converged network adapters , embedded storage switches, storage I/O controller and SAN storage switch...
, HP, IBMIBMInternational Business Machines Corporation or IBM is an American multinational technology and consulting corporation headquartered in Armonk, New York, United States. IBM manufactures and sells computer hardware and software, and it offers infrastructure, hosting and consulting services in areas...
, Juniper NetworksJuniper NetworksJuniper Networks is an information technology and computer networking products multinational company, founded in 1996. It is head quartered in Sunnyvale, California, USA. The company designs and sells high-performance Internet Protocol network products and services...
, QLogicQLogicQLogic Corporation is an Aliso Viejo, California-based designer and supplier of storage networking, high performance computing networking, and converged infrastructure solutions...
. The ad-hoc group formed to create version 0 proposals for enhancements that enable networking protocol convergence over Ethernet, specially Fibre ChannelFibre ChannelFibre Channel, or FC, is a gigabit-speed network technology primarily used for storage networking. Fibre Channel is standardized in the T11 Technical Committee of the InterNational Committee for Information Technology Standards , an American National Standards Institute –accredited standards...
. Pproposed specifications to IEEE 802.1 working groups included:- The Priority-based Flow ControlPriority-based flow controlPriority-based flow control, as defined in computer networking by the standard IEEE 802.1Qbb, provides a link-level flow control mechanism that can be controlled independently for each Class of Service , as defined by IEEE P802.1p. The goal of this mechanism is to ensure zero loss under congestion...
(PFC) Version 0 Specification was submitted to the IEEE 802.1Qbb working group. - The Enhanced Transmission Selection (ETS) Version 0 Specification was submitted to the IEEE 802.1Qaz working group.
- The Data Center Bridging eXchange (DCBX) Version 0 Specification was submitted to the IEEE 802.1Qaz working group.
- The Priority-based Flow Control
IEEE Task Group
The following have been adopted as IEEE standards:- Priority-based Flow ControlPriority-based flow controlPriority-based flow control, as defined in computer networking by the standard IEEE 802.1Qbb, provides a link-level flow control mechanism that can be controlled independently for each Class of Service , as defined by IEEE P802.1p. The goal of this mechanism is to ensure zero loss under congestion...
(PFC): IEEE 802.1Qbb provides a link level flow control mechanism that can be controlled independently for each frame priority. The goal of this mechanism is to ensure zero loss under congestion in DCB networks.
- Enhanced Transmission Selection (ETS): IEEE 802.1Qaz provides a common management framework for assignment of bandwidth to frame priorities.
- Congestion Notification: IEEE 802.1Qau provides end to end congestion management for protocols that are capable of transmission rate limiting to avoid frame loss. It is expected to benefit protocols such as TCP that do have native congestion management as it reacts to congestion in a more timely manner.
- Data Center Bridging Capabilities Exchange Protocol (DCBX): a discovery and capability exchange protocol that is used for conveying capabilities and configuration of the above features between neighbors to ensure consistent configuration across the network. This protocol leverages functionality provided by IEEE 802.1AB (LLDP). It is actually included in the 802.1az standard.
Other groups
- The IETF TRILLTRILL (computing)TRILL is a IETF Standard implemented by devices called RBridges or Routing Bridges. TRILL combines the advantages of bridges and routers and is the application of link state routing to the VLAN-aware customer-bridging problem. RBridges are compatible with and can incrementally replace previous...
(Transparent Interconnection of Lots of Links) standard provides least cost pair-wise data forwarding without configuration in multi-hop networks with arbitrary topology, safe forwarding even during periods of temporary loops, and support for multipathing of both unicast and multicast traffic. TRILL accomplishes this by using IS-ISIS-ISIntermediate System To Intermediate System , is a routing protocol designed to move information efficiently within a computer network, a group of physically connected computers or similar devices....
(Intermediate System to Intermediate System) link state routing and by encapsulating traffic using a header that includes a hop count. TRILL supports VLANs and frame priorities. Devices that implement TRILL are called RBridges. RBridges can incrementally replace IEEE 802.1 customer bridges. TRILL Working Group Charter
- IEEE 802.1aq Shortest Path Bridging (IEEE 802.1aq) 802.1aq specifies shortest path bridging of unicast and multicast Ethernet frames, to calculate multiple active topologies (virtual LANs) that can share learnt station location information. Two modes of operation are described, depending on whether the source Bridge is 802.1ad802.1adIEEE 802.1QinQ is an Ethernet networking standard formally known as IEEE 802.1adStandard approved 8 December 2005 and published May 26, 2006. and is an amendment to IEEE standard IEEE 802.1Q-1998. It is for Ethernet frame formats...
(QinQ) which is known as SPBV or 802.1ah (MACinMAC), which is known as SPBM. SPBV supports a VLAN using a VLAN Identifier (VID) per node to identify the shortest path tree (SPT) associated with that node. SPBM supports a VLAN by using one or more Backbone MAC addresses to identify each node and its associated SPT, and it can support multiple forwarding topologies for load sharing across equal cost trees using a single B-VID per forwarding topology. Both SPBV and SPBM use link state routing technology. SPBM by virtue of its MACinMAC encapsulation is more suitable for a large data centre than SPBV. 802.1aq defines 16 tunable multipath options as part of the base protocol, with an extensible multipathing mechanism to allow many more multipath variations in the future. 802.1aq supports the dynamic creation of virtual LAN's that interconnect all members with symmetric shortest path routes. The virtual LAN's can be deterministically assigned to the different multi paths providing a degree of traffic engineering in addition to multipathing and can grow or shrink with simple membership changes. 802.1aq is fully backward compatible with all 802.1 protocols. 802.1aq is expected to become an IEEE standard in 2012.
- Fibre Channel over EthernetFibre Channel over EthernetFibre Channel over Ethernet is an encapsulation of Fibre Channel frames over Ethernet networks. This allows Fibre Channel to use 10 Gigabit Ethernet networks while preserving the Fibre Channel protocol...
: T11 FCoE This project utilizes existing Fibre Channel protocols to run on Ethernet to enable servers to have access to Fibre Channel storage via Ethernet. As noted above, one of the drivers behind enhancing Ethernet is to support storage traffic. While iSCSIISCSIIn computing, iSCSI , is an abbreviation of Internet Small Computer System Interface, an Internet Protocol -based storage networking standard for linking data storage facilities. By carrying SCSI commands over IP networks, iSCSI is used to facilitate data transfers over intranets and to manage...
was available, it depends on TCP/IP and there was a desire to support storage traffic at layer 2. This gave rise to the development of the FCoE protocol, which needed reliable Ethernet transport. The standard was finalized in June 2009 by the ANSI T11 committee.
- IEEE 802.1p/QIEEE 802.1QIEEE 802.1Q is the networking standard that supports Virtual LANs on an Ethernet network. The standard defines a system of VLAN tagging for Ethernet frames and the accompanying procedures to be used by bridges and switches in handling such frames...
provides 8 traffic classes for priority based forwarding.
- IEEE 802.3IEEE 802.3IEEE 802.3 is a working group and a collection of IEEE standards produced by the working group defining the physical layer and data link layer's media access control of wired Ethernet. This is generally a local area network technology with some wide area network applications...
bd provides a mechanism link-level per priority pause flow control.
These new protocols will require new hardware and software in both the network and the server interconnect. These products are being developed by companies such as Avaya, Brocade, Cisco, Dell, EMC, Emulex, HP, Huawei, IBM, and Qlogic.