Explicit Congestion Notification
Encyclopedia
Explicit Congestion Notification (ECN) is an extension to the Internet Protocol
and to the Transmission Control Protocol
and is defined in RFC 3168 (2001). ECN allows end-to-end notification of network congestion
without dropping packets. ECN is an optional feature that is only used when both endpoints support it and are willing to use it. It is only effective when supported by the underlying network.
Conventionally, TCP/IP networks signal congestion by dropping packets. When ECN is successfully negotiated, an ECN-aware router may set a mark in the IP header instead of dropping a packet in order to signal impending congestion. The receiver of the packet echoes the congestion indication to the sender, which reduces its transmission rate as though it detected a dropped packet.
Rather than responding properly or ignoring the bits, some outdated or buggy network equipment drop packets which have ECN bits set.
Without ECN congestion indication echo is achieved indirectly by the detection of lost packets. With ECN, the congestion is indicated by setting the CE bits (see below) at an IP packet and is echoed back by the receiver to the transmitter by setting proper bits in the Transport Layer's protocol header. For example, when using TCP, the congestion indication is echoed back by setting the ECE bit (see below).
When both endpoints support ECN they mark their packets with
(AQM) queue (e.g. a queue that uses Random Early Detection
(RED)) that is experiencing congestion and the corresponding router supports ECN, it may change the codepoint to
. This act is referred to as “marking” and its purpose is to inform the receiving endpoint of impending congestion
. At the receiving endpoint, this congestion indication is handled by the upper layer protocol (transport layer protocol) and needs to be echoed back to the transmitting node in order to signal it to reduce its transmission rate.
Because the
Use of ECN on a TCP connection is optional; for ECN to be used, it must be negotiated at connection establishment by including suitable options in the SYN and SYN-ACK segments.
When ECN has been negotiated on a TCP connection, the sender indicates that IP packets that carry TCP segments of that connection are carrying traffic from an ECN Capable Transport by marking them with an ECT codepoint. This allows intermediate routers that support ECN to mark those IP packets with the CE codepoint instead of dropping them in order to signal impending congestion.
Upon receiving an IP packet with the Congestion Experienced codepoint, the TCP receiver echoes back this congestion indication using the ECE flag in the TCP header. When an endpoint receives a TCP segment with the ECE bit it reduces its congestion window as for a packet drop. It then acknowledges the congestion indication by sending a segment with the CWR bit set.
A node keeps transmitting TCP segments with the ECE bit set until it receives a segment with the CWR bit set.
To see affected packets with Tcpdump
, use the filter predicate
A recent proposal suggests marking SYN-ACK packets as ECN-capable. This improvement, known as ECN+, has been shown to provide dramatic improvements to performance of short-lived TCP connections.
and SCTP
. The general principle is similar to TCP
, although the details of the on-the-wire encoding differ.
It should in principle be possible to use ECN with protocols layered above UDP
. However, UDP requires that congestion control be performed by the application, and current networking APIs do not give access to the ECN bits.
(AQM) policy, the benefits of ECN depend on the precise AQM being used. A few observations, however, appear to hold across different AQMs.
As expected, ECN reduces the number of packets dropped by a TCP connection, which, by avoiding a retransmission, reduces latency and especially jitter. This effect is most drastic when the TCP connection has a single outstanding segment, when it is able to avoid an RTO timeout; this is often the case for interactive connections (such as remote logins) and transactional protocols (such as HTTP requests, the conversational phase of SMTP, or SQL requests).
Effects of ECN on bulk throughput are less clear because modern TCP implementations are fairly good at resending dropped segments in a timely manner when the sender's window is large.
Use of ECN has been found to be detrimental to performance on highly congested networks when using AQM algorithms that never drop packets. Modern AQM implementations avoid this pitfall by dropping rather than marking packets at very high load.
10.5 and 10.6 implements ECN support for TCP. It is controlled using the following boolean sysctl
variables:
To make the settings persistent, put following lines in /etc/sysctl.conf:
The Linux
kernel supports three states of ECN for TCP: "No ECN" (0), "Use ECN" (1) and "Support ECN"/"Server Mode" (2) (i.e. only advertise ECN support when asked for). Its default behaviour is to support ECN if the other side supports it (server-mode). In most kernel versions, full ECN usage (1) can be activated through the sysctl
interface:
FreeBSD
8.0 implements ECN support for TCP. It can be activated through the sysctl
interface:
, routers must be configured with a suitable queue discipline in order to perform ECN marking.
Cisco IOS routers perform ECN marking if configured with the WRED
queuing discipline since version 12.2(8)T.
Linux routers perform ECN marking if configured with one of the RED
or GRED queue disciplines with an explicit ecn parameter, or by using the sfb discipline.
Modern BSD implementations, such as FreeBSD
, NetBSD
and OpenBSD
, have support for ECN marking in the ALTQ
queueing implementation for a number of queuing disciplines, notably RED
and Blue
.
Internet Protocol
The Internet Protocol is the principal communications protocol used for relaying datagrams across an internetwork using the Internet Protocol Suite...
and to the Transmission Control Protocol
Transmission Control Protocol
The Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
and is defined in RFC 3168 (2001). ECN allows end-to-end notification of network congestion
Network congestion
In data networking and queueing theory, network congestion occurs when a link or node is carrying so much data that its quality of service deteriorates. Typical effects include queueing delay, packet loss or the blocking of new connections...
without dropping packets. ECN is an optional feature that is only used when both endpoints support it and are willing to use it. It is only effective when supported by the underlying network.
Conventionally, TCP/IP networks signal congestion by dropping packets. When ECN is successfully negotiated, an ECN-aware router may set a mark in the IP header instead of dropping a packet in order to signal impending congestion. The receiver of the packet echoes the congestion indication to the sender, which reduces its transmission rate as though it detected a dropped packet.
Rather than responding properly or ignoring the bits, some outdated or buggy network equipment drop packets which have ECN bits set.
Operation
ECN requires specific support at the Internet and Transport layers for the following reasons:- In TCP/IP routers operate mostly on the Internet layer while transmission rate is handled by the endpoints at the Transport layer
- Congestion may be handled only by the transmitter but since it is known to have happened only after a packet was sent, there must be an echo of the congestion indication by the receiver to the transmitter.
Without ECN congestion indication echo is achieved indirectly by the detection of lost packets. With ECN, the congestion is indicated by setting the CE bits (see below) at an IP packet and is echoed back by the receiver to the transmitter by setting proper bits in the Transport Layer's protocol header. For example, when using TCP, the congestion indication is echoed back by setting the ECE bit (see below).
Operation of ECN with IP
ECN uses the two least significant (right-most) bits of the DiffServ field in the IPv4 or IPv6 header to encode four different codepoints:- 00: Non ECN-Capable Transport — Non-ECT
- 10: ECN Capable Transport — ECT(0)
- 01: ECN Capable Transport — ECT(1)
- 11: Congestion Encountered — CE
When both endpoints support ECN they mark their packets with
ECT(0) or ECT(1). If the packet traverses an Active Queue ManagementActive Queue Management
In Internet routers, active queue management is a technique that consists in dropping or ECN-marking packets before a router's queue is full.-Queue management:...
(AQM) queue (e.g. a queue that uses Random Early Detection
Random early detection
Random early detection , also known as random early discard or random early drop is an active queue management algorithm. It is also a congestion avoidance algorithm....
(RED)) that is experiencing congestion and the corresponding router supports ECN, it may change the codepoint to
CE instead of dropping the packetPacket loss
Packet loss occurs when one or more packets of data travelling across a computer network fail to reach their destination. Packet loss is distinguished as one of the three main error types encountered in digital communications; the other two being bit error and spurious packets caused due to noise.-...
. This act is referred to as “marking” and its purpose is to inform the receiving endpoint of impending congestion
Network congestion
In data networking and queueing theory, network congestion occurs when a link or node is carrying so much data that its quality of service deteriorates. Typical effects include queueing delay, packet loss or the blocking of new connections...
. At the receiving endpoint, this congestion indication is handled by the upper layer protocol (transport layer protocol) and needs to be echoed back to the transmitting node in order to signal it to reduce its transmission rate.
Because the
CE indication can only be handled effectively by an upper layer protocol that supports it, ECN is only used in conjunction with upper layer protocols (e.g. TCP) that
- support congestion control and,
- have a method for echoing the CE indication to the transmitting endpoint.
Operation of ECN with TCP
TCP supports ECN using two flags in the TCP header. Those two bits are used to echo back the congestion indication (i.e. signal the sender to reduce the amount of information it sends) and to acknowledge that the congestion-indication echoing was received. These are the ECN-Echo (ECE) and Congestion Window Reduced (CWR) bits.Use of ECN on a TCP connection is optional; for ECN to be used, it must be negotiated at connection establishment by including suitable options in the SYN and SYN-ACK segments.
When ECN has been negotiated on a TCP connection, the sender indicates that IP packets that carry TCP segments of that connection are carrying traffic from an ECN Capable Transport by marking them with an ECT codepoint. This allows intermediate routers that support ECN to mark those IP packets with the CE codepoint instead of dropping them in order to signal impending congestion.
Upon receiving an IP packet with the Congestion Experienced codepoint, the TCP receiver echoes back this congestion indication using the ECE flag in the TCP header. When an endpoint receives a TCP segment with the ECE bit it reduces its congestion window as for a packet drop. It then acknowledges the congestion indication by sending a segment with the CWR bit set.
A node keeps transmitting TCP segments with the ECE bit set until it receives a segment with the CWR bit set.
To see affected packets with Tcpdump
Tcpdump
tcpdump is a common packet analyzer that runs under the command line. It allows the user to intercept and display TCP/IP and other packets being transmitted or received over a network to which the computer is attached...
, use the filter predicate
(tcp[13] & 0xc0 != 0).ECN and TCP control packets
Since TCP does not perform congestion control on control packets (pure ACKs, SYN, FIN segments), control packets are usually not marked as ECN-capable.A recent proposal suggests marking SYN-ACK packets as ECN-capable. This improvement, known as ECN+, has been shown to provide dramatic improvements to performance of short-lived TCP connections.
Operation of ECN with other transport protocols
ECN is also defined for other transport-layer protocols that perform congestion control, notably DCCPDatagram Congestion Control Protocol
The Datagram Congestion Control Protocol is a message-oriented Transport Layer protocol. DCCP implements reliable connection setup, teardown, Explicit Congestion Notification , congestion control, and feature negotiation. DCCP was published as RFC 4340, a proposed standard, by the IETF in March,...
and SCTP
Stream Control Transmission Protocol
In computer networking, the Stream Control Transmission Protocol is a Transport Layer protocol, serving in a similar role to the popular protocols Transmission Control Protocol and User Datagram Protocol...
. The general principle is similar to TCP
Transmission Control Protocol
The Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
, although the details of the on-the-wire encoding differ.
It should in principle be possible to use ECN with protocols layered above UDP
User Datagram Protocol
The User Datagram Protocol is one of the core members of the Internet Protocol Suite, the set of network protocols used for the Internet. With UDP, computer applications can send messages, in this case referred to as datagrams, to other hosts on an Internet Protocol network without requiring...
. However, UDP requires that congestion control be performed by the application, and current networking APIs do not give access to the ECN bits.
Effects on performance
Since ECN is only effective in combination with an Active Queue ManagementActive Queue Management
In Internet routers, active queue management is a technique that consists in dropping or ECN-marking packets before a router's queue is full.-Queue management:...
(AQM) policy, the benefits of ECN depend on the precise AQM being used. A few observations, however, appear to hold across different AQMs.
As expected, ECN reduces the number of packets dropped by a TCP connection, which, by avoiding a retransmission, reduces latency and especially jitter. This effect is most drastic when the TCP connection has a single outstanding segment, when it is able to avoid an RTO timeout; this is often the case for interactive connections (such as remote logins) and transactional protocols (such as HTTP requests, the conversational phase of SMTP, or SQL requests).
Effects of ECN on bulk throughput are less clear because modern TCP implementations are fairly good at resending dropped segments in a timely manner when the sender's window is large.
Use of ECN has been found to be detrimental to performance on highly congested networks when using AQM algorithms that never drop packets. Modern AQM implementations avoid this pitfall by dropping rather than marking packets at very high load.
Implementations
Many modern implementations of the TCP/IP protocol suite have some support for ECN; however, they usually ship with ECN disabled.Microsoft Windows
Windows Server 2008, Windows 7, and Windows Vista support ECN for TCP but it is disabled by default. ECN support can be enabled with the following shell command:netsh interface tcp set global ecncapability=enabledApple Macintosh
Mac OS XMac OS X
Mac OS X is a series of Unix-based operating systems and graphical user interfaces developed, marketed, and sold by Apple Inc. Since 2002, has been included with all new Macintosh computer systems...
10.5 and 10.6 implements ECN support for TCP. It is controlled using the following boolean sysctl
Sysctl
Sysctl is an interface for examining and dynamically changing parameters in the BSD and Linux operating systems. The implementation mechanism in these two systems is very different....
variables:
net.inet.tcp.ecn_negotiate_in (enable ECN on incoming connections that already have ECN flags set), net.inet.tcp.ecn_initiate_out (try to initiate outgoing connections with ECN enabled). Both variables default to 0, but can be set to 1 to enable the respective behavior:sysctl -w net.inet.tcp.ecn_negotiate_in=1 and sysctl -w net.inet.tcp.ecn_initiate_out=1To make the settings persistent, put following lines in /etc/sysctl.conf:
net.inet.tcp.ecn_initiate_out=1
net.inet.tcp.ecn_negotiate_in=1
Linux
The Linux
Linux
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
kernel supports three states of ECN for TCP: "No ECN" (0), "Use ECN" (1) and "Support ECN"/"Server Mode" (2) (i.e. only advertise ECN support when asked for). Its default behaviour is to support ECN if the other side supports it (server-mode). In most kernel versions, full ECN usage (1) can be activated through the sysctl
Sysctl
Sysctl is an interface for examining and dynamically changing parameters in the BSD and Linux operating systems. The implementation mechanism in these two systems is very different....
interface:
sysctl net.ipv4.tcp_ecn=1BSD
FreeBSD
FreeBSD
FreeBSD is a free Unix-like operating system descended from AT&T UNIX via BSD UNIX. Although for legal reasons FreeBSD cannot be called “UNIX”, as the direct descendant of BSD UNIX , FreeBSD’s internals and system APIs are UNIX-compliant...
8.0 implements ECN support for TCP. It can be activated through the sysctl
Sysctl
Sysctl is an interface for examining and dynamically changing parameters in the BSD and Linux operating systems. The implementation mechanism in these two systems is very different....
interface:
sysctl net.inet.tcp.ecn.enable=1ECN support in IP by routers
Since ECN marking in routers is dependent on some form of active queue managementActive Queue Management
In Internet routers, active queue management is a technique that consists in dropping or ECN-marking packets before a router's queue is full.-Queue management:...
, routers must be configured with a suitable queue discipline in order to perform ECN marking.
Cisco IOS routers perform ECN marking if configured with the WRED
Random early detection
Random early detection , also known as random early discard or random early drop is an active queue management algorithm. It is also a congestion avoidance algorithm....
queuing discipline since version 12.2(8)T.
Linux routers perform ECN marking if configured with one of the RED
Random early detection
Random early detection , also known as random early discard or random early drop is an active queue management algorithm. It is also a congestion avoidance algorithm....
or GRED queue disciplines with an explicit ecn parameter, or by using the sfb discipline.
Modern BSD implementations, such as FreeBSD
FreeBSD
FreeBSD is a free Unix-like operating system descended from AT&T UNIX via BSD UNIX. Although for legal reasons FreeBSD cannot be called “UNIX”, as the direct descendant of BSD UNIX , FreeBSD’s internals and system APIs are UNIX-compliant...
, NetBSD
NetBSD
NetBSD is a freely available open source version of the Berkeley Software Distribution Unix operating system. It was the second open source BSD descendant to be formally released, after 386BSD, and continues to be actively developed. The NetBSD project is primarily focused on high quality design,...
and OpenBSD
OpenBSD
OpenBSD is a Unix-like computer operating system descended from Berkeley Software Distribution , a Unix derivative developed at the University of California, Berkeley. It was forked from NetBSD by project leader Theo de Raadt in late 1995...
, have support for ECN marking in the ALTQ
ALTQ
ALTQ is an ALTernate Queueing framework for BSD. ALTQ provides queueing disciplines and other QoS related components required to realize resource-sharing and Quality of Service. It is most commonly implemented on BSD-based routers...
queueing implementation for a number of queuing disciplines, notably RED
Random early detection
Random early detection , also known as random early discard or random early drop is an active queue management algorithm. It is also a congestion avoidance algorithm....
and Blue
Blue (queue management algorithm)
Blue is an Active Queue Management algorithm. Like RED, it operates by randomly dropping or ECN-marking packets in a router's queue before it overflows...
.
See also
- Data Center TCPData Center TCPData Center TCP utilizes Explicit Congestion Notification to enhance the Transmission Control Protocol congestion control algorithm. It is used in data center networks. Whereas the standard TCP congestion control algorithm is only able to detect the presence of congestion, DCTCP, using ECN, is...
- Network congestion avoidance
- Type of ServiceType of ServiceThe type of service field in the IPv4 header has had various purposes over the years, and has been defined in different ways by five RFCs...
field - Transmission Control ProtocolTransmission Control ProtocolThe Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
External links
- RFC 4774 — Specifying Alternate Semantics for the Explicit Congestion Notification (ECN) Field
- Sally Floyd's ECN page