Netflow
Encyclopedia
Cisco Systems
Cisco Systems, Inc. is an American multinational corporation headquartered in San Jose, California, United States, that designs and sells consumer electronics, networking, voice, and communications technology and services. Cisco has more than 70,000 employees and annual revenue of US$...
for collecting IP
Internet Protocol
The Internet Protocol is the principal communications protocol used for relaying datagrams across an internetwork using the Internet Protocol Suite...
traffic information. NetFlow has become an industry standard for traffic monitoring and is supported by platforms other than Cisco IOS and NXOS such as Juniper
Juniper Networks
Juniper Networks is an information technology and computer networking products multinational company, founded in 1996. It is head quartered in Sunnyvale, California, USA. The company designs and sells high-performance Internet Protocol network products and services...
routers, Enterasys Switches, vNetworking in version 5 of vSphere, Linux
Linux
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
, FreeBSD
FreeBSD
FreeBSD is a free Unix-like operating system descended from AT&T UNIX via BSD UNIX. Although for legal reasons FreeBSD cannot be called “UNIX”, as the direct descendant of BSD UNIX , FreeBSD’s internals and system APIs are UNIX-compliant...
, NetBSD
NetBSD
NetBSD is a freely available open source version of the Berkeley Software Distribution Unix operating system. It was the second open source BSD descendant to be formally released, after 386BSD, and continues to be actively developed. The NetBSD project is primarily focused on high quality design,...
and OpenBSD
OpenBSD
OpenBSD is a Unix-like computer operating system descended from Berkeley Software Distribution , a Unix derivative developed at the University of California, Berkeley. It was forked from NetBSD by project leader Theo de Raadt in late 1995...
.
Protocol description
Cisco routers and Enterasys Switches that have the NetFlow feature enabled generate NetFlow records; these are exported from the router in User Datagram Protocol (UDPUser Datagram Protocol
The User Datagram Protocol is one of the core members of the Internet Protocol Suite, the set of network protocols used for the Internet. With UDP, computer applications can send messages, in this case referred to as datagrams, to other hosts on an Internet Protocol network without requiring...
) or Stream Control Transmission Protocol (SCTP
Stream Control Transmission Protocol
In computer networking, the Stream Control Transmission Protocol is a Transport Layer protocol, serving in a similar role to the popular protocols Transmission Control Protocol and User Datagram Protocol...
) packets and collected using a netflow collector.
Other vendors provide similar features for their routers but with different names:
- Jflow or cflowd for Juniper NetworksJuniper NetworksJuniper Networks is an information technology and computer networking products multinational company, founded in 1996. It is head quartered in Sunnyvale, California, USA. The company designs and sells high-performance Internet Protocol network products and services...
- NetStream for 3Com/H3C3Com3Com was a pioneering digital electronics manufacturer best known for its computer network infrastructure products. The company was co-founded in 1979 by Robert Metcalfe, Howard Charney, Bruce Borden, and Greg Shaw...
- NetStream for Huawei TechnologyHuaweiHuawei is a Chinese multinational networking and telecommunications equipment and services company headquartered in Shenzhen, Guangdong, China...
- Cflowd for Alcatel-Lucent
- Rflow for EricssonEricssonEricsson , one of Sweden's largest companies, is a provider of telecommunication and data communication systems, and related services, covering a range of technologies, including especially mobile networks...
- AppFlow Citrix
NetFlow and IPFIX
Although initially implemented by Cisco, NetFlow is described in an informational document: RFC 3954 – Cisco Systems NetFlow Services Export Version 9. The NetFlow protocol itself has been superseded by Internet Protocol Flow Information eXport (IPFIX). Based on the NetFlow Version 9 implementation, IPFIX is the industry standard with RFC 5101, RFC 5102, etc. Network infrastructure vendors are already adding IPFIX support to their devices.Network Flows
A network flow has been defined in many ways. The traditional Cisco definition is to use a 7-tupleTuple
In mathematics and computer science, a tuple is an ordered list of elements. In set theory, an n-tuple is a sequence of n elements, where n is a positive integer. There is also one 0-tuple, an empty sequence. An n-tuple is defined inductively using the construction of an ordered pair...
key, where a flow is defined as a unidirectional sequence of packets all sharing all of the following 7 values:
- Source IP addressIP addressAn Internet Protocol address is a numerical label assigned to each device participating in a computer network that uses the Internet Protocol for communication. An IP address serves two principal functions: host or network interface identification and location addressing...
- Destination IP addressIP addressAn Internet Protocol address is a numerical label assigned to each device participating in a computer network that uses the Internet Protocol for communication. An IP address serves two principal functions: host or network interface identification and location addressing...
- Source port for UDPUser Datagram ProtocolThe User Datagram Protocol is one of the core members of the Internet Protocol Suite, the set of network protocols used for the Internet. With UDP, computer applications can send messages, in this case referred to as datagrams, to other hosts on an Internet Protocol network without requiring...
or TCPTransmission Control ProtocolThe Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
, 0 for other protocols - Destination port for UDPUser Datagram ProtocolThe User Datagram Protocol is one of the core members of the Internet Protocol Suite, the set of network protocols used for the Internet. With UDP, computer applications can send messages, in this case referred to as datagrams, to other hosts on an Internet Protocol network without requiring...
or TCPTransmission Control ProtocolThe Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
, type and code for ICMPInternet Control Message ProtocolThe Internet Control Message Protocol is one of the core protocols of the Internet Protocol Suite. It is chiefly used by the operating systems of networked computers to send error messages indicating, for example, that a requested service is not available or that a host or router could not be...
, or 0 for other protocols - IP protocol
- Ingress interface (SNMPSimple Network Management ProtocolSimple Network Management Protocol is an "Internet-standard protocol for managing devices on IP networks. Devices that typically support SNMP include routers, switches, servers, workstations, printers, modem racks, and more." It is used mostly in network management systems to monitor...
ifIndex) - IP Type of ServiceType of ServiceThe type of service field in the IPv4 header has had various purposes over the years, and has been defined in different ways by five RFCs...
Flexible NetFlow and IPFIX support user-defined flow keys.
The router will output a flow record when it determines that the flow is finished. It does this by flow aging: when the router sees new traffic for an existing flow it resets the aging counter. Also, TCP session termination in a TCP flow causes the router to expire the flow. Routers can also be configured to output a flow record at a fixed interval even if the flow is still ongoing. In Flexible NetFlow (FNF) an administrator could actually define flow properties on the router.
NetFlow Record
A NetFlow record can contain a wide variety of information about the traffic in a given flow. NetFlow version 5 (one of the most commonly used versions, followed by version 9) contains the following:- Version number
- Sequence number
- Input and output interface indices used by SNMPSimple Network Management ProtocolSimple Network Management Protocol is an "Internet-standard protocol for managing devices on IP networks. Devices that typically support SNMP include routers, switches, servers, workstations, printers, modem racks, and more." It is used mostly in network management systems to monitor...
(ifIndex in IF-MIB). - Timestamps for the flow start and finish time, in milliseconds since the last boot.
- Number of bytes and packets observed in the flow
- Layer 3Network LayerThe network layer is layer 3 of the seven-layer OSI model of computer networking.The network layer is responsible for packet forwarding including routing through intermediate routers, whereas the data link layer is responsible for media access control, flow control and error checking.The network...
headers:- Source & destination IP addresses
- Source and destination port numbers
- IP protocol
- Type of ServiceType of ServiceThe type of service field in the IPv4 header has had various purposes over the years, and has been defined in different ways by five RFCs...
(ToS) value
- In the case of TCP flows, the union of all TCP flags observed over the life of the flow.
- Layer 3 RoutingRoutingRouting is the process of selecting paths in a network along which to send network traffic. Routing is performed for many kinds of networks, including the telephone network , electronic data networks , and transportation networks...
information:- IP address of the immediate next-hop (not the BGP nexthop) along the route to the destination
- Source & destination IP masks (prefix lengths in the CIDRClassless Inter-Domain RoutingClassless Inter-Domain Routing is a method for allocating IP addresses and routing Internet Protocol packets. The Internet Engineering Task Force introduced CIDR in 1993 to replace the previous addressing architecture of classful network design in the Internet...
notation)
Some routers will also include the source and destination Autonomous System
Autonomous system (Internet)
Within the Internet, an Autonomous System is a collection of connected Internet Protocol routing prefixes under the control of one or more network operators that presents a common, clearly defined routing policy to the Internet....
(AS) number. Depending on the router configuration, it can also be the immediate neighbor AS. That information is not always accurate, and the local AS will be indicated by the same value (zero) as any unknown AS.
NetFlow version 9 can include all of these fields and can optionally include additional information such as Multiprotocol Label Switching
Multiprotocol Label Switching
Multiprotocol Label Switching is a mechanism in high-performance telecommunications networks that directs data from one network node to the next based on short path labels rather than long network addresses, avoiding complex lookups in a routing table. The labels identify virtual links between...
(MPLS) labels and IPv6
IPv6
Internet Protocol version 6 is a version of the Internet Protocol . It is designed to succeed the Internet Protocol version 4...
addresses and ports,
By analyzing flow data, a picture of traffic flow and traffic volume in a network can be built. The NetFlow record format has evolved over time, hence the inclusion of version numbers. Cisco maintains details of the different version numbers and the layout of the packets for each version.
NetFlow records are usually sent via a UDP or SCTP
Stream Control Transmission Protocol
In computer networking, the Stream Control Transmission Protocol is a Transport Layer protocol, serving in a similar role to the popular protocols Transmission Control Protocol and User Datagram Protocol...
in newer software, and for efficiency reasons, the router does not store flow records once they are exported. Therefore, if the NetFlow record is dropped due to network congestion
Network congestion
In data networking and queueing theory, network congestion occurs when a link or node is carrying so much data that its quality of service deteriorates. Typical effects include queueing delay, packet loss or the blocking of new connections...
, it is lost forever—there's no way for the router to resend it (this is correct for UDP NetFlow only). The IP address of the netflow collector and the port upon which it is listening must be configured on the sending router but is usually either on ports 2055, 9555, or 9995. NetFlow is also enabled on a per-interface basis to avoid unnecessary burdening of the router's CPU. NetFlow is generally based on the packets input to interfaces where it is enabled. This avoids double counting and saves work for the router. It also allows the router to export NetFlow records for dropped packets.
ICMP
Internet Control Message Protocol
The Internet Control Message Protocol is one of the core protocols of the Internet Protocol Suite. It is chiefly used by the operating systems of networked computers to send error messages indicating, for example, that a requested service is not available or that a host or router could not be...
records use the Destination Port number field to define the ICMP Type and CODE in HEX. The type and code are concatenated before the conversion. An ICMP Host Unreachable message would be TYPE 3 with a CODE of 01 (301). NetFlow converts this value to a hex value of 769 when exported.
Sampled NetFlow
This variant was originally introduced by Cisco, but is now used in all high-end routers that implement NetFlow.Maintaining NetFlow data can be computationally expensive for the router and burden the router's CPU or hardware to the point where it runs out of capacity. To avoid problems caused by router CPU load, Cisco provides "Sampled NetFlow". Rather than looking at every packet to maintain NetFlow records, the router looks at every nth packet, where n can be configured (as in Deterministic NetFlow, used on Cisco's GSRs) or it is a randomly selecting interval (as used in Random Sampled NetFlow, used on all other Cisco platforms).
The sampling is often the same for all interfaces, but can be adjusted per interface for some routers.
When Sampled NetFlow is used, the NetFlow records must be adjusted for the effect of sampling - traffic volumes, in particular, are now an estimate rather than the actual measured flow volume.
The sampling rate is indicated in a header field of NetFlow version 5 (same sampling rate for all interfaces) or in option records of NetFlow version 9 (sampling rate per interface)
Cisco's NetFlow Security Event Logging
Introduced with the launch of the Cisco ASACisco ASA
In computer networking, Cisco ASA 5500 Series Adaptive Security Appliances, or simply Cisco ASA 5500 Series, is Cisco's line of network security devices introduced in 2005, that succeeded three existing lines of popular Cisco products:...
5580 products, NetFlow Security Event Logging utilizes NetFlow v9 fields and templates in order to efficiently deliver security telemetry in high performance environments. NetFlow Security Event Logging scales better than syslog
Syslog
Syslog is a standard for computer data logging. It allows separation of the software that generates messages from the system that stores them and the software that reports and analyzes them...
while offering the same level of detail and granularity in logged events.
NetFlow Monitoring Based on Standalone Probes
Historically, Deep packet inspection
Deep packet inspection
Deep Packet Inspection is a form of computer network packet filtering that examines the data part of a packet as it passes an inspection point, searching for protocol non-compliance, viruses, spam, intrusions or predefined criteria to decide if the packet can...
is easier to implement in a dedicated probe than in a router. However, this approach also has some drawbacks:
- probes must be deployed on every link that must be observed, causing additional hardware, setup and maintenance costs.
- probes will not report separate input and output interface information like a report from a router would.
- probes may have problems reporting reliably the NetFlow fields related to routing, like AS NumbersAutonomous system (Internet)Within the Internet, an Autonomous System is a collection of connected Internet Protocol routing prefixes under the control of one or more network operators that presents a common, clearly defined routing policy to the Internet....
or IP masksClassless Inter-Domain RoutingClassless Inter-Domain Routing is a method for allocating IP addresses and routing Internet Protocol packets. The Internet Engineering Task Force introduced CIDR in 1993 to replace the previous addressing architecture of classful network design in the Internet...
, because they can hardly be expected to use exactly the same routing information as a router.
NetFlow collection from dedicated probes is well suited for observation of critical links, whereas NetFlow on routers provides a Network-wide view of the traffic that can be used for capacity planning, accounting, performance monitoring, and security.
Versions
| Version | Comment |
|---|---|
| v1 | First implementation, now obsolete, and restricted to IPv4 IPv4 Internet Protocol version 4 is the fourth revision in the development of the Internet Protocol and the first version of the protocol to be widely deployed. Together with IPv6, it is at the core of standards-based internetworking methods of the Internet... (without IP mask CIDR notation CIDR notation is a compact specification of an Internet Protocol address and its associated routing prefix. Classless Inter-Domain Routing is an Internet Protocol address allocation and route aggregation methodology used within the Internet addressing architecture that replaced the IPv4 classful... and AS Numbers Autonomous system (Internet) Within the Internet, an Autonomous System is a collection of connected Internet Protocol routing prefixes under the control of one or more network operators that presents a common, clearly defined routing policy to the Internet.... ). |
| v2 | Cisco internal version, never released. |
| v3 | Cisco internal version, never released. |
| v4 | Cisco internal version, never released. |
| v5 | Most common version, available (as of 2009) on many routers from different brands, but restricted to IPv4 IPv4 Internet Protocol version 4 is the fourth revision in the development of the Internet Protocol and the first version of the protocol to be widely deployed. Together with IPv6, it is at the core of standards-based internetworking methods of the Internet... flows. |
| v6 | No longer supported by Cisco. Encapsulation information (?). |
| v7 | Like version 5 with a source router field. Used (only?) on Cisco Catalyst switches. |
| v8 | Several aggregation form, but only for information that is already present in version 5 records |
| v9 | Template Based, available (as of 2009) on some recent routers. Mostly used to report flows like IPv6 IPv6 Internet Protocol version 6 is a version of the Internet Protocol . It is designed to succeed the Internet Protocol version 4... , MPLS Multiprotocol Label Switching Multiprotocol Label Switching is a mechanism in high-performance telecommunications networks that directs data from one network node to the next based on short path labels rather than long network addresses, avoiding complex lookups in a routing table. The labels identify virtual links between... , or even plain IPv4 IPv4 Internet Protocol version 4 is the fourth revision in the development of the Internet Protocol and the first version of the protocol to be widely deployed. Together with IPv6, it is at the core of standards-based internetworking methods of the Internet... with BGP nexthop. |
| IPFIX | aka v10; IETF Standardized NetFlow 9 with several extensions like Enterprise-defined fields types, and variable length fields. |
See also
- Traffic flow (computer networking)
- IP Flow Information ExportIP Flow Information ExportInternet Protocol Flow Information Export is an IETF working group. It was created from the need for a common, universal standard of export for Internet Protocol flow information from routers, probes, and other devices that is used by mediation systems, accounting/billing systems, and network...
(IPFIX) - IETF work to standardize flow export, based on NetFlow version 9 - sFlowSFlowsFlow is a technology for monitoring network, wireless andhost devices.The sFlow.org consortium is the authoritative source for the sFlow protocol specifications: previous version of sFlow, including RFC 3176, have been deprecated.- Operation :...
- alternative to NetFlow (mandatory sampling, no flow cache, no templates) - nProbe - an extensible NetFlow v5/v9/IPFIX GPL probe for IPv4/v6
External links
- NetFlow/FloMA: Pointers and Software Provided by SWITCH. - One of the most comprehensive list including all the open source and research works.
- FloCon - The Annual Conference put on by CERT/CC dealing with NetFlow and other flow analysis works.
- Basic NetFlow information on the Cisco Site
- RFC3334 - Policy-Based Accounting
- RFC3954 - NetFlow Version 9
- RFC3955 - Candidate Protocols for IP Flow Information Export (IPFIX)
- RFC5101 - Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information (IPFIX)
- RFC5102 - Information Model for IP Flow Information Export
- RFC5103 - Bidirectional Flow Export Using IP Flow Information Export
- RFC5153 - IPFIX Implementation Guidelines
- RFC5470 - Architecture for IP Flow Information Export
- RFC5471 - Guidelines for IP Flow Information Export (IPFIX) Testing
- RFC5472 - IP Flow Information Export (IPFIX) Applicability
- RFC5473 - Reducing Redundancy in IP Flow Information Export (IPFIX) and Packet Sampling (PSAMP) Reports
- List of software related to flow accounting
- AppFlow specifications and standards track discussion