5.2. Background on the NetShark architecture¶
This document is a brief introduction to the capabilities and organization of the NetShark appliance. No background or previous experience with NetShark or Packet Analyzer (formerly Pilot) is assumed, but you are encouraged to watch the Packet Analyzer with NetShark - An Introduction and/or spend some time using Packet Analyzer to familiarize yourself with the platform and its capabilities.
5.2.1. Views¶
All analysis and reporting from a NetShark appliance revolves around the concept of a view. Views are extremely flexible and can be used for many different things but at the core, a view consists simply of a source of packets to analyze and a set of statistics to be computed on those packets. The power of the NetShark platform derives from the number of statistics that can be exported and the number of ways those statistics can be combined and organized.
An example of a simple view is “Bandwidth over Time”. In this view, the length statistic is summed across all packets for fixed small time intervals (e.g., 1 second). By plotting the sums for a series of consecutive time intervals, we get a graph of how the traffic load on a specific network link has varied over time.
A more complex example is the Packet Analyzer “Served Web Pages by
Client IP” view. In Packet Analyzer, this view presents a table with
three columns: ip address, URL, and count. There is a separate row in
the table for each unique (client ip address, URL) pair, showing the
number of times the given URL has been fetched from the given client
ip address. This view could additionally be filtered, for example
showing only URLs fetched using the Chrome web browser or showing only
URLs with mime type image/jpeg. In this case, the statistics
extracted are more complex – extracting the URL (or browser or
content type) from an HTTP session requires reassembling the
underlying TCP connection from individual packets. And the
organization of the statistics is a little more complex, the NetShark
internally constructs and updates a table much like the one presented
in the Packet Analyzer user interface.
These examples are just meant to help illustrate what views are at a high level. The details about exactly how views are constructed and accessed are later in this document and elsewhere in the FlyScript documentation.
5.2.2. Packet Sources¶
This section describes the abstractions provided by a NetShark appliance for organizing and identifying the packets that can be used as the basis for creating a view.
5.2.2.1. Interfaces¶
Perhaps the simplest packet source is a raw network interface, also called a capture port. A capture port on a NetShark appliance is typically connected to a switch port that mirrors live traffic to the NetShark for monitoring and analysis (often called a SPAN port). When a view is applied to an interface, every packet that arrives on the interface is available to the view for processing. A view applied to an interface is also a live view, meaning that as new packets arrive, they are immediately processed by the view and show up in the view outputs.
5.2.2.2. Capture Jobs¶
A capture job is a long-running task on a NetShark appliance that captures, stores, and indexes live traffic. It logically consists of a physical network interface and an optional filter to limit which packets should be captured. The actual packets and associated indexing information are all stored locally on the NetShark appliance, using a custom filesystem tailored to the task of capturing traffic at a high rate and indexing the capture traffic for efficient queries. The capabilities of capture jobs are accessed from Python code using Capture Job Objects.
There are actually two different logical packets source associated with a capture job. The job itself refers to all the packets stored on disk that have been previously captured as part of the job. A view created with a capture job as the packet source is applied to all packets that are available on disk, but is not applied to new packets as they arrive. Instead, each capture job has an associated virtual device that is used for ongoing live traffic. The virtual device corresponding to a capture job appears like an interface, as described above.
The virtual device associated with a capture job may appear redundant with just applying a view with a filter to an interface. The filters associated with capture jobs are generally more efficient than view filters, so on an appliance handling a high sustained rate of packet capture, using a capture job is preferrable. A full description of the tradeoffs is beyond the scope of this document, for more details see the NetShark manual.
5.2.2.3. Trace Clips¶
The packets captured as part of a capture job are kept in a first-in-first-out (FIFO) structured store. For cases when some packets need to be retained for longer, a trace clip is used. A trace clip identifies some subset of the packets stored as part of a capture job and optionally locks them, meaning that those packets will be kept on disk until the trace clip is deleted. Note that locking packets on disk as part of a trace clip effectively reduces the size of the FIFO store available for other ongoing capture jobs.
The exact set of packets that should go into a trace clip is
identified using filters. Filters are described more fully below.
But trace clips typically include at least a time filter to select
packets from a specific time interval, and may include filters to
select just traffic to/from a specific IP address and/or a specific
TCP or UDP port number. Trace clips are accessed in SteelScript via the
methods NetShark.get_clips() and Job4.add_clip().
5.2.3. Extractors¶
Now that we have established how packets are directed into views, we turn to the topic of how information is extracted from packets. The central concept in this section is extractor fields. An extractor field is an individual piece of information. Simple extractor fields are computed from a single packet, typical examples are the packet length in bytes, or the IP source address of an IP packet. More complex extractor fields work on a stream of packets to compute higher level statistics. Examples of more compliated extractor fields are the jitter in a VOIP call or the number of retransmissions on a TCP connection – these statistics can only be derived by looking at the relationship between multiple packets.
Every extractor field has a name of the form extractor.name used
to uniquely identify the field, as well as a human-readable
description that briefly explains what the field is. Extractor is
a high-level category such as generic for fields that apply to any
type of packet, tcp for fields that are specific to TCP, voip
for fields specific to voice-over-IP, etc. The name is a unique
short descriptive name. For example, generic.bits is the length
of a packet in bits, and tcp.src_port is the source port from the
TCP header of a TCP packet. In addition, each field has a fixed type
(e.g., string, integer, IP address, etc.) and always yields values of
that type.
See Extractor Fields for details on using extractor fields from SteelScript.
The next two sections explain how extractor fields are used in packet filters and in views.
5.2.4. Filters¶
Packet filters were mentioned above as part of creating views and
trace clips. As the name suggests, a filter is applied to a packet
source to select some subset of the original packets for view
processing or for inclusion in a trace clip. NetShark implements two
types of filters: native filters and Wireshark filters. A native
filter is simply a test applied to an extractor field. For
example, ip.src="10.1.2.3" matches all packets with source IP
address 10.1.2.3, or generic.bytes >= 100 matches all packets at
least 100 bytes long.
NetShark includes an extensive set of built-in extractor fields for common protocols including IP, TCP, UDP, RTP, HTTP, and others. Many powerful filters can be built using these extractors, but the number of applications that NetShark has built-in extractors for is modest compared to the Wireshark network analyzer. Wireshark has well over 100,000 extractor fields – for cases that are not covered by the shark built-in extractors, a wireshark filter can be used instead. A wireshark filter is specified using the Wireshark Display Filter syntax. Although wireshark filters offer great flexibility, they are implemented by starting a separate wireshark process and sending packets to that process for filtering, so they are significantly slower than native filters and may not be appropriate for applying to very large traces or high-rate live traffic.
See Filters for details on working with filters from SteelScript.
5.2.5. Inside a View¶
We now have all the concepts needed to explain how NetShark views are constructed. A view consists of two major pieces: which packets should be analyzed, and what analysis should be done on those packets. The first piece (which packets should be analyzed) is specified as a packet source and optionally, some filters.
The actual analysis is organized around extractor fields. When a view is created, a list of extractor fields is supplied, each of which is designated as either a key or a value. We begin with value fields. Each value field corresponds to a single output for the view, but of course there can be many packets processed by the view, so the field has an associated calculation that specifies how the individual samples are aggregated into a single statistic. Possible calculations include summing the individual samples, computing the average, or finding the minimum or maximum value.
To make this concrete, consider a simple example of a view with just one field: packet size. If we use the averaging calculation, this view computes the average size of all packets in the input. By choosing a different calculation we could easily compute the minimum or maximum packet size, or the total number of bytes across all packets. This computation is automatically organized into a series of discrete time intervals, at a frequency specified when the view is created. Using our example above, by using the addition operator to aggregate packet sizes, we actually get a time series with the total bandwidth used in each 1 second interval – effectively a view of bandwidth over time. Similarly, with any other extractor fields and aggregation operators, the view output is a time series indicating how the aggregated value field has changed over time.
Returning to key fields, a view is actually organized as a table with a column for each extractor field. If there are no key fields, as in our examples above, the table simply has one default row where all statistics are collected. But when key fields are specified, a tuple is created consisting of the values of all the key columns, and a new row is added to the table for each unique tuple. To make this concrete, one could create a simple view to compute a list of all TCP/IP connections by specifying four key columns: source IP address, source TCP port, destination IP address, and destination TCP port. In this way, a new row would be created in the view table for each unique connection 4-tuple, so reading the contents of the view table yields a list of all connections.
Combining key and value columns, we can merge our examples above by creating a view with key columns for the source and destination addresses and ports and a value column for total packet size. We now get a table of all connections with the total number of bytes sent on each connection. If we sort this table by the total bytes sent, we have a “top talkers” report!