5.4. NetShark, Packet Sources and Views

The NetShark package offers a set of interfaces to control and work with a SteelCentral NetShark appliance. The functionality in the module includes:

5.4.1. NetShark Objects

class steelscript.netshark.core.NetShark(host, port=None, auth=None, force_version=None)

The NetShark class is the main interface to interact with a NetShark appliance.

Among other things, it makes it possible to manage views, jobs, files and trace clips, and to query and modify the appliance settings.

__init__(host, port=None, auth=None, force_version=None)

Establishes a connection to a NetShark appliance.

Parameters:
  • host (str) – the name or IP address of the NetShark to connect to
  • port (int) – the TCP port on which the NetShark appliance listens. if this parameter is not specified, the function will try to automatically determine the port.
  • auth – defines the authentication method and credentials to use to access the NetShark. It should be an instance of UserAuth or OAuth.
  • force_version (str) – is the API version to use when communicating. if unspecified, this will use the latest version supported by both this implementation and the NetShark appliance.

See the base Service class for more information about additional functionality supported.

The following methods provide general information about the a NetShark:

The following methods are used to access view. Each of these methods returns a View.

The following methods are used to access packet sources (e.g., to obtain an object that can be used as an argument to create_view(), etc. The objects they return are described below in the section Packet Source Objects.

The following methods are used to work directly with trace files on the NetShark appliance filesystem:

The following methods are used to access extractor fields.

The following method is used to create an export from a source:

Complete method descriptions:

create_clip(job, filters, description, locked=True)

Create a clip in the NetShark appliance

create_dir(path)

Create a new directory. It will trigger an exception if the directory exist.

Returns:a reference to the new directory
create_export(source, timefilter, filters=None, wait_for_data=False, wait_duration=10)

Create a pcap Export object.

Parameters:
  • source – source to use for packets: Job, View, Interface or TraceClip
  • timefilter – time range of packets to export
  • filters – additional filters
  • wait_for_data – Boolean to indicate if additional retries should be made if the export request initially fails. If True, up to three attempts will be made with an interval of wait_duration.
  • wait_duration – interval of time to wait between export attempts, if wait_for_data is True.
Returns:

Export4

create_job(interface, name, packet_retention_size_limit, packet_retention_packet_limit=None, packet_retention_time_limit=None, bpf_filter=None, snap_length=65525, indexing_size_limit=None, indexing_synced=False, indexing_time_limit=None, start_immediately=True, requested_start_time=None, requested_stop_time=None, stop_rule_size_limit=None, stop_rule_packet_limit=None, stop_rule_time_limit=None)

Create a Capture Job on this NetShark, and return it as a a capture job object that can be used to work with the new job.

Parameters:
  • interface – is an interface object identifying the source of the packets that this job will capture. The method get_interfaces() returns interface objects for all interfaces on this netshark.
  • name – is the name that the job will have after being created, and that will be used to identify the job for successive operations.
  • packet_retention_size_limit – is the maximum size on disk that the job will reach before old packets begin to be discarded. packet_retention_size_limit can be expressed as a number (in which case the unit will be bytes), or as a string. The format of the string can be Number and unit (e.g.”1.3GB”), or percentage of total disk spoace (e.g. “20%”).
  • packet_retention_packet_limit – the maximum number of packets that the job can contain before old packets begin to be discarded.
  • packet_retention_time_limit – a datetime.timedelta object specifying the maximum time interval that the job may contain before old packets begin to be discarded.
  • bpf_filter – is a filter, based on the Wireshark capture filter syntax, that selects which incoming packets should be saved in the job.
  • snap_length – is the portion of each packet that will be written on disk, in bytes. The default value of 65535 ensure that every packet is captured in its entirety.
  • indexing_size_limit – is the maximum size on disk that the index may reach before old index data may be discarded. It is specified in the same format as keep_size.
  • indexing_synced – if True the job’s microflow index will be automatically pruned to cover the same time extension of the captured packets.
  • indexing_time_limit – is a datetime.timedelta object indicating how long index data should be retained. This argument is only meaningful if index_and_capture_synced is False.
  • start_immediately – if True, the Job is started after it has been created.
  • requested_start_time – and requested_stop_time are datetime.datetime objects that, if specified, determine the absolute time when the job will start/stop.
  • stop_rule_size_limit – is specified, the job will stop storing new packets when the given size on disk is reached. It is specified in the same format as keep_size.
  • stop_rule_packet_limit – if specified, the job will stop storing new packets when the given number of packets is reached.
  • stop_rule_time_limit – if specified, it should be a datetime.timedelta object, and the job will stop storing new packets when the given time has elapsed.
create_merged_file(path, files=None)

Creates a merged file. ‘path’ is the new file full name and ‘files’ is a File objects list

Returns:a reference to the new file
create_multisegment_file(path, files=None)

Creates a multisegment file. ‘path’ is the new file full name and ‘files’ is a File objects list

Returns:a reference to the new file
create_view(src, columns, filters=None, start_time=None, end_time=None, name=None, charts=None, sync=True, sampling_time_msec=None)

Create a new view on this NetShark.

Parameters:
  • src – identifies the source of packets to be analyzed. It may be any packet source object.
  • columns – specifies what information is extracted from packets and presented in this view. It should be a list of Key and Value objects
  • filters – an optional list of filters that can be used to limit which packets from the packet source are processed by this view.
Returns:

View4

create_view_from_template(source, template, name=None, sync=True)

Create a new view on this NetShark using template.

For 9.5.x and earlier, template should be an XML view document, for later releases, template should be a JSON view description.

Returns:View4
download_log(path=None, log_type='COMPLETE', case_id=None)

Download log archive to local machine into path If path is None a temporary file is created

log_type can be:

  • ‘CURRENT’
  • ‘PROBE’
  • ‘PACKETRECORDER’
  • ‘COMPLETE’ (default)

case_id is an integer that represent the case id

exists(path)

Check if a path exists, works for files or directories.

Returns:true if the path exists, false otherwise
find_extractor_field_by_name(fname)

Return a specific extractor field given its name

get_capture_job_by_name(jobname, force_refetch=False)

Return a CaptureJob object for the capture job jobname

If force_refetch is True, the list of jobs will be re-fetched from the NetShark. Otherwise, the list may be cached.

get_capture_jobs(force_refetch=False)

Return a list of CaptureJob objects, corresponding to the capture jobs currently running on the NetShark.

If force_refetch is True, the list of jobs will be re-fetched from the NetShark. Otherwise, the list may be cached.

get_clips(force_refetch=False)

Return a list of TraceClip objects, corresponding to the trace clips available on the NetShark.

If force_refetch is True, the list of clips will be re-fetched from the NetShark. Otherwise, the list may be cached.

get_dir(path)

Get a directory. It will trigger an exception if the directory does not exist.

Returns:a reference to the directory
get_extractor_fields()

Return a list of all extractor fields available on this netshark

get_file(path)

Given a path retrieve the File associated with it

get_files(force_refetch=False)

Return a list of TraceFile, MergedFile or Multisegment objects, corresponding to the trace files available on the NetShark.

If force_refetch is True, the list of files will be re-fetched from the NetShark. Otherwise, the list may be cached.

get_interface_by_name(ifname, force_refetch=False)

Return an Interface object corresponding to the interface named ifname.

If force_refetch is True, the list of interfaces will be re-fetched from the NetShark. Otherwise, the list may be cached.

get_interfaces(force_refetch=False)

Return a list of Interface objects, corresponding to the capture interfaces available on the NetShark.

If force_refetch is True, the list of interfaces will be re-fetched from the NetShark. Otherwise, the list may be cached.

get_logininfo()

Return a dictionary with the information useful prior to logging in to the netshark, including the protocol version, banner, and whether or not a login purpose should be requested from the user.

get_open_view_by_handle(handle)

Look up the view handle and return its View object

get_open_views()

Get a list of View objects, one for each open view on the NetShark appliance.

get_protocol_version()

Return the API protocol version used by this netshark.

get_serverinfo()

Get the NetShark appliance overall info.

Returns:a named tuple with the server parameters
get_stats()

Get the NetShark appliance storage info

get_trace_clip_by_description(description, force_refetch=False)

Return a TraceClip object for the trace clip with the given description.

If force_refetch is True, the list of jobs will be re-fetched from the NetShark. Otherwise, the list may be cached.

Note: Clips don’t have descriptions by default. A description can be added to a clip by right-clicking on it in Pilot.

model

Returns the NetShark software model

ping()

Ping the probe.

Returns:True/False depending whether the server is up and running or not
restart()

Restart the NetShark appliance. This will issue a system reboot.

restart_probe()

Restart the NetShark probe. This will restart the Pilot server only.

search_extractor_fields(string)

Search through the extractor fields to find the ones that match the given string in either the field name or the description.

upload_trace_file(path, local_file)

Uploads a new trace file. ‘path’ is the complete destination path, ‘local_file’ is the the local file to upload.

Returns:reference to the new trace file
version

Returns the NetShark software version

5.4.2. Packet Source Objects

The objects described in this section are used to access packet sources. None of these objects are directly instantiated from external code, they are returned by methods on NetShark or other routines. Any of the objects in this section may be used as the src argument to NetShark.create_view().

There are three basic data sources:

  • Capture Jobs
  • Interfaces
  • Trace Clips

5.4.2.1. Capture Job Objects

Capture job objects are used to work with capture jobs. These objects are not instantiated directly but are returned from NetShark.get_capture_jobs() and NetShark.get_capture_job_by_name().

Note

See Job4 if connecting to a NetShark using API 4.0

class steelscript.netshark.core.Job5(shark, data)

Bases: steelscript.netshark.core._source4.Job4

Capture job objects have the following properties:

The following methods access information about a job:

The following methods are useful for controlling a capture job:

The following methods can be used to create and delete jobs, though create() does the same thing as NetShark.create_clip().

Finally, these methods are useful for creating trace clips and for downloading raw packets from a capture job.

Complete method and property descriptions:

add_clip(filters, description, locked=True)

Create a new trace clip under this job.

filters limit which packets from the clip should go into this clip. Since all packets in the new clip will be kept on disk if the clip is locked, this field typically includes time filters to limit the clip to a fixed time interval.

description is a name for the clip, it is shown next to the clip in the Pilot user interface.

If locked is True, packets in the new clip will not be deleted from disk as long as the clip exists. Note that locking packets in trace clips necessarily reduces the amount of disk capacity for existing capture jobs.

Returns a trace clip object representing the new clip.

clear(restart=False)

Clear data in the NetShark appliance

create(shark, interface, name, packet_retention_size_limit, packet_retention_packet_limit=None, packet_retention_time_limit=None, bpf_filter=None, snap_length=65525, indexing_size_limit=None, indexing_synced=False, indexing_time_limit=None, start_immediately=True, requested_start_time=None, requested_stop_time=None, stop_rule_size_limit=None, stop_rule_packet_limit=None, stop_rule_time_limit=None)

Create a new capture job

delete()

Delete job from the NetShark appliance

download(path=None)

Download the Job packets to a path. If path is None packets will be exported to a temporary file. A file object that contains the packets is returned.

dpi_enabled
get(shark, id, name=None)
get_all(shark)

Get the complete list of capture jobs on given a NetShark.

netshark is an steelscript.netshark.core.netshark.NetShark object

Returns a list of CaptureJob objects

size_on_disk, keep_size, packet_start_time,
packet_end_time,
get_index_info()

Return statistics about the capture job index.

get_state()

Return the state of the job (e.g. RUNNING, STOPPED)

get_stats()

Return statistics about the capture job.

get_status()

Return status information about the capture job.

handle

The internal capture job handle. The handle is sometimes required for advanced operations on the job.

interface

an Interface object, representing the interface used as a packet source for this job.

is_live()

Can be used to determine if a source object is live or offline. The result is always False for a CaptureJob.

name

The capture job name

packet_end_time
packet_start_time
save()
size_limit

The capture job maximum size, corresponding to the one shown by the NetShark UI shows.

size_on_disk

The capture job actual size, corresponding to the one shown by the NetShark UI shows.

source_options
source_path
start()

Start a job in the NetShark appliance

stop()

Stop a job in the NetShark appliance

update(data)

Update the data of the current object with new data from the server

5.4.2.2. Interface Objects

Note

See Interface4 if connecting to a NetShark using API 4.0

class steelscript.netshark.core.Interface5(shark, data)

Bases: steelscript.netshark.core._source4.Interface4

description

The interface description.

get(shark, id, name=None)
get_all(shark)
is_live()

Can be used to determine if a source object is live or offline. The result is always True for an Interface4.

name

The interface device name.

save()
source_options
source_path
update()

Update current object with the values from the server

5.4.2.3. Trace Clip Objects

Note

See section Trace Clip Objects (v4) for details.

5.4.3. Extractor Fields

Extractor Field objects represent individual extractor fields. These objects are returned by NetShark.get_extractor_fields(), NetShark.find_extractor_field_by_name(), and NetShark.search_extractor_fields().

Each extractor field is a python namedtuple with the following fields:

  • name: the name of the field, e.g., ip.source_ip
  • desc: a brief description of the field
  • type: a string describing the type of the field (e.g., integer, string ip address, etc.)

5.4.4. View Objects

View objects are returned from NetShark.create_view().

A View object encapsulates everything needed to read data from an existing view on a NetShark. Every view has one or more associated outputs. For example, the standard “Bandwidth over time” view has separate outputs for “bits over time”, “bytes over time”, and “packets over time”. In SteelScript, a View object contains an associated Output object for each output. To read data from a view, you must first locate the appropriate Output object, then use the method Output4.get_data().

class steelscript.netshark.core.View4(shark, handle, config=None, source=None)
all_outputs()

Return a list of Output objects, one for each output in this view

close()

Close this view on the server (which permanently deletes the view plus any associated configuration and output data).

get_data(*args, **kwargs)

Returns the data from the output in this view. Shorthand for all_outputs()[0].get_data().

Raises a LookupError if the view has more than one output.

For a full description of the function arguments, refer to the method Output.get_data().

get_iterdata(*args, **kwargs)

Returns an iterator for the data from the output in this view. Shorthand for all_outputs()[0].get_iterdata().

Raises a LookupError if the view has more than one output.

For a full description of the function arguments, refer to the method Output.get_iterdata()

get_legend()

Returns the legend from the output in this view. Shorthand for all_outputs()[0].get_legend().

Raises a LookupError if the view has more than one output.

get_output(id)

Return the Output object corresponding to the output id.

get_progress()

For views applied to non-live sources (i.e., to trace clips or trace files), returns an integer between 0 and 100 indicating the percentage of the packet source that has been processed. Output data is not available on the view until this value reaches 100%

get_timeinfo()

Return a dictionary object with details about the time range covered by this view. The returned object has the following entries:

  • start: the time of the first packet for which data is available
  • end: the end time of the last sample
  • delta: the sampling time of each sample

This function adds a delta to the end time of the view provided by netshark If you need the timeinfo provided by netshark as it is use _get_timeinfo

is_ready()

Returns a boolean indicating whether data can be retrieved from this view or not. If this function returns False, the view data is still being computed, its progress can be followed with the method get_progress().

5.4.5. Export Objects

class steelscript.netshark.core.Export4(shark, id, source)

Bases: steelscript.netshark.core._interfaces.Export

create(shark, source, timefilter, filters=None, wait_for_data=False, wait_duration=10)

Create a new pcap export from the given source.

delete()

Delete the export on the shark.

Note that downloading an export automatically deletes the export.

details()

Return details about an export.

download(filename, overwrite=False)

Download the packets as a pcapfile to a local file.

5.4.6. Output Objects

class steelscript.netshark.core.Output4(view, ouid)

Bases: steelscript.netshark.core._interfaces.Output

get_data(start=None, end=None, delta=None, aggregated=False, sortby=None, sorttype='descending', fromentry=0, toentry=0)

Get the data for this view. This function downloads the whole dataset before returning it, so it’s useful when random access to the view data is necessary.

The arguments have the same meanings as corresponding arguments to get_iterdata(), see its documentation for a full explanation of all arguments and their meanings.

get_iterdata(start=None, end=None, delta=None, aggregated=False, sortby=None, sorttype='descending', fromentry=0, toentry=0)

Returns an iterator to the output data. This function is ideal for sequential parsing of the view data, because it downloads the dataset incrementally as it is accessed.

start and end are datetime.datetime objects representing the earliest and latest packets that should be considered. If start and end are unspecified, the start/end of the underlying packet source are used.

delta is a datetime.timedelta object that can be used to override the default data aggregation interval. If this parameter is unspecified, the underlying view sample interval (which defaults to 1 second) is used. If this parameter is specified, it must be an even multiple of the underlying view sample interval.

If aggregated is True, the parameter delta is automatically computed to be the full extent of this request (i.e., the difference between the effective start and end times). This is useful if you do not care about timeseries data (e.g., if the data from this view is to be plotted in a single chart that has no time component).

The sortby parameter is one of the fields of the output (x1, x2 ...)

The sorttype can be: * ascending: the output is sorted from smallest to largest * descending: the output is sorted largest to smallest

The fromentry parameter represent the first sorted item we want to appear in the output. 0 means from the first one.

The toentry parameter represent the last sorted item we want to appear in the output. 0 means all of them.

get_legend()

Return the legend for this output. The legend consists of an ordered list of entries, one for each column of data in this output. Each entry is a dictionary object with the following entries:

  • name: A short name for the field
  • description: A slightly longer, more descriptive name for the field
  • field: the name of the extractor field that produces this column
  • calculation: how data from multiple packets in a sample is aggregated (e.g., “SUM”, “AVG”, etc.)

The following parameters are intended for internal netshark use:

  • type
  • id
  • base
  • dimension

5.4.7. Filters

Filters are used to reduce the traffic that is fed to NetShark views, and are basic tools when doing data analysis with a NetShark Appliance. NetShark supports 4 main classes of filters: Time filters, the native NetShark filters, BPF filters (also known as Wireshark capture filters) and Wireshark display filters.

5.4.7.1. TimeFilter Objects

class steelscript.netshark.core.filters.TimeFilter(start, end)
__init__(start, end)

Create a TimeFilter representing a time range.

Parameters:
  • start (datetime) – Start time
  • end (datetime) – End time
classmethod parse_range(string)

Creata a TimeFilter based on human readable string.

Parameters:string (str) – time string

The string parameter can be any time range string such as:

  • 12:00 PM to 1:00 PM
  • last 2 weeks

5.4.7.2. NetSharkFilter Objects

class steelscript.netshark.core.filters.NetSharkFilter(string)
__init__(string)

Create a filter in native NetShark syntax.

Parameters:string (str) – filter string

5.4.7.3. BpfFilter Objects

class steelscript.netshark.core.filters.BpfFilter(string)
__init__(string)

Create a filter in BPF syntax.

Parameters:string (str) – filter string

5.4.7.4. WiresharkDisplayFilter Objects

class steelscript.netshark.core.filters.WiresharkDisplayFilter(string)
__init__(string)

Create a filter in Wireshark display syntax..

Parameters:string (str) – filter string

5.4.8. Utilities

This is a set of utility functions for working with views.

Utilities for writing view data

steelscript.netshark.core.viewutils.print_data(legend, stream, timeformat='%Y/%m/%d %H:%M:%S.%f', include_sample_times=True, widths=None, limit=None, line_prefix='')

Print the data of a given view output to stdout.

widths is an optional list of integers, specifying how many characters wide each column should be. If it is not specified, reasonable defaults are chosen.

If limit is specified, only the first limit rows are printed.

line_prefix is a string that is printed at the start of every line.

steelscript.netshark.core.viewutils.write_csv(filename, legend, stream, include_column_names=True, include_sample_times=True)

Saves the data of a view output to a comma separated values (csv) file.

legend is an output legend, typically just the result of output.get_legend()

stream is a series of data samples, typically the result of output.get_data() or output.get_iterdata()

If include_column_names is True, the first line in the file will be a summary row indicating the fileds that the file contains.

If include_sample_times is True, the first column will be a timestamp.

class steelscript.netshark.core.viewutils.OutputMixer

Helper class that blends multiple data streams (ie from View.get_data) into a single combined stream. For example, given a “Bandwidth Over Time” view with separate outputs for bytes and packets, this class can be used to create a single output stream with bytes and packets columns.

Mixing is only supported on simple time-based views that do not include any keys (e.g., bandwidth over time, etc.)

See examples/netshark/readview.py for typical usage.

Mixing multiple view outputs

add_source(src, prefix=None)

Add new source to mixer

src is time-based view object

get_iterdata(*args, **kwargs)

Return a generator for the combined stream of outputs from each source object

get_legend()

Return the legend for each of the source objects

class sourceobj(output, prefix, offset)
offset

Alias for field number 2

output

Alias for field number 0

prefix

Alias for field number 1