5.4. NetShark, Packet Sources and Views¶

5.4.1. NetShark Objects¶

class steelscript.netshark.core.NetShark(host, port=None, auth=None, force_version=None)¶

The NetShark class is the main interface to interact with a NetShark appliance.

Among other things, it makes it possible to manage views, jobs, files and trace clips, and to query and modify the appliance settings.

__init__(host, port=None, auth=None, force_version=None)¶

Establishes a connection to a NetShark appliance.

Parameters:

host (str) – the name or IP address of the NetShark to connect to port (int) – the TCP port on which the NetShark appliance listens. if this parameter is not specified, the function will try to automatically determine the port. auth – defines the authentication method and credentials to use to access the NetShark. It should be an instance of UserAuth or OAuth. force_version (str) – is the API version to use when communicating. if unspecified, this will use the latest version supported by both this implementation and the NetShark appliance.

See the base Service class for more information about additional functionality supported.

The following methods provide general information about the a NetShark:

• get_serverinfo()
• get_protocol_version()
• get_logininfo()
• get_stats()

The following methods are used to access view. Each of these methods returns a View.

• get_open_views()
• get_open_view_by_handle()
• create_view()
• create_view_from_template()

The following methods are used to access packet sources (e.g., to obtain an object that can be used as an argument to create_view(), etc. The objects they return are described below in the section Packet Source Objects.

• get_interfaces()
• get_interface_by_name()
• get_capture_jobs()
• get_capture_job_by_name()
• create_job()
• get_clips()
• create_clip()
• get_trace_clip_by_description()
• get_files()

The following methods are used to work directly with trace files on the NetShark appliance filesystem:

• get_dir()
• get_file()
• exists()
• create_dir()
• create_multisegment_file()
• create_merged_file()
• upload_trace_file()

The following methods are used to access extractor fields.

• get_extractor_fields()
• find_extractor_field_by_name()
• search_extractor_fields()

The following method is used to create an export from a source:

• create_export()

Complete method descriptions:

create_clip(job, filters, description, locked=True)¶

Create a clip in the NetShark appliance

create_dir(path)¶

Create a new directory. It will trigger an exception if the directory exist.

Returns:	a reference to the new directory

create_export(source, timefilter, filters=None, wait_for_data=False, wait_duration=10)¶

Create a pcap Export object.

Parameters:

source – source to use for packets: Job, View, Interface or TraceClip timefilter – time range of packets to export filters – additional filters wait_for_data – Boolean to indicate if additional retries should be made if the export request initially fails. If True, up to three attempts will be made with an interval of wait_duration. wait_duration – interval of time to wait between export attempts, if wait_for_data is True.

Returns:

Export4

create_job(interface, name, packet_retention_size_limit, packet_retention_packet_limit=None, packet_retention_time_limit=None, bpf_filter=None, snap_length=65525, indexing_size_limit=None, indexing_synced=False, indexing_time_limit=None, start_immediately=True, requested_start_time=None, requested_stop_time=None, stop_rule_size_limit=None, stop_rule_packet_limit=None, stop_rule_time_limit=None)¶

Create a Capture Job on this NetShark, and return it as a a capture job object that can be used to work with the new job.

Parameters:

interface – is an interface object identifying the source of the packets that this job will capture. The method get_interfaces() returns interface objects for all interfaces on this netshark. name – is the name that the job will have after being created, and that will be used to identify the job for successive operations. packet_retention_size_limit – is the maximum size on disk that the job will reach before old packets begin to be discarded. packet_retention_size_limit can be expressed as a number (in which case the unit will be bytes), or as a string. The format of the string can be Number and unit (e.g.”1.3GB”), or percentage of total disk spoace (e.g. “20%”). packet_retention_packet_limit – the maximum number of packets that the job can contain before old packets begin to be discarded. packet_retention_time_limit – a datetime.timedelta object specifying the maximum time interval that the job may contain before old packets begin to be discarded. bpf_filter – is a filter, based on the Wireshark capture filter syntax, that selects which incoming packets should be saved in the job. snap_length – is the portion of each packet that will be written on disk, in bytes. The default value of 65535 ensure that every packet is captured in its entirety. indexing_size_limit – is the maximum size on disk that the index may reach before old index data may be discarded. It is specified in the same format as keep_size. indexing_synced – if True the job’s microflow index will be automatically pruned to cover the same time extension of the captured packets. indexing_time_limit – is a datetime.timedelta object indicating how long index data should be retained. This argument is only meaningful if index_and_capture_synced is False. start_immediately – if True, the Job is started after it has been created. requested_start_time – and requested_stop_time are datetime.datetime objects that, if specified, determine the absolute time when the job will start/stop. stop_rule_size_limit – is specified, the job will stop storing new packets when the given size on disk is reached. It is specified in the same format as keep_size. stop_rule_packet_limit – if specified, the job will stop storing new packets when the given number of packets is reached. stop_rule_time_limit – if specified, it should be a datetime.timedelta object, and the job will stop storing new packets when the given time has elapsed.

create_merged_file(path, files=None)¶

Creates a merged file. ‘path’ is the new file full name and ‘files’ is a File objects list

Returns:	a reference to the new file

create_multisegment_file(path, files=None)¶

Creates a multisegment file. ‘path’ is the new file full name and ‘files’ is a File objects list

Returns:	a reference to the new file

create_view(src, columns, filters=None, start_time=None, end_time=None, name=None, charts=None, sync=True, sampling_time_msec=None)¶

Create a new view on this NetShark.

Parameters:	src – identifies the source of packets to be analyzed. It may be any packet source object. columns – specifies what information is extracted from packets and presented in this view. It should be a list of Key and Value objects filters – an optional list of filters that can be used to limit which packets from the packet source are processed by this view.
Returns:	View4

create_view_from_template(source, template, name=None, sync=True)¶

Create a new view on this NetShark using template.

For 9.5.x and earlier, template should be an XML view document, for later releases, template should be a JSON view description.

Returns:	View4

download_log(path=None, log_type='COMPLETE', case_id=None)¶

Download log archive to local machine into path If path is None a temporary file is created

log_type can be:

• ‘CURRENT’
• ‘PROBE’
• ‘PACKETRECORDER’
• ‘COMPLETE’ (default)

case_id is an integer that represent the case id

exists(path)¶

Check if a path exists, works for files or directories.

Returns:	true if the path exists, false otherwise

find_extractor_field_by_name(fname)¶

Return a specific extractor field given its name

get_capture_job_by_name(jobname, force_refetch=False)¶

Return a CaptureJob object for the capture job jobname

If force_refetch is True, the list of jobs will be re-fetched from the NetShark. Otherwise, the list may be cached.

get_capture_jobs(force_refetch=False)¶

Return a list of CaptureJob objects, corresponding to the capture jobs currently running on the NetShark.

If force_refetch is True, the list of jobs will be re-fetched from the NetShark. Otherwise, the list may be cached.

get_clips(force_refetch=False)¶

Return a list of TraceClip objects, corresponding to the trace clips available on the NetShark.

If force_refetch is True, the list of clips will be re-fetched from the NetShark. Otherwise, the list may be cached.

get_dir(path)¶

Get a directory. It will trigger an exception if the directory does not exist.

Returns:	a reference to the directory

get_extractor_fields()¶

Return a list of all extractor fields available on this netshark

get_file(path)¶

Given a path retrieve the File associated with it

get_files(force_refetch=False)¶

Return a list of TraceFile, MergedFile or Multisegment objects, corresponding to the trace files available on the NetShark.

If force_refetch is True, the list of files will be re-fetched from the NetShark. Otherwise, the list may be cached.

get_interface_by_name(ifname, force_refetch=False)¶

Return an Interface object corresponding to the interface named ifname.

If force_refetch is True, the list of interfaces will be re-fetched from the NetShark. Otherwise, the list may be cached.

get_interfaces(force_refetch=False)¶

Return a list of Interface objects, corresponding to the capture interfaces available on the NetShark.

If force_refetch is True, the list of interfaces will be re-fetched from the NetShark. Otherwise, the list may be cached.

get_logininfo()¶

Return a dictionary with the information useful prior to logging in to the netshark, including the protocol version, banner, and whether or not a login purpose should be requested from the user.

get_open_view_by_handle(handle)¶

Look up the view handle and return its View object

get_open_views()¶

Get a list of View objects, one for each open view on the NetShark appliance.

get_protocol_version()¶

Return the API protocol version used by this netshark.

get_serverinfo()¶

Get the NetShark appliance overall info.

Returns:	a named tuple with the server parameters

get_stats()¶

Get the NetShark appliance storage info

get_trace_clip_by_description(description, force_refetch=False)¶

Return a TraceClip object for the trace clip with the given description.

If force_refetch is True, the list of jobs will be re-fetched from the NetShark. Otherwise, the list may be cached.

Note: Clips don’t have descriptions by default. A description can be added to a clip by right-clicking on it in Pilot.

model¶

Returns the NetShark software model

ping()¶

Ping the probe.

Returns:	True/False depending whether the server is up and running or not

restart()¶

Restart the NetShark appliance. This will issue a system reboot.

restart_probe()¶

Restart the NetShark probe. This will restart the Pilot server only.

search_extractor_fields(string)¶

Search through the extractor fields to find the ones that match the given string in either the field name or the description.

upload_trace_file(path, local_file)¶

Uploads a new trace file. ‘path’ is the complete destination path, ‘local_file’ is the the local file to upload.

Returns:	reference to the new trace file

version¶

Returns the NetShark software version

5.4.2. Packet Source Objects¶

The objects described in this section are used to access packet sources. None of these objects are directly instantiated from external code, they are returned by methods on NetShark or other routines. Any of the objects in this section may be used as the src argument to NetShark.create_view().

There are three basic data sources:

• Capture Jobs
• Interfaces
• Trace Clips

5.4.3. Extractor Fields¶

Extractor Field objects represent individual extractor fields. These objects are returned by NetShark.get_extractor_fields(), NetShark.find_extractor_field_by_name(), and NetShark.search_extractor_fields().

Each extractor field is a python namedtuple with the following fields:

• name: the name of the field, e.g., ip.source_ip
• desc: a brief description of the field
• type: a string describing the type of the field (e.g., integer, string ip address, etc.)

5.4.4. View Objects¶

View objects are returned from NetShark.create_view().

A View object encapsulates everything needed to read data from an existing view on a NetShark. Every view has one or more associated outputs. For example, the standard “Bandwidth over time” view has separate outputs for “bits over time”, “bytes over time”, and “packets over time”. In SteelScript, a View object contains an associated Output object for each output. To read data from a view, you must first locate the appropriate Output object, then use the method Output4.get_data().

class steelscript.netshark.core.View4(shark, handle, config=None, source=None)¶

all_outputs()¶

Return a list of Output objects, one for each output in this view

close()¶

Close this view on the server (which permanently deletes the view plus any associated configuration and output data).

get_data(*args, **kwargs)¶

Returns the data from the output in this view. Shorthand for all_outputs()[0].get_data().

Raises a LookupError if the view has more than one output.

For a full description of the function arguments, refer to the method Output.get_data().

get_iterdata(*args, **kwargs)¶

Returns an iterator for the data from the output in this view. Shorthand for all_outputs()[0].get_iterdata().

Raises a LookupError if the view has more than one output.

For a full description of the function arguments, refer to the method Output.get_iterdata()

get_legend()¶

Returns the legend from the output in this view. Shorthand for all_outputs()[0].get_legend().

Raises a LookupError if the view has more than one output.

get_output(id)¶

Return the Output object corresponding to the output id.

get_progress()¶

For views applied to non-live sources (i.e., to trace clips or trace files), returns an integer between 0 and 100 indicating the percentage of the packet source that has been processed. Output data is not available on the view until this value reaches 100%

get_timeinfo()¶

Return a dictionary object with details about the time range covered by this view. The returned object has the following entries:

• start: the time of the first packet for which data is available
• end: the end time of the last sample
• delta: the sampling time of each sample

This function adds a delta to the end time of the view provided by netshark If you need the timeinfo provided by netshark as it is use _get_timeinfo

is_ready()¶

Returns a boolean indicating whether data can be retrieved from this view or not. If this function returns False, the view data is still being computed, its progress can be followed with the method get_progress().

5.4.5. Export Objects¶

class steelscript.netshark.core.Export4(shark, id, source)¶

Bases: steelscript.netshark.core._interfaces.Export

create(shark, source, timefilter, filters=None, wait_for_data=False, wait_duration=10)¶

Create a new pcap export from the given source.

delete()¶

Delete the export on the shark.

Note that downloading an export automatically deletes the export.

details()¶

Return details about an export.

download(filename, overwrite=False)¶

Download the packets as a pcapfile to a local file.

5.4.6. Output Objects¶

class steelscript.netshark.core.Output4(view, ouid)¶

Bases: steelscript.netshark.core._interfaces.Output

get_data(start=None, end=None, delta=None, aggregated=False, sortby=None, sorttype='descending', fromentry=0, toentry=0)¶

Get the data for this view. This function downloads the whole dataset before returning it, so it’s useful when random access to the view data is necessary.

The arguments have the same meanings as corresponding arguments to get_iterdata(), see its documentation for a full explanation of all arguments and their meanings.

get_iterdata(start=None, end=None, delta=None, aggregated=False, sortby=None, sorttype='descending', fromentry=0, toentry=0)¶

Returns an iterator to the output data. This function is ideal for sequential parsing of the view data, because it downloads the dataset incrementally as it is accessed.

start and end are datetime.datetime objects representing the earliest and latest packets that should be considered. If start and end are unspecified, the start/end of the underlying packet source are used.

delta is a datetime.timedelta object that can be used to override the default data aggregation interval. If this parameter is unspecified, the underlying view sample interval (which defaults to 1 second) is used. If this parameter is specified, it must be an even multiple of the underlying view sample interval.

If aggregated is True, the parameter delta is automatically computed to be the full extent of this request (i.e., the difference between the effective start and end times). This is useful if you do not care about timeseries data (e.g., if the data from this view is to be plotted in a single chart that has no time component).

The sortby parameter is one of the fields of the output (x1, x2 ...)

The sorttype can be: * ascending: the output is sorted from smallest to largest * descending: the output is sorted largest to smallest

The fromentry parameter represent the first sorted item we want to appear in the output. 0 means from the first one.

The toentry parameter represent the last sorted item we want to appear in the output. 0 means all of them.

get_legend()¶

Return the legend for this output. The legend consists of an ordered list of entries, one for each column of data in this output. Each entry is a dictionary object with the following entries:

• name: A short name for the field
• description: A slightly longer, more descriptive name for the field
• field: the name of the extractor field that produces this column
• calculation: how data from multiple packets in a sample is aggregated (e.g., “SUM”, “AVG”, etc.)

The following parameters are intended for internal netshark use:

• type
• id
• base
• dimension

5.4.7. Filters¶

Filters are used to reduce the traffic that is fed to NetShark views, and are basic tools when doing data analysis with a NetShark Appliance. NetShark supports 4 main classes of filters: Time filters, the native NetShark filters, BPF filters (also known as Wireshark capture filters) and Wireshark display filters.

5.4.8. Utilities¶

This is a set of utility functions for working with views.

Utilities for writing view data

steelscript.netshark.core.viewutils.print_data(legend, stream, timeformat='%Y/%m/%d %H:%M:%S.%f', include_sample_times=True, widths=None, limit=None, line_prefix='')¶

Print the data of a given view output to stdout.

widths is an optional list of integers, specifying how many characters wide each column should be. If it is not specified, reasonable defaults are chosen.

If limit is specified, only the first limit rows are printed.

line_prefix is a string that is printed at the start of every line.

steelscript.netshark.core.viewutils.write_csv(filename, legend, stream, include_column_names=True, include_sample_times=True)¶

Saves the data of a view output to a comma separated values (csv) file.

legend is an output legend, typically just the result of output.get_legend()

stream is a series of data samples, typically the result of output.get_data() or output.get_iterdata()

If include_column_names is True, the first line in the file will be a summary row indicating the fileds that the file contains.

If include_sample_times is True, the first column will be a timestamp.

class steelscript.netshark.core.viewutils.OutputMixer¶

Helper class that blends multiple data streams (ie from View.get_data) into a single combined stream. For example, given a “Bandwidth Over Time” view with separate outputs for bytes and packets, this class can be used to create a single output stream with bytes and packets columns.

Mixing is only supported on simple time-based views that do not include any keys (e.g., bandwidth over time, etc.)

See examples/netshark/readview.py for typical usage.

Mixing multiple view outputs

add_source(src, prefix=None)¶

Add new source to mixer

src is time-based view object

get_iterdata(*args, **kwargs)¶

Return a generator for the combined stream of outputs from each source object

get_legend()¶

Return the legend for each of the source objects

class sourceobj(output, prefix, offset)¶

offset¶

Alias for field number 2

output¶

Alias for field number 0

prefix¶

Alias for field number 1