11.9. Tables¶

11.9.1.1. Sample Table Definition¶

The following table is taken from the Wave sample plugin:

import steelscript.wave.appfwk.datasources.wave_source as wave

# Define a waves table with 3 separate waves.
table = wave.WaveTable.create(
    name='wave-table', duration='15min', resolution='1s', beta=4)

# Add columns for time and the related waves
table.add_column('time', 'Time', datatype=Column.DATATYPE_TIME, iskey=True)
table.add_column('sin1', 'Sine Wave 1', func='sin', period='5min', alpha=3)
table.add_column('sin2', 'Sine Wave 2', func='sin', period='8min', alpha=5)
table.add_column('cos',  'Cosine Wave', func='cos', period='3min', alpha=2.5)

11.9.2.1. Adding Columns¶

Columns define the keys and values of the data set that this table will collect. They are added to a table using DatasourceTable.add_column().

When a query is run, the data source associated with a table inspects the list of key and value columns and generates a data table matching the requested column set.

DatasourceTable.add_column(name, label=None, sortasc=False, sortdesc=False, **kwargs)¶

Add a column to this table.

Parameters:	name (str) – Unique identifier for this column label (str) – Display string, defaults to name sortasc, sortdesc (bool) – Sort table based on this columns data

Standard keyword arguments:

Parameters:

iskey (bool) – Set True for key columns datatype (enum) – type of data his column contains, defaults to ‘float’: float integer time string html date units (enum) – Units for data in this column, defaults to none: none - no units s - seconds ms - milliseconds B - bytes B/s - bytes per second b - bits b/s - bits per second pct - percentage position (int) – Display position relative to other columns, automatically computed by default synthetic (bool) – Set True to compute this columns value according to compute_expression compute_expression (str) – Computation expression for syntetic columns compute_post_resample (bool) – If true, compute this synthetic column after resampling (time series only) resample_operation (str) – Operation to use on this column to aggregate multiple rows during resampling, defaults to sum

11.9.2.2. Synthetic Columns¶

In addition to columnar data generated by a data source, additional synthetic columns may be attached to a table. Synthetic columns provide an easy way to perform computations on other data columns in the same table.

This is best explained by an example based on the WaveTable above:

table.add_column('sin1', 'Sine Wave 1', func='sin', period='5min', alpha=3)
table.add_column('sin1-doubled', synthetic=True, compute_expression='2*{sin1}')

The first is a normal column whose data will be provided by the wave data source. The second column is a synthetic column that is simply the ‘sin1’ column multiplied by 2.

The compute_expression column keyword defines the operation to perform:

• Other column values are referenced using the syntax {name}, where name is the name assigned to another column in the same table. Any number of other columns may be referenced
• Standard mathematical operators may be used: +, -, *, /, and others.
• {name} is actually a Python Pandas Series object, and thus functions on series data can be leveraged either by methods on the series object or by using the full package path:
  • {name}.quantile(0.95) will compute the 95th percentile for the data in column {name}
  • pandas.stats.moments.ewma({name}, span=20) will compute the EWMA (exponential weighted moving average) of the {name} column using a span of 20 data points.

For more advanced analysis techniques, see Analysis Tables.

11.9.2.3. Resampling Time Series Tables¶

When working with time series data, a common operation is to resample that data:

• Incoming data is at 1 minute resolution, but the output needs to be at 5 minute resolution
• Incoming data has erratic non-normalized timestamps, the output should be graphed at steady 1 minute resolution

The application framework will automatically resample timeseries data when the resample=True at creation. In addition, there must be a criteria field named either resample_resolution or just resolution, which sets the target resample interval.

When resampling, data from multiple rows must be aggregated (each row represents a timestamp or time interval). The aggregation operation is different for different types of data:

• Counted metrics such as “total bytes” involves computing the “sum” of all rows covered by the new interval.
• Peak metrics such as “peak network RTT” require computing the “max” of all metrics.

Each data column may be set up with a different resample_operation based. The default is sum, but this is not always appropriate for all data types.

Note that when using synthetic columns as describe above, you can choose to compute the synthetic columns before or after resampling by setting compute_post_resample.