Viewing Diagnostic Reports and Logs : Viewing Appliance TCP Dumps
  
Viewing Appliance TCP Dumps
You can create, download, and upload TCP capture files in the TCP Dumps page.
Capture files contain summary information for every Internet packet received or transmitted on the interface to help diagnose problems in the system.
RiOS provides an easy way to create and retrieve multiple capture files from the SCC. You can create capture files from multiple interfaces at the same time, limit the size of the capture file, and schedule a specific date and time to create a capture file. Scheduling and limiting a capture file by time or size allows unattended captures.
RiOS 7.0 and later supports remote capture analysis using the SteelCentral Packet Analyzer on capture files created and stored on the SteelHead without transferring the entire packet capture across the network. You do not need to transfer full packets until you need them.
Note: You cannot upload a capture file to the SCC using Packet Analyzer.
The top of the TCP Dumps page displays a list of existing capture files and the bottom of the page displays controls to create a capture file. The bottom of the page also includes the capture files that are currently running, and controls to create a trigger that stops a capture when a specific event occurs. The Running Capture Name list includes captures running at a particular time. It includes captures started manually and also any captures that were scheduled previously and are now running.
Note: The SCC automatically uploads the tcpdumps to itself (then zips them up) once the capture is finished. You may want to ensure that the tcpdumps do not saturate their WAN links by either minimizing the size of the captures or employing QoS to rate limit the transfers.
To add or remove stored TCP trace dumps
1. Choose Diagnostics > Appliance Logs: TCP Dumps to display the TCP Dumps page.
Figure: TCP Dumps Report
2. Complete the configuration as described in this table.
Control
Description
Add a New TCP Dump
Displays the controls for creating a TCP trace dump.
Capture Name
Specify the name of the capture file. Use a unique filename to prevent overwriting an existing TCP dump. The default filename uses this format:
<hostname>_<interface>_<time-stamp>.cap
Where <hostname> is the hostname of the SCC, <interface> is the name of the interface selected for the trace (for example, lan0_0, wan0_0), and <time-stamp> is in the YYYY-MM-DD-HH-MM-SS format.
If this trace dump relates to an open Riverbed Support case, specify the capture filename case_<number> where <number> is your Riverbed Support case number: for example, case_12345.
Note: The .cap file extension is not included with the filename when it appears in the capture queue.
Appliances
Select an appliance from the list.
Endpoints
Specify IP addresses and port numbers to capture packets between them:
IPs - Specify IP addresses of endpoints on one side. Separate multiple IP addresses using commas. You can enter IPv6 addresses separated by commas. The default setting is all IP addresses.
Ports - Specify ports on one side. Separate multiple ports using commas. The default setting is all ports.
—and—
IPs - Specify IP addresses of endpoints on the other side. Separate multiple IP addresses using commas. You can enter IPv6 addresses separated by commas. The default setting is all IP addresses.
Ports - Specify ports on the other side. Separate multiple ports using commas. The default setting is all ports.
To capture traffic flowing in only one direction or to enter a custom command, use the CLI tcpdump command. For details, see the Riverbed Command-Line Interface Reference Manual.
Capture Interfaces
Captures the TCP trace dump on the selected interface(s). You can select all interfaces or a base, in-path, or RSP interface. The default setting is none. You must specify a capture interface.
If you select several interfaces at a time, the data is automatically placed into separate capture files.
When path selection is enabled, Riverbed recommends that you collect packet traces on all LAN and WAN interfaces.
Capture Parameters
These parameters let you capture information about dot1q VLAN traffic. You can match traffic based on VLAN-tagged or untagged packets, or both. You can also filter by port number or host IP address and include or exclude ARP packets. Select one of these parameters for capturing VLAN packets:
•  Capture Untagged Traffic Only - Select this option for the following captures:
–  All untagged VLAN traffic.
–  Untagged 7850 traffic and ARP packets. You must also specify arp in the custom flags field on this page.
–  Only untagged ARP packets. You must also specify arp in the custom flags field on this page.
•  Capture VLAN-Tagged Traffic Only - Select this option for the following captures:
–  Only VLAN-tagged traffic.
–  VLAN-tagged packets with host 10.11.0.6 traffic and ARP packets. You must also specify 10.11.0.6 in the IPs field, and specify or arp in the custom flags field on this page.
–  VLAN-tagged ARP packets only. You must also specify arp in the custom flags field on this page.
•  Capture both VLAN and Untagged Traffic - Select this option for the following captures:
–  All VLAN traffic.
–  Both tagged and untagged 7850 traffic and ARP packets. You must also specify the following in the custom flags field on this page:
port 7850 or arp
—or—
vlan and port 7850 or arp
–  Both tagged and untagged 7850 traffic only. You must also specify 7850 in one of the port fields on this page.
–  Both tagged and untagged ARP packets. You must also specify the following in the custom flags field on this page:
arp
—or—
vlan and arp
Capture Duration (Seconds)
Specify how long the capture runs, in seconds. The default value is 30. Specify 0 or continuous to initiate a continuous trace. Continuous traces are not permitted from the SCC, so the value must be at least 1.
Maximum Capture Size (MB)
Specify the maximum capture file size, in megabytes. The default value is 100 MB. Riverbed recommends a maximum capture file size of 1024 MB (1 GB).
Riverbed recommends a maximum capture file size of 1024 MB (1 GB).
Buffer Size (KB)
Optionally, specify the maximum amount of data, in kilobytes, allowed to queue up while awaiting processing by the TCP trace dump. The default value is 154 KB.
Snap Length
Optionally, specify the snap length value for the capture file, which equals the number of bytes captured for each packet. Having a snap length smaller than the maximum packet size on the network enables you to store more packets, but you might not be able to inspect the full packet content. Specify 0 for a full packet capture (recommended for CIFS, MAPI, and SSL captures). The default value is 1518 bytes.
Number of Files to Rotate
Specify how many TCP trace dump files to rotate. The default value is 5. The maximum value is a 32-bit integer.
Specify how many capture files to keep for each interface before overwriting the oldest file. To stop file rotation, you can specify 0; however, Riverbed recommends rotating files, because stopping the rotation can fill the disk partition.
This limits the number of files created to the specified number, and begins overwriting files from the beginning, thus creating a rotating buffer.
Custom Flags
Specify custom flags as additional statements within the filter expression. Custom flags are added to the end of the expression created from the Endpoints fields and the Capture Parameters radio buttons (pertaining to VLANs).
If you require an “and” statement between the expression created from other fields and the expression that you are entering in the custom flags field, you must include the “and” statement at the start of the custom flags field.
Do not use host, src, or dst statements in the custom flags field. Although it is possible in trivial cases to get these to start without a syntax error, they do not capture GRE-encapsulated packets that some modes of SteelHead communications use, such as WCCP deployments or Interceptor connection-setup traffic. Riverbed recommends using bidirectional filters by specifying endpoints.
For complete control of your filter expression, use the CLI tcpdump command. For details, see the Riverbed Command-Line Interface Reference Manual.
For examples, see Custom Flag Use Examples.
Schedule Dump
Schedules the trace dump to run at a later date.
•  Start Date - Specify a date to initiate the trace dump in this format: YYYY/MM/DD.
•  Start Time - Specify a time to initiate the trace dump in this format: HH:MM:SS.
Add
Adds the TCP trace dump to the capture queue.
Remove Selected
Under Stored TCP Dumps, select the TCP Dump check box and click Remove Selected.
Troubleshooting
Control
Description
Add a New TCP Dump
Displays the controls for creating a TCP trace dump.
Capture Name
Specify the name of the capture file. The default filename uses this format:
<hostname>_<interface>_<time-stamp>.cap
Where <hostname> is the hostname of the SCC, <interface> is the name of the interface selected for the trace (for example, lan0_0, wan0_0), and <time-stamp> is in the YYYY-MM-DD-HH-MM-SS format.
If this trace dump relates to an open Riverbed Support case, specify the capture filename case_<number> where <number> is your Riverbed Support case number: for example, case_12345.
Note: The .cap file extension is not included with the filename when it appears in the capture queue.
Capture Traffic Between
IPs - Specify the source IP addresses. Separate multiple IP addresses with a comma to include all addresses bidirectionally. The default setting is all IP addresses.
Ports - Specify the source ports. Separate multiple ports with a comma. The default setting is all ports.
and:
IPs - Specify the destination IP addresses. Separate multiple IP addresses with a comma to include all addresses bidirectionally. The default setting is all IP addresses.
Ports - Specify the destination ports. Separate multiple ports with a comma. The default setting is all ports.
Capture Interfaces
Captures the TCP trace dump on the selected interface(s). You can select all interfaces or a physical, MIP, SCA, VSP, or miscellaneous interface. The default setting is none. You must specify a capture interface.
If you select several interfaces at a time, the data is automatically placed into separate capture files.
Capture Duration (Seconds)
Specify how long the capture runs, in seconds. The default value is 30. Leave this value blank to initiate a continuous trace. When a continuous trace reaches the maximum space allocation of 100 MB, the oldest file is overwritten.
Maximum Capture Size (MB)
Specify the maximum capture file size in MBs. The default value is 100. The recommended maximum capture file size is 1024 MBs (1 GB).
Buffer Size
Optionally, specify the maximum number of packets allowed to queue up while awaiting processing by the TCP trace dump. The default value is 154.
Snap Length
Optionally, specify the snap length value for the trace dump. Specify 0 for a full packet capture (recommended for CIFS, MAPI, and SSL traces). The default value is 1518.
Number of Files to Rotate
Specify how many TCP trace dump files to rotate. The default value is 5.
Only Capture VLAN Packets
Captures only VLAN-tagged packets within a trace dump for a trunk port (802.1Q). Enabling this setting filters the trace dump by capturing only VLAN-tagged packets. This setting applies to physical interfaces only because logical interfaces (inpath0_0, mgmt0_0) do not recognize VLAN headers.
Custom Flags
Specify custom flags to capture unidirectional traces. Examples:
To capture all traffic to or from a single host
host x.x.x.x
To capture all traffic between a pair of hosts
host x.x.x.x and host y.y.y.y
To capture traffic between two hosts and two SteelHead inner channels:
(host x.x.x.x and host y.y.y.y) or (host a.a.a.a and host b.b.b.b)
Schedule Dump
Schedules the trace dump to run at a later date and time.
Start Date
Specify a date to initiate the trace dump in this format: YYYY/MM/DD
Start Time
Specify a time to initiate the trace dump in this format: HH:MM:SS
Add
Adds the TCP trace dump to the capture queue.
Control
Description
Enable Cascade Shark
Permits remote capture analysis using Cascade Pilot. When enabled, the TCP dump capture files appear in the list of completed TCP dumps in Cascade Pilot for more detailed analysis. The Shark function does not require a separate license and is disabled by default.
The Shark function uses port 61898 during the typical interaction with Cascade Pilot (for example, status requests, views configuration, and output transfer). It uses port 61899 during packet transfers.
When Cascade Shark is disabled, Shark function is shut down and no processes are listening on port 61898 or 61899.
To use the Shark function, you must give the username Shark a password.
Capture files do not appear in Cascade Pilot until they are complete.
You can create a TCP dump without the Cascade Shark enabled. The capture file appears in the TCP dump list in Cascade Pilot the next time you enable Cascade Shark and point Cascade Pilot to process TCP dumps from this SteelHead.
You must be able to reach the SteelHead with Cascade Shark enabled from the computer running Cascade Pilot.
Add a New TCP Dump
Displays the controls for creating a TCP trace dump.
Capture Name
Specify the name of the capture file. Use a unique filename to prevent overwriting an existing TCP dump. The default filename uses this format:
<hostname>_<interface>_<time-stamp>.cap
Where <hostname> is the hostname of the SCC, <interface> is the name of the interface selected for the trace (for example, lan0_0, wan0_0), and <time-stamp> is in the YYYY-MM-DD-HH-MM-SS format.
If this trace dump relates to an open Riverbed Support case, specify the capture filename case_<number> where <number> is your Riverbed Support case number: for example, case_12345.
Note: The .cap file extension is not included with the filename when it appears in the capture queue.
Capture Traffic Between
IPs - Specify the source IP addresses. Separate multiple IP addresses with a comma to include all addresses bidirectionally. The default setting is all IP addresses.
Ports - Specify the source ports. Separate multiple ports with a comma. The default setting is all ports.
and:
IPs - Specify the destination IP addresses. Separate multiple IP addresses with a comma to include all addresses bidirectionally. The default setting is all IP addresses.
Ports - Specify the destination ports. Separate multiple ports with a comma. The default setting is all ports.
Capture Interfaces
Captures the TCP trace dump on the selected interface(s). You can select all interfaces or a base, in-path, or RSP interface. The default setting is none. You must specify a capture interface.
If you select several interfaces at a time, the data is automatically placed into separate capture files.
Capture Parameters
Select one of these traffic types to capture:
•  Capture Untagged Traffic Only - Captures only traffic without a VLAN tag. Enabling this setting filters the trace dump by capturing all untagged packets.
•  Capture VLAN-Tagged Traffic Only - Captures only VLAN-tagged packets within a trace dump for a trunk port (802.1Q). Enabling this setting filters the trace dump by capturing only VLAN-tagged packets. This setting applies to physical interfaces only because logical interfaces (inpath0_0, mgmt0_0) do not recognize VLAN headers.
•  Capture both VLAN and Untagged Traffic - Captures VLAN-tagged and untagged packets within a trace dump.
Capture Duration (Seconds)
Specify how long the capture runs, in seconds. The default value is 30. Specify 0 or continuous to initiate a continuous trace. When a continuous trace reaches the maximum space allocation of 100 MB, the oldest file is overwritten.
Maximum Capture Size (MB)
Specify the maximum capture file size, in megabytes. The default value is 100 MB. Riverbed recommends a maximum capture file size of 1024 MB (1 GB).
Buffer Size
Optionally, specify the maximum amount of data, in kilobytes, allowed to queue up while awaiting processing by the TCP trace dump. The default value is 154 KB.
Snap Length
Optionally, specify the snap length value for the trace dump. Specify 0 for a full packet capture (recommended for CIFS, MAPI, and SSL traces). The default value is 1518.
Number of Files to Rotate
Specify how many TCP trace dump files to rotate. The default value is 5.
Custom Flags
Specify custom flags to capture unidirectional traces. Examples:
To capture all traffic to or from a single host
host x.x.x.x
To capture all traffic between a pair of hosts
host x.x.x.x and host y.y.y.y
To capture traffic between two hosts and two SteelHead inner channels:
(host x.x.x.x and host y.y.y.y) or (host a.a.a.a and host b.b.b.b)
Schedule Dump
Schedules the trace dump to run at a later date and time.
Start Date
Specify a date to initiate the trace dump in this format: YYYY/MM/DD.
Start Time
Specify a time to initiate the trace dump in this format: HH:MM:SS.
Add
Adds the TCP trace dump to the capture queue.
If a problem occurs with an immediate or scheduled TCP dump, this message appears:
“Error in tcpdump command. See System Log for details.”
Check the trace dump for any syntax errors.
Custom Flag Use Examples
The examples in this table focus on the custom flag entry but rely on other fields to create a complete filter.
Filter Purpose
Custom Flag
To capture all traffic on VLAN 10 between two specified endpoints: 1.1.1.1 and 2.2.2.2
and vlan 10
To capture any packet with a SYN or an ACK
tcp[tcpflags] & (tcp-syn|tcp-ack) != 0
To capture any packet with a SYN
tcp[tcpflags] & (tcp-syn) != 0
—or—
tcp[13] & 2 == 2
To capture any SYN to or from host 1.1.1.1
and (tcp[tcpflags] & (tcp-syn) != 0)
—or—
and (tcp[13] & 2 == 2)
Stopping a TCP Dump After an Event Occurs
Capture files offer visibility into intermittent network issues, but the amount of traffic they capture can be overwhelming. Also, because rotating logs is common, after a capture logs an event, the SCC log rotation can overwrite debugging information specific to the event.
RiOS 8.5.x makes troubleshooting easier because it provides a trigger that can stop a continuous capture after a specific log event occurs. The result is a smaller file to help pinpoint what makes the event happen.
The stop trigger continuously scans the system logs for a search pattern. When it finds a match, it stops all running captures.
To stop a TCP dump after a specific log event
1. Choose Diagnostics > Appliance Logs: TCP Dumps to display the TCP Dumps page.
2. Under TCP Dump Stop Trigger, select an appliance from the drop-down list.
Figure: TCP Dump Stop Trigger
3. In the Pattern control, enter a Perl regular expression (regex) to find in a log. RiOS compares the Perl regex against each line in the system logs and the trigger stops if it finds a match.
The simplest regex is a word, or a string of characters. For example, if you set the pattern to Limit, the trigger matches the line Connection Limit Reached.
Notes:
•  Perl regular expressions are case-sensitive.
•  Perl treats a space character ' ' like any other character in a regexp.
•  You cannot use all characters as is in a match. Perl reserves some characters, called metacharacters, for use in regex notation. The metacharacters are:
{}[]()^$.|*+?\
You can match a metacharacter by putting a backslash before it. For example, to search for a backslash in the logs, you must enter two backslashes (\\) as the pattern.
•  The pattern follows Perl regular expression syntax. For details, go to:
http://perldoc.perl.org/perlre.html
•  You cannot change the pattern while a scan is running. You must stop the scan before changing a pattern.
•  You do not need to wrap the pattern with the metacharacters to match the beginning or end of a line (^ $), or with the wildcard character (*).
4. Specify the amount of time to pause before stopping all running dumps when RiOS finds a match. This gives the system some time to log more data without abruptly cutting off the dumps. The default is 30 seconds. Specify 0 for no delay; the dump stops immediately.
After a trigger has fired, the capture can stop by itself before the delay expires; for example, the capture duration can expire.
5. Click Start Scan.
When the scan stops, RiOS sends an email to all email addresses on the Email page appearing under Report Events via Email. The email notifies users that the trigger has stopped.
The page indicates “Last Triggered never” if a TCP Dump stop trigger has never triggered on the appliance. After the delay duration of the stop trigger, RiOS displays the last triggered time, that is the delay time plus the expired time.
To stop a running scan
•  Click Stop Scan to halt the background process that monitors the system logs. RiOS dims this button when the stop trigger is idling.
Stop Trigger Limitations
These limitations apply to the trigger:
•  You cannot create a trigger to stop a specific capture; the trigger affects all running captures.
•  If the search pattern contains a typo, the trigger might never find a match.
•  Only one instance of a trigger can run at one time.
Viewing a TCP Dump
The top of the TCP Dumps page displays a list of existing captures.
To view a capture file
1. Choose Diagnostics > Appliance Logs: TCP Dumps to display the TCP Dumps page.
2. Under Stored TCP Dumps, select the Capture Name to open the file.
3. Click Download to view a previously saved capture file.
4. To remove a capture file, select the check box next to the name and click Remove Selected.
To print a capture file
1. Choose Diagnostics > Appliance Logs: TCP Dumps to display the TCP Dumps page.
2. Under Download Link, select the capture filename to open the file.
3. When the file opens, choose File > Print in your web browser to open the Print dialog box.
To stop a running capture
1. Choose Diagnostics > Appliance Logs: TCP Dumps to display the TCP Dumps page.
2. Select the capture filename in the Running Capture Name list.
Click Stop Selected Captures.
Uploading a TCP Dump
Riverbed offers a couple of ways to upload TCP dump files to the support server for sharing with the support team while diagnosing issues.
To upload the trace to Riverbed Support
1. On the TCP Dumps page, select the TCP dump filename.
2. Optionally, specify a case number that corresponds to the TCP dump. Riverbed Support recommends using a case number; for example, 194170.
To specify a URL instead of a case number, you must use the CLI. You can enter the CLI command file tcpdump upload URL. When you specify a URL, the dump file goes directly to the URL.
If the URL points to a directory on the upload server, it must have a trailing /
For example:
ftp://ftp.riverbed.com/incoming/
(not ftp://ftp.riverbed.com/incoming)
The filename as it exists on the appliance will then match the filename on the upload server.
For details, see the Riverbed Command-Line Interface Reference Manual.
3. Click Upload.
Because uploading a TCP dump can take a while (especially when including ESXi information on a SteelHead EX), a progress bar displays the percentage of the total upload completed, the case number (if applicable), and the date and time the upload began. When the TCP dump finishes uploading, the date, time, and a status of either uploaded (appears in green) or failed (appears in red).
Successful uploads show the status, the case number (if applicable), and the date and time the upload finished.
For uploads that fail, an explanation, the case number (if applicable), and the upload starting date and time appear.