Viewing Diagnostic Reports and Logs : Managing TCP Dumps
  
Managing TCP Dumps
You create, view, upload, and stop TCP dumps in the TCP Dumps page. This section contains these topics:
•  Creating Appliance TCP dumps
•  Capturing TCP dumps for Interceptor clusters
•  Troubleshooting
•  Custom flag use examples
•  Stopping a TCP dump after an event occurs
•  Viewing a TCP dump
•  Uploading a TCP dump
Creating Appliance TCP dumps
You can create, download, and upload TCP dump capture files for appliances in the TCP Dumps page.
Capture files contain summary information for every internet packet received or transmitted on the interface to help diagnose problems in the system.
RiOS provides an easy way to create and retrieve multiple capture files. You can create capture files from multiple interfaces at the same time, limit the size of the capture file, and schedule a specific date and time to create a capture file. Scheduling and limiting a capture file by time or size allows unattended captures.
The top of the TCP Dumps page displays a list of existing capture files and the bottom of the page displays controls to create a capture file. The bottom of the page also includes the capture files that are currently running, and controls to create a trigger that stops a capture when a specific event occurs. The Running Capture Name list includes captures running at a particular time. It includes captures started manually and also any captures that were scheduled previously and are now running.
Note: The SCC automatically uploads the tcpdumps to itself and zips them once the capture is finished. You may want to ensure that the TCP dumps don’t saturate their WAN links by either minimizing the size of the captures or employing QoS to rate limit the transfers.
Capturing TCP dumps for Interceptor clusters
With RiOS 9.6, the SCC customizes the display in the TCP Dumps page to guide you to specify the correct endpoints (that is, IP addresses) so that you can capture all relevant packets to debug Interceptor cluster configurations. This feature reduces the number of TCP dumps taken for debugging and cluster set up.
Previously you would have to create multiple TCP dumps, such as Interceptor—>SteelHead, Server—>Client, and so forth to obtain all the data you need to debug cluster configuration. With RiOS 9.6, you can capture all relevant packets in a single TCP dump. TCP Dump capture supports IPv4 traffic with correct addressing and full-transparency mode. For IPv6 traffic, all relevant packets are captured as long as there are no IPv6 extended headers in the data packet originating on the client or server.
In the SCC you can select an Interceptor appliance or a SteelHead local to the Interceptor to capture flows for Client or Server endpoints. You can also specify whether you want to capture packets from the inner channel.
This table summarizes possible inner channel capture use cases.
Capture use cases
Packets captured
Interceptor without inner channel capture (lan0_0)
•  All responses from the server
•  All GRE messages between the Interceptor and the SteelHead
Interceptor without inner channel capture (wan0_0)
•  All probe messages
Interceptor with inner channel capture (lan0_0)
•  All requests from client
•  All responses from the server
•  All GRE messages between the Interceptor and the SteelHead
•  All heartbeat messages between the Interceptor and the SteelHead
Interceptor with inner channel capture (wan0_0)
•  All packets from client
•  All responses from the server
•  All probe messages
SteelHead (local to the Interceptor) without inner channel capture (wan0_0)
•  All requests from client
•  All responses from the server
•  All GRE messages between the Interceptor and the SteelHead
SteelHead (local to the Interceptor) with inner channel capture (wan0_0)
•  All requests from client
•  All responses from the server
•  All GRE messages between the Interceptor and the SteelHead
•  All heartbeat messages between the Interceptor and the SteelHead
SteelHead not local to the Interceptor
No option to specify the location of the Interceptor (client/server). You can only capture traffic between SteelHead IP addresses in a comma separated list.
To add or remove stored TCP trace dumps
1. Choose Diagnostics > Appliance Logs: TCP Dumps to display the TCP Dumps page.
2. Complete the configuration as described in this table.
Control
Description
Add a New TCP Dump
Displays the controls for creating a TCP trace dump.
Capture Name
Specify the name of the capture file. Use a unique filename to prevent overwriting an existing TCP dump. The default filename uses this format:
<hostname>_<interface>_<time-stamp>.cap
Where <hostname> is the hostname of the SCC, <interface> is the name of the interface selected for the trace (for example, lan0_0, wan0_0), and <time-stamp> is in the yyyy/mm/dd hh:mm:ss format.
If this trace dump relates to an open Riverbed Support case, specify the capture filename case_<number> where <number> is your Riverbed Support case number: for example, case_12345.
Note: The .cap file extension isn’t included with the filename when it appears in the capture queue.
Appliances
Select an appliance from the list. The SCC displays Interceptors, SteelHeads, and SteelHeads that are local to the Interceptor.
If you select an Interceptor appliance or a SteelHead that is local to the Interceptor, the Endpoints section displays the Select Interceptor Location: Client/Server option so that you can capture data based on the location of the Interceptor in your network.
If you select an appliance that isn’t local to the Interceptor the page displays only the Endpoints option for SteelHead deployments for capturing packets between endpoints.
Endpoints (SteelHead deployments)
Specify source and destination IP addresses and corresponding ports to capture packets between endpoints. For example, specify the client-side IP addresses and server-side addresses to capture packets between these endpoints.
Capture traffic between:
•  IPs - Specify All to capture all IP addresses on one side of the network or specify particular IP addresses separated by commas. You can specify IPv4 or IPv6 addresses. The default setting is All.
•  Ports - Specify All to capture all corresponding ports or specify particular ports separated by commas. The default setting is All.
—and—
•  IPs - Specify All to capture all IP addresses on the other side of the network or specify particular IP addresses separated by commas. You can specify IPv4 or IPv6 addresses. The default setting is All.
•  Ports - Specify All to capture all corresponding ports or specify particular ports separated by commas. The default setting is All.
Note: To capture traffic flowing in only one direction or to enter a custom command, use the CLI tcpdump command. For details, see the Riverbed Command-Line Interface Reference Manual.
Endpoints (Interceptor clusters)
Select Interceptor Location - Select either Client or Server from the drop-down list. Your choice determines the endpoints (that is, IP addresses) that you can specify.
If you select Client:
•  IPs - Specify All to capture all the client-side endpoints or specify one or more IP addresses separated by commas. Specify client-side addresses only. You can specify IPv4 or IPv6 addresses. The default setting is All.
•  Ports - Specify All to capture all corresponding ports or specify one or more ports separated by commas. The default setting is All.
If you select Server:
•  IPs - Either specify All to capture all server-side endpoints or specify one or more IP addresses separated by commas. Specify client-side addresses only. You can specify IPv4 or IPv6 addresses. The default setting is All.
•  Ports - Specify All to capture all corresponding ports or specify one or more ports separated by commas. The default setting is All.
•  Capture Inner Channel Data - Captures all inner channel requests between the endpoints. The default setting is off.
•  Appliance IP address - Specify the in-path IP address of the local SteelHead.
•  Service Port - Specify the service port of the in-path local SteelHead. The default service port number is 7800.
Capture Interfaces
Captures packet traces on the selected interfaces. You can select all interfaces or a base or in-path interface. The default setting is none. You must specify a capture interface.
If you select several interfaces at a time, the data is automatically placed into separate capture files.
When path selection is enabled, we recommend that you collect packet traces on all LAN and WAN interfaces.
Capture Parameters
These parameters let you capture information about dot1q VLAN traffic. You can match traffic based on VLAN-tagged or untagged packets, or both. You can also filter by port number or host IP address and include or exclude ARP packets. Select one of these parameters for capturing VLAN packets:
•  Capture Untagged Traffic Only - Select this option for these captures:
–  All untagged VLAN traffic.
–  Untagged 7850 traffic and ARP packets. You must also specify or arp in the custom flags field in this page.
–  Only untagged ARP packets. You must also specify and arp in the custom flags field in this page.
•  Capture VLAN-Tagged Traffic Only - Select this option for these captures:
–  Only VLAN-tagged traffic.
–  VLAN-tagged packets with host 10.11.0.6 traffic and ARP packets. You must also specify 10.11.0.6 in the IPs field, and specify or arp in the custom flags field in this page.
–  VLAN-tagged ARP packets only. You must also specify and arp in the custom flags field in this page.
•  Capture both VLAN and Untagged Traffic - Select this option for these captures:
–  All VLAN traffic.
–  Both tagged and untagged 7850 traffic and ARP packets. You must also specify these values in the custom flags field in this page:
(port 7850 or arp) or (vlan and (port 7850 or arp))
–  Both tagged and untagged 7850 traffic only. You must also specify 7850 in one of the port fields in this page. No custom flags are required.
–  Both tagged and untagged ARP packets. You must also specify these values in the custom flags field in this page:
(arp) or (vlan and arp)
Capture Duration (Seconds)
Specify a positive integer to set how long the capture runs, in seconds. The default value is 30. Specify 0 or continuous to initiate a continuous trace.
For continuous capture, we recommend specifying a maximum capture size and a nonzero rotate file number to limit the size of the TCP dump.
Maximum Capture Size
Specify the maximum capture file size in megabytes. The default value is 100. After the file reaches the maximum capture size, TCP dump starts writing capture data into the next file, limited by the Number of Files to Rotate field.
We recommend a maximum capture file size of 1024 MB (1 GB).
Buffer Size
Optionally, specify the maximum amount of data, in kilobytes, allowed to queue while awaiting processing by the capture file. The default value is 154 kilobytes.
Snap Length (bytes)
Optionally, select the snap length value for the capture file or specify a custom value. The snap length equals the number of bytes the report captures for each packet. Having a snap length smaller than the maximum packet size on the network enables you to store more packets, but you might not be able to inspect the full packet content.
Select 65535 for a full packet capture (recommended for CIFS, MAPI, and SSL captures). The default value is 1518 bytes.
When using jumbo frames, we recommend selecting 9018.
The default custom value is 16383 bytes.
Number of Files to Rotate
Specify how many capture files to keep for each interface before overwriting the oldest file. To stop file rotation, you can specify 0; however, we recommend rotating files, because stopping the rotation can fill the disk partition.
This control limits the number of files created to the specified number and begins overwriting files from the beginning, thus creating a rotating buffer.
The default value is 5. The maximum value is 2147483647.
Custom Flags
Specify custom flags as additional statements within the filter expression. Custom flags are added to the end of the expression created from the Endpoints fields and the Capture Parameters radio buttons (pertaining to VLANs).
If you require an “and” statement between the expression created from other fields and the expression that you are entering in the custom flags field, you must include the “and” statement at the start of the custom flags field.
Do not use host, src, or dst statements in the custom flags field. Although it is possible in trivial cases to get these statements to start without a syntax error, they don’t capture GRE-encapsulated packets that some modes of SteelHead communications use, such as WCCP deployments or Interceptor connection-setup traffic. We recommend using bidirectional filters by specifying endpoints.
For complete control of your filter expression, use the CLI tcpdump command. For details, see the Riverbed Command-Line Interface Reference Manual.
Schedule Dump
Schedules the trace dump to run at a later date.
•  Start Date - Specify a date to initiate the trace dump in this format: yyyy/mm/dd
•  Start Time - Specify a time to initiate the trace dump in this format: hh:mm:ss
Add
Adds the TCP trace dump to the capture queue.
Remove Selected
Under Stored TCP Dumps, select the TCP Dump check box and click Remove Selected.
Troubleshooting
If the tcpdump command results in a syntax error with an immediate or scheduled TCP dump, this message appears:
Error in tcpdump command. See System Log for details.
Review the system log to see the full tcpdump command attempt. Check the expression for issues such as a missing “and” statement as well as contradictory instructions such as looking for VLAN-tagged traffic and nontagged traffic.
Custom flag use examples
The examples in this table focus on the custom flag entry but rely on other fields to create a complete filter.
Filter purpose
Custom flag
To capture all traffic on VLAN 10 between two specified endpoints: 1.1.1.1 and 2.2.2.2
and vlan 10
To capture any packet with a SYN or an ACK
tcp[tcpflags] & (tcp-syn|tcp-ack) != 0
To capture any packet with a SYN
tcp[tcpflags] & (tcp-syn) != 0
-or-
tcp[13] & 2 == 2
To capture any SYN to or from host 1.1.1.1
and (tcp[tcpflags] & (tcp-syn) != 0)
-or-
and (tcp[13] & 2 == 2)
IPv6 custom flag use examples
The examples in this table focus on the custom flag entry but rely on other fields to create a complete filter.
To build expressions for TCP dump, IPv6 filtering doesn’t currently support the TCP, UDP, and other upper-layer protocol types that IPv4 does. Also, these IPv6 examples are based on the assumption that only a single IPv6 header is present.
Filter purpose
Custom flag
To capture all FIN packets to or from host 2001::2002
and (ip6[53] & 1!=0)
To capture all IPv6 SYN packets
ip6 or proto ipv6 and (ip6[53] & 2 == 2)
Stopping a TCP dump after an event occurs
Capture files offer visibility into intermittent network issues, but the amount of traffic they capture can be overwhelming. Also, because rotating logs is common, after a capture logs an event, the SteelHead log rotation can overwrite debugging information specific to the event.
RiOS 8.5.x and later make troubleshooting easier because they provide a trigger that can stop a continuous capture after a specific log event occurs. The result is a smaller file to help pinpoint what makes the event happen.
The stop trigger continuously scans the system logs for a search pattern. When it finds a match, it stops all running captures.
To stop a capture after a specific log event
1. Choose Diagnostics > Appliance Logs: TCP Dumps to display the TCP Dumps page.
2. Schedule a capture.
3. Under TCP Dump Stop Trigger: Pattern, enter a Perl regular expression (regex) to find in a log. RiOS compares the Perl regex against each new line in the system logs and the trigger stops if it finds a match.
The simplest regex is a word or a string of characters. For example, if you set the pattern to “Limit,” the trigger matches the line “Connection Limit Reached.”
Notes:
–  Perl regular expressions are case sensitive.
–  Perl treats the space character like any other character in a regex.
–  Perl reserves some characters, called metacharacters, for use in regex notation. The metacharacters are:
–  { } [ ] ( ) ^ $ . | * + ? \
–  You can match a metacharacter by putting a backslash before it. For example, to search for a backslash in the logs, you must enter two backslashes (\\) as the pattern.
–  The pattern follows Perl regular expression syntax. For details, go to:
–  http://perldoc.perl.org/perlre.html
–  You can’t change the pattern while a scan is running. You must stop the scan before changing a pattern.
–  You don’t need to wrap the pattern with the metacharacters to match the beginning or end of a line (^ $) or with the wildcard character (*).
4. Specify the amount of time to pause before stopping all running captures when RiOS finds a match. The time delay gives the system some time to log more data without abruptly cutting off the capture. The default is 30 seconds. Specify 0 for no delay; the capture stops immediately.
After a trigger has fired, the capture can stop by itself before the delay expires. For example, the capture duration can expire.
5. Click Start Scan.
When the scan stops, RiOS sends an email to all email addresses on the Administration: System Settings > Email page appearing under Report Events via Email. The email notifies users that the trigger has fired.
The page indicates “Last Triggered: Never” if a TCP Dump stop trigger has never triggered on the SteelHead. After the delay duration of the stop trigger, RiOS displays the last triggered time.
Before changing the Perl regular expression or amount of delay, you must first stop the process.
To stop a running scan
•  Click Stop Scan to halt the background process that monitors the system logs. RiOS dims this button when the stop trigger is idling.
Stop trigger limitations
These limitations apply to the trigger:
•  You can’t create a trigger to stop a specific capture; the trigger affects all running captures.
•  If the search pattern contains a typo, the trigger might never find a match.
•  Only one instance of a trigger can run at one time.
Viewing a TCP dump
The top of the TCP Dumps page displays a list of existing captures.
To view a capture file
1. Choose Diagnostics > Appliance Logs: TCP Dumps to display the TCP Dumps page.
2. Under Stored TCP Dumps, select the capture name to open the file.
3. Click Download to view a previously saved capture file.
4. To remove a capture file, select the check box next to the name and click Remove Selected.
To print a capture file
1. Choose Diagnostics > Appliance Logs: TCP Dumps to display the TCP Dumps page.
2. Under Download Link, select the capture filename to open the file.
3. When the file opens, choose File > Print in your web browser to open the Print dialog box.
To stop a running capture
1. Choose Diagnostics > Appliance Logs: TCP Dumps to display the TCP Dumps page.
2. Select the capture filename in the Running Capture Name list.
3. Click Stop Selected Captures.
Uploading a TCP dump
Riverbed offers a way to upload capture files to the support server for sharing with the support team while diagnosing issues.
To upload the capture file to Riverbed Support
1. In continuous mode, on the TCP Dumps page, select the running capture and click Stop Selected Captures.
For timed captures that are complete, skip to Step 2.
The capture appears as a download link in the list of Stored TCP Dumps.
2. Select the capture filename.
3. Optionally, specify a case number that corresponds to the capture. We recommend using a case number: for example, 194170.
To specify a URL instead of a case number, you must use the CLI. You can enter the CLI command file tcpdump upload url. When you specify a URL, the capture file goes directly to the URL.
If the URL points to a directory on the upload server, it must have a trailing backslash (/).
For example:
ftp://ftp.riverbed.com/incoming/
(not ftp://ftp.riverbed.com/incoming)
The filename as it exists on the appliance will then match the filename on the upload server.
For details, see the Riverbed Command-Line Interface Reference Manual.
4. Click Upload.
Because uploading a capture file can take a while, a progress bar displays the percentage of the total upload completed, the case number (if applicable), and the date and time the upload began. When the capture file finishes uploading, the date, time, and a status of either uploaded (appears in green) or failed (appears in red) are indicated.
Successful uploads show the status, the case number (if applicable), and the date and time the upload finished.
For uploads that fail, an explanation, the case number (if applicable), and the upload starting date and time appear.