Troubleshooting
This chapter describes how to troubleshoot the SteelHead installation. This chapter describes how to troubleshoot these issues:
Cables
Improper cabling prevents smooth traffic flows between the SteelHead and the router or switch.
Solution
To ensure that the traffic flows when the SteelHead is optimizing traffic, and when the SteelHead transitions to bypass mode, use the appropriate crossover or straight-through cable to connect the SteelHead to a router or switch. Verify the cable selection by removing the power cable from the appliance, and then test connectivity through it. Make sure that you have connected your cables as follows:
• SteelHead to router: crossover cable
• SteelHead to switch: straight-through cable
• SteelHead to SteelHead: crossover cable
• SteelHead to a host: crossover cable
For details, go to the
Riverbed Knowledge Base article S15298 at the following URL:
https://supportkb.riverbed.com/support/index?page=content&id=S15298.
Duplex mismatch
These symptoms occur due to a duplex mismatch:
• Access is not faster after configuring the SteelHead.
• The Link Duplex alarm triggers.
• The interface counters display error messages. An alarm or log message about error counts appears.
• The pass-through rule is ineffective. (This is a definite indication of duplex mismatch.)
• There are many retransmissions in packet traces.
• You can’t connect to an attached device.
• You can connect with a device when you choose auto-negotiation, but you can’t connect with the same device when you manually set the speed or duplex.
• Good performance for one direction of data flow, but poor performance in the opposite direction.
Possible cause
• You have probably set the duplex value for your router to 100 Full (fixed) and set the duplex value for the SteelHead to Auto.
Example
This example shows applications that appear slower with SteelHeads configured in an in-path deployment. The timed performance numbers to transfer a 20-MB file over FTP are:
• no SteelHead – 3:16
• cold SteelHead – 5:08
• warm SteelHead – 3:46
Adding a pass-through rule for an application doesn’t help. Slow connections appear as optimized in the Management Console on the Current Connections report page. However, stopping the SteelHead service while leaving the system powered on and an in-path configuration returns performance to original levels.
Solutions
To resolve the duplex mismatch error:
• Connect to the SteelHead CLI and enter the flood-ping command to check the duplex mismatch:
ping –f –I >in-path-ip> –s 1400 <clientIP>
• Change the interface speed and duplex to match.
• Ensure there’s a speed and duplex match between each in-path interface and its peer network interface. If they don’t match, you might have a large number of errors on the interface when it’s in the bypass mode, because the switch and the router aren’t set with the same duplex settings. Also, ensure connectivity when service is down.
If matching speed and duplex don’t reduce collisions or errors, try hard-setting one end and auto-setting the other. Try the half-duplex mode.
• If all combinations fail, as a last resort, add an intermediary hub or switch that is more compatible with both network interfaces.
In-path SteelHeads connection
When there are SteelHeads with in-path connection issues, the two sites are connected in-path and you can ping them, but they can’t connect to each other to optimize data.
Possible cause
The firewall is running port filtering and drops your probe packets. The firewall is filtering the IP and port address of the source and destination (bandwidth limitation) systems.
Solutions
To resolve the SteelHead connection issue:
• open port 7800 on both firewalls.
• use the port visibility mode.
• if there’s no encryption, place the SteelHead after the firewall.
Oplock issues
These symptoms occur due to opportunistic lock (oplock) issues:
• File access is not faster or tasks such as drag-and-drop are fast but applications might benefit from acceleration.
• The Current Connections report page in the Management Console (select Reports > Networking: Current Connections) displays slow connections as optimized.
Possible causes
• The client is running an old antivirus software such as McAfee v4.5, the most common type, which competes with the application for an oplock instead of opening as read-only. The antivirus causes multiple file opens.
• The server has oplocks disabled.
Example
You can open a previously-accessed file in 5 seconds on PC1, but you can’t open the same file under 24 seconds on PC2. If you close the file on PC1, you can open it in 5 seconds on PC2. However, it takes you 24 seconds to open the same file on PC1.
Solutions
Windows Common Internet File System (CIFS) uses oplock to determine the level of safety the OS or the application has in working with a file. Oplock is a lock that a client requests on a file in a remote server.
An oplock controls the consistency of optimizations such as read-ahead. Oplock levels are reduced when you make conflicting opens to a file.
To prevent any compromise to data integrity, the SteelHead only optimizes data when a client has exclusive access to the data.
When an oplock is not available, the SteelHead doesn’t perform application-level latency optimization but still performs scalable data referencing (SDR) and data compression as well as TCP optimization. Therefore, even without the benefits of latency optimization, SteelHeads still increase WAN performance, but not as effectively as when application optimizations are available.
To resolve oplock issues:
• Upgrade your antivirus software to the latest version.
• Use Filemon (sysinternals) to check for file access.
• Ensure that the server has oplock enabled by verifying registry settings on Windows servers or the Filer configuration (for NetApp or EMC servers).
• Run a network analyzer such as Riverbed SteelCentral Packet Analyzer Plus, which is fully integrated with Wireshark, and determine that the server grants oplocks when the client opens a file.
• Check whether the client is running an antivirus software that is scanning the files over the WAN or that the antivirus software doesn’t break the oplock.
CIFS overlapping open optimization denies multi-user access
The CIFS overlapping open optimization issue prevents a client from accessing a file when different clients access the file at the same time.
Solution
To resolve the CIFS overlapping open optimization issue, configure CIFS overlapping open optimization on the client-side SteelHead as follows:
2. On the client-side SteelHead, choose Optimization: CIFS (SMB1) to display the CIFS (SMB1) page.
Figure: CIFS (SMB1) page
3. Under Overlapping Open Optimization (Advanced), complete the configuration as described in this table.
Control | Description |
Enable Overlapping Open Optimization | Enables overlapping opens to obtain better performance with applications that perform multiple opens on the same file. For example, CAD applications. By default, this setting is disabled. Enable this setting on the client-side SteelHead. With overlapping opens enabled, the SteelHead optimizes data where exclusive access is available (when locks are granted). When an oplock is not available, the SteelHead doesn’t perform application-level latency optimizations but still performs SDR and compression on the data as well as TCP optimizations. If a remote user opens a file that is optimized using the overlapping opens feature and a second user opens the same file, they might receive an error message if it doesn’t go through a SteelHead. For example, certain applications that are sent over the LAN. If this occurs, you should disable overlapping opens for such applications. Use the radio buttons to set either an include list or exclude list of file types subject to overlapping opens optimization. |
Optimize only the following extensions | Specify a list of extensions you want to include in overlapping opens optimization. |
Optimize all except the following extensions | Specify a list of extensions you don’t want to include. For example, you should specify any file extensions that use Enable Applock Optimization. |
4. Click Apply to apply your settings to the current configuration.
5. Click Save to save your settings permanently.
IP address configuration
If you have not configured IP addresses correctly, the SteelHeads can’t connect to each other or to your network.
Solutions
To verify the IP address has been configured correctly:
• Ensure the SteelHeads are reachable through the IP address by pinging their primary and in-path interfaces.
• Ensure that the SteelHeads in the network can reach each other through their own interfaces.
Connect to the SteelHead CLI. For details, see the Riverbed Command-Line Interface Reference Manual. Enter this command to ping from a specific in-path interface on a SteelHead to another in-path interface:
ping -f -I {Local-SteelHead-Inpath-IP} -s 1400 {Remote-SteelHead-Inpath-IP}
• Ensure that the default gateways, both for the SteelHead and for its in-path interfaces, are correct.
• For physical or virtual in-path installations, verify that the server-side SteelHead can be auto-discovered by the client-side SteelHead.
Connect to the SteelHead CLI. For details, see the Riverbed Command-Line Interface Reference Manual. Enter the command:
tproxytrace -i inpath0_0 <example-server-IP-address>:<example-server-TCP-port>
This causes the SteelHead to generate a fake TCP SYN packet, destined for the specified IP address and TCP port, and send it to the specified in-path interface. A remote SteelHead should respond if it sees the SYN packet.
• Verify that the client-side SteelHead is visible to the server-side SteelHead.
Connect to the SteelHead CLI. For details, see the Riverbed Command-Line Interface Reference Manual. Enter the command:
tproxytrace -i inpath0_0 <example-client-IP-address>: <example-client-TCP-port>
Asymmetric routing
If there’s an asymmetric routing issue, many connections fail during data transfer or they fail to start.
Possible cause
Asymmetric routing occurs when a TCP connection takes one path to the destination and another when returning to the source. If the SteelHead sees only the LAN to WAN or only the WAN to LAN packets, it can’t optimize the data.
Solutions
To resolve the asymmetric routing issue, do one of the following:
• Rank these solutions from most to least preferable with respect to complexity and cost and select one:
– configure a fixed-target rule.
– use a logical in-path configuration such as WCCP or PBR.
– use four-port or six-port SteelHead.
– configure connection-forwarding with two SteelHeads.
• Remove the asymmetry.
Packet ricochet
These symptoms occur due to packet ricochet:
• Performance is less than expected.
• A log message similar to this appears:
kernel:[intercept.NOTICE] nat_check: SYN packet for already natted connection a.a.a.a:xxxx ->
b.b.b.b:yyyy ==> a.a.a.a:xxxx -> c.c.c.c:7801
where:
a.a.a.a is the client IP address
xxxx is the source port
b.b.b.b is the server IP address
yyyy is the destination port
c.c.c.c:7801 is the local SteelHead in-path IP address
Possible cause
Traffic to the LAN is traveling to the WAN router on the way to the LAN.
Solutions
To resolve packet ricochet issues:
• Change the in-path gateway to the LAN router.
• Add static routes to LAN subnets through the LAN router.
• Enable in-path simplified routing.
Packet ricochet: ICMP redirects
These symptoms occur due to packet ricochet Internet Control Messaging Protocol (ICMP) redirects:
• Connections fail on first attempt, but succeed on second attempt.
• On one or both sites, the in-path interface on the SteelHead is on a different network than the local host.
• There are no in-path routes defined.
Possible causes
• Traffic to the LAN is travelling to the WAN router on the way to the LAN, but the router drops the packet.
• Outer connections to clients or servers are routed through the WAN interface to the WAN gateway, and then routed through the SteelHead to the next hop LAN gateway.
• The WAN router is probably dropping the SYN from the SteelHead before issuing an ICMP redirect.
Solutions
To resolve the packet ricochet ICMP redirects issue, do one of the following:
• Change the router ICMP configuration to forward the packet or turn off ICMP redirect.
• Change the in-path gateway to the LAN router.
• Add static routes to LAN subnets through the LAN router.
• Add in-path routes to local destinations to prevent the ICMP redirect and subsequent drop.
Simplified routing
Simplified routing changes the process used to select the destination Ethernet address for packets transmitted from in-path interfaces.
Simplified routing collects the IP address for the next hop MAC address from each packet it receives to address traffic. With simplified routing, you can use either the WAN or LAN-side device as a default gateway. The SteelHead learns the right gateway to use by watching where the switch or router sends the traffic, and by associating the next-hop Ethernet addresses with IP addresses. Enabling simplified routing eliminates the need to add static routes when the SteelHead is in a different subnet from the client and the server.
Without simplified routing, if a SteelHead is installed in a different subnet from the client or server, you must define one router as the default gateway and static routes for the other routers so that traffic is not redirected back through the SteelHead. In some cases, even with the static routes defined, the Access Control List (ACL) on the default gateway can still drop traffic that should have gone through the other router. Enabling simplified routing eliminates this issue.
Simplified routing has these constraints:
• You can’t enable WCCP.
• The default route must exist on each SteelHead in your network.
For detailed information, see the SteelHead Deployment Guide.
To enable simplified routing
1. Choose Networking > Networking: Simplified Routing to display the Simplified Routing page.
2. Under Mapping Data Collection Setting, complete the configuration as described in this table.
Control | Description |
Collect Mappings From | Select one of these options from the drop‑down list: • None - Don’t collect mappings. • Destination Only - Collects destination MAC data. Use this option in connection-forwarding deployments. This is the default setting. • Destination and Source - Collect mappings from destination and source MAC data. Use this option in connection-forwarding deployments. • All - Collect mappings for destination, source, and inner MAC data. Also collect data for connections that are un-NATted (connections that aren’t translated using NAT). You can’t enable this option in connection-forwarding deployments. Riverbed recommends that you use this option to maximize the effects of simplified routing. |
3. Click Apply to save your settings to the running configuration.
4. Click Save to save your settings permanently.
Autodiscovery failure
When autodiscovery fails, all traffic passes through with the SteelHead in-path (physically or logically).
Possible causes
• Cisco PIX 7.x or Raptor firewalls
• Satellite
• Intrusion Detection System (IDS) or Intrusion Prevention System (IPS)
Solutions
• Create a fixed-target rule on the client-side SteelHead.
• Specify the Target Appliance IP Address and its port as 7800 on the opposite SteelHead (in-path without autodiscovery).
• Configure end nodes (firewalls) to allow your probe to pass through.
• Configure the SteelHead IP address as the friendly IP address for IDS or IPS.
• Cisco PIX Firewall IOS 7.0 might block the autodiscovery probe. Some firewall configurations strip TCP options or drop packets with these options. You can keep this configuration and switch to fixed-target rules or change the configuration on the firewall.
Protocol optimization errors
When there are protocol optimization errors, the SteelHead doesn’t optimize expected protocols.
Solutions
To resolve protocol optimization errors, check:
• that connections have been successfully established.
• that SteelHeads on the other side of a connection are turned on.
• for secure or interactive ports that are preventing protocol optimization.
• for any pass-through rules that could be causing some protocols to pass through the SteelHeads unoptimized.
• that the LAN and WAN cables aren’t inadvertently swapped.
Server-side out-of-path connection caveats
Consider these the caveats for a server-side out-of-path (OOP) SteelHead connection:
• OOP configuration doesn’t support autodiscovery. You must create a fixed-target rule on the client-side SteelHead.
• You must create an OOP connection from an in-path or logical in-path SteelHead and direct it to port 7810 on the primary interface of the server-side SteelHead. This setting is mandatory.
• Interception is not supported on the primary interface.
• An OOP configuration provides nontransparent optimization from the server perspective. Clients connect to servers, but servers treat it like a server-side SteelHead connection. This affects:
– log files.
– server-side ACLs.
– bidirectional applications such as rsh.
• You can use OOP configurations along with in-path or logical in-path configurations.
Specific problems
This section describes specific problems you might encounter in the SteelHead.
Problem | Solution |
The show interfaces CLI command displays 4294967295 as the number of errors on an interface. | The bypass card is not properly installed; reinstall it. For details, see the Network Interface Card Installation Guide. |
After installing a SteelHead, only the network connection lights are illuminated (turned on). | Press the small I/0 button on the front of the SteelHead. |
The SteelHead blocks traffic when going into bypass mode. | If a SteelHead blocks traffic when going into bypass mode, verify that connections to its neighboring devices are correctly configured. Ensure that the cable from the SteelHead to the switch is a straight-through cable and the cable from the SteelHead to the router is a crossover cable. Also, ensure that there are no network speed or duplex mismatches. |
The SteelHead doesn’t come out of bypass mode when the network connection is restored. | If a SteelHead doesn’t come out of bypass mode, verify that: • The in-path interface has an IP address. For example, at the system prompt, enter the show interfaces CLI command. • In-path interception is enabled. For example, at the system prompt, enter the show in-path CLI command. Expected results are: Enabled: yes Optimizations Enabled On: inpath0_0 • The bypass service is running. For example, at the system prompt, enter the show service CLI command. To enable the SteelHead service if it’s not running, use the CLI command service enable. • You have a valid and active SH10BASE license. Your license file should also contain entries for SH10CIFS and SH10EXCH licenses, even if they have not been activated. For example, at the system prompt, enter the show licenses CLI command. For questions about licenses, contact Riverbed Support at https://support.riverbed.com. |
The SteelHead fails to boot. | Ensure that the power strip or the uninterruptable power supply (UPS) the SteelHead is plugged into is turned on and is functioning properly. |
Your SteelHead does not boot and is beeping, which can be due to a RAID error or a failed power supply. | First, determine whether the cause of the beeping and failure to boot is a RAID error. To determine if there is a boot failure 1. Use a keyboard and monitor to connect to the SteelHead. 2. Run the megamgr utility to check for a hard disk failure. 3. If there is a single hard disk failure on a SteelHead, you can use the extra hard disk in slot 12 to replace the failed disk. For assistance, or if there is a hard disk failure on more than one disk in a SteelHead or on any other SteelHead model, contact Riverbed Support at https://support.riverbed.com. If the beeping and failure to boot are not caused by a RAID error, they could be due to a defective power supply. For assistance, contact Riverbed Support at https://support.riverbed.com. |
Resetting a lost password
To reset your password, you must have access to the serial console or monitor and must be able to see the entire boot process to perform these steps:
1. Start or reboot the appliance.
2. When prompted, press any key to continue.
3. Immediately press E. A GNU GRUB menu appears.
• For a SteelHead upgraded to 4.0 from 2.0 or 3.0, the menu prompts you to select the Riverbed SteelHead, diagnostics, or a restore/recovery image. Select Riverbed SteelHead and skip to
Step 5.
• For a SteelHead manufactured with 4.0 (that has not had previous versions), the menu prompts you to select the disk image to use. Continue with
Step 4.
• For software versions prior to 4.0, the menu displays root and kernel parameters. Skip to
Step 6.
4. Press V or ^ to select the disk image to boot.
5. Press E.
A GRUB menu appears, with options similar to the following:
-----------------
0: root (hd0,1)
1: kernel /vmlinuz ro root=/dev/sda5 console=tty0 console=ttyS0,9600n8
-----------------
6. Press V or ^ to select the kernel boot parameters entry.
7. Press E to edit the kernel boot parameters. The CLI displays a partially completed line of text similar to the following:
kernel /vmlinuz ro root=/dev/sda5 console=tty0 console=ttyS0,9600n8
8. The line of text contains two console= entries. Modify this line as follows:
– If you are accessing the SteelHead remotely, delete
console=tty0
– If you are accessing the SteelHead directly (through a keyboard and monitor connected to the appliance), delete
console=ttyS0
– At the end of the line, type a space and append the line with
single fastboot
– You must include a space before the word single.
Note: Use the arrow keys to access the entire command line.
9. Press Enter.
10. Press B to continue booting.
The system starts.
11. At the command prompt, enter /sbin/resetpw.sh.
The password is blank.
12. Type reboot and press Enter to reboot the appliance.
Network integration checklist
Before you begin configuring the SteelHead, check these configuration settings:
• Speed and duplex.
• QoS integration.
• Multihop optimization.
• Packet ricochet.
• VPN: Ensure the encryption is on the WAN side of the SteelHead.
• Firewall: Ensure probes are passed, especially Cisco PIX and Raptor. If inside the SteelHead, try probe caching for src IP rules; if outside, check firewall performance.
• In-path: Is it a VLAN trunk? (Configure trunking).
• Incorrectly designed load balancing implementations.
• Remove or manage asymmetry.
• Fail-to-wire or fail-to-block, you need Link State Protocol (LSP) for quicker convergence.
• WCCP or VLAN bridge: Router model and IOS revision.
• Does the network use Network Address Translation (NAT) or Port Address Translation (PAT)?