Troubleshooting SteelHead-v
This section describes how to troubleshoot SteelHead-v. It includes the following sections:
Duplex Mismatch
If the pass-through rule becomes ineffective, this indicates duplex mismatch. Duplex mismatch could also be the problem if you experience these issues:
• Access is not faster after configuring SteelHead-v.
• The interface counters display error messages. An alarm or log message about error counts appears.
• There are many retransmissions in packet traces.
• You cannot connect to an attached device.
• You can connect to a device when you choose autonegotiation, but you cannot connect to the same device when you manually set the speed or duplex.
• Good performance in one direction of data flow, but poor performance in the opposite direction.
Possible Cause
You have probably set the duplex value for your router to 100Full (fixed) and for the SteelHead-v to Auto.
Example
The following example shows applications that appear slower with SteelHead-v appliances configured in an in-path deployment. The timed performance in minutes and seconds to transfer a 20-MB file over FTP are:
• no SteelHead-v – 3:16.
• cold SteelHead-v – 5:08.
• warm SteelHead-v – 3:46.
Adding a pass-through rule for an application does not help. Slow connections appear as optimized in the Management Console on the Current Connections report page. However, stopping the SteelHead-v service while leaving the system powered on in an in-path configuration returns performance to original levels.
Solutions
You resolve duplex mismatch on the virtual host. You cannot configure speed and duplex settings through the Management Console of the SteelHead-v.
To resolve the duplex mismatch error:
• connect to the SteelHead-v CLI and enter the ping command with the flood (-f) option to check the duplex mismatch:
ping –f –I <in-path-ip> –s 1400 <client-ip>
• ensure there is a speed and duplex match between each in-path interface and its peer network interface. If they do not match, you might have a large number of errors on the interface when it is in the bypass mode, because the switch and the router are not set with the same duplex settings. Also, ensure connectivity when service is down.
If matching speed and duplex do not reduce collisions or errors, try hard-setting one end and auto-setting the other. Try the half-duplex mode.
If all combinations fail, as a last resort, add an intermediary hub or switch that is more compatible with both network interfaces.
Oplock Issues
The following symptoms occur due to opportunistic lock (oplock) issues:
• File access is not faster or tasks such as drag-and-drop are fast but applications might benefit from acceleration.
• The Current Connections report page in the Management Console (select Reports > Networking: Current Connections) displays slow connections as optimized.
Possible Causes
• The client is running an old antivirus software such as McAfee 4.5, the most common type, which competes with the application for an oplock instead of opening as read-only. The antivirus causes multiple file opens.
• The server has oplocks disabled.
Example
You can open a previously accessed file in 5 seconds on PC1, but you cannot open the same file in under 24 seconds on PC2. If you close the file on PC1, you can open it in 5 seconds on PC2. However, it takes you 24 seconds to open the same file on PC1.
Solutions
Windows Common Internet File System (CIFS) uses oplock to determine the level of safety the OS or the application has in working with a file. Oplock is a lock that a client requests on a file in a remote server.
An oplock controls the consistency of optimizations such as read-ahead. Oplock levels are reduced when you make conflicting opens to a file.
To prevent any compromise to data integrity, SteelHead-v optimizes data only when a client has exclusive access to the data.
When an oplock is not available, SteelHead-v does not perform application-level latency optimization but still performs Scalable Data Referencing (SDR) and data compression, as well as TCP optimization. Therefore, even without the benefits of latency optimization, SteelHead-v appliances still increase WAN performance, but not as effectively as when application optimizations are available.
To resolve oplock issues:
• Upgrade your antivirus software to the latest version.
• Use FileMon (sysinternals) to check for file access, if available on your version of Windows.
• Ensure that the server has oplock enabled by verifying registry settings on Windows servers or the system configuration (for NetApp or EMC servers).
• Run a network analyzer such as
SteelCentral Pilot, which is fully integrated with Wireshark, and determine that the server grants oplocks when the client opens a file.
• Check whether the client is running an antivirus software that is scanning the files over the WAN or that the antivirus software does not break the oplock.
CIFS Overlapping Open Optimization Denies Multiuser Access
The CIFS overlapping open optimization issue prevents a client from accessing a file when different clients access the file at the same time.
Solution
To resolve the CIFS overlapping open optimization issue, configure CIFS overlapping open optimization on the client-side SteelHead as follows:
1. Connect to the SteelHead Management Console.
For details, see the SteelHead Management Console User’s Guide.
2. On the client-side SteelHead, choose Optimization > Protocols: CIFS (SMB1) to display the CIFS (SMB1) page.
Figure: CIFS (SMB1) Page
3. Under Overlapping Open Optimization (Advanced), complete the configuration as described in the following table.
Control | Description |
Enable Overlapping Open Optimization | Enables overlapping opens to obtain better performance with applications that perform multiple opens on the same file: for example, CAD applications. By default, this setting is disabled. Note: Enable this setting on the client-side SteelHead. With overlapping opens enabled, the SteelHead optimizes data where exclusive access is available (when locks are granted). When an oplock is not available, the SteelHead does not perform application-level latency optimizations but still performs SDR and compression on the data, as well as TCP optimizations. Note: If a remote user opens a file that is optimized using the overlapping opens feature on a 3.x.x or later SteelHead, or if the file does not go through a SteelHead and a second user opens the same file, the second user might receive an error message if the file fails to go through. For example, certain applications that are sent over the LAN may cause this problem. If this occurs, you should disable overlapping opens for such applications. Use the radio buttons to set either an include list or exclude list of file types subject to overlapping opens optimization. |
Optimize only the following extensions | Specify a list of extensions that you want to include in overlapping opens optimization. |
Optimize all except the following extensions | Specify a list of extensions that you do not want to include. For example, you should specify any file extensions that use Enable Applock Optimization. |
4. Click Apply to apply your settings to the current configuration.
5. Click Save to save your settings permanently.
IP Address Configuration
If you have not configured IP addresses correctly, the SteelHeads cannot connect to each other or to your network.
Solutions
Follow these guidelines below to correctly configure IP addresses:
• Ensure that the SteelHeads are reachable through the IP address by pinging their primary and in-path interfaces.
• Ensure that the SteelHeads in the network can reach each other through their own interfaces.
Connect to the SteelHead CLI. For details, see the Riverbed Command-Line Interface Reference Manual. Enter the following command to ping from a specific in-path interface on a SteelHead to another in-path interface:
ping -f -I <local-SteelHead-inpath-ip> -s 1400 <remote-SteelHead-inpath-ip>
• Ensure that the default gateways, both for the SteelHead and for its in-path interfaces, are correct.
• For physical or virtual in-path installations, verify that the server-side SteelHead can be auto-discovered by the client-side SteelHead.
Connect to the SteelHead CLI. For details, see the Riverbed Command-Line Interface Reference Manual. Enter the following command:
tproxytrace -i inpath0_0 <example-server-ip-address>:<example-server-tcp-port>
This command causes the SteelHead to generate a fake TCP SYN packet, destined for the specified IP address and TCP port, and send it to the specified in-path interface. A remote SteelHead should respond if it sees the SYN packet.
• Verify that the client-side SteelHead is visible to the server-side SteelHead.
Connect to the SteelHead CLI. For details, see the Riverbed Command-Line Interface Reference Manual. Enter the following command:
tproxytrace -i inpath0_0 <example-client-ip-address>: <example-client-tcp-port>
Asymmetric Routing
If there is an asymmetric routing issue, many connections fail during data transfer or they fail to start.
Possible Cause
Asymmetric routing occurs when a TCP connection takes one path to the destination and another when returning to the source. If the SteelHead-v sees only the LAN-to-WAN or only the WAN-to-LAN packets, it cannot optimize the data.
Solutions
To resolve the asymmetric routing issue, perform one of the following actions:
• Rank the following solutions from most to least preferable, with respect to complexity and cost, and select one:
– Configure a fixed-target rule.
– Use a logical in-path configuration such as WCCP or PBR.
– Configure connection-forwarding with two SteelHeads.
• Remove the asymmetry.
Packet Ricochet
The following symptoms occur due to packet ricochet:
• Performance is less than expected.
• The following log message appears:
» [fionr taelrcreeapdt/y lnoactaltekde rnceoln/neiccotireo.n c:119426.316] 8.n7a3t._1c5h:e1c6k1: 1 SYN ==> packet 192.168.208.12:80 ==> 192.168.72.9:7801
Possible Cause
Traffic to the LAN is traveling to the WAN router on the way to the LAN.
Solutions
To resolve packet ricochet issues:
• change the in-path gateway to the LAN router.
• add static routes to LAN subnets through the LAN router.
• enable in-path simplified routing.
Packet Ricochet—ICMP Redirects
The following symptoms occur due to packet ricochet Internet Control Messaging Protocol (ICMP) redirects:
• Connections fail on first attempt, but succeed on second attempt.
• On one or both sites, the in-path interface on the SteelHead is on a different network than the local host.
• There are no in-path routes defined.
Possible Causes
• Traffic to the LAN is traveling to the WAN router on the way to the LAN, but the router drops the packet.
• Outer connections to clients or servers are routed through the WAN interface to the WAN gateway, and then routed through the SteelHead to the next hop LAN gateway.
• The WAN router is probably dropping the SYN from the SteelHead before issuing an ICMP redirect.
Solutions
To resolve the packet ricochet ICMP redirects issue, perform one of the following actions:
• Change the router ICMP configuration to forward the packet or turn off ICMP redirect.
• Change the in-path gateway to the LAN router.
• Add static routes to LAN subnets through the LAN router.
• Add in-path routes to local destinations to prevent the ICMP redirect and subsequent drop.
Simplified Routing
Simplified routing changes the process used to select the destination Ethernet address for packets transmitted from in-path interfaces.
Simplified routing collects the IP address for the next-hop MAC address from each packet it receives to address traffic. With simplified routing, you can use either the WAN-side or LAN-side device as a default gateway. SteelHead-v sets the correct gateway to use based on where the switch or router sends the traffic, and by associating the next-hop Ethernet addresses with IP addresses. Enabling simplified routing eliminates the need to add static routes when the SteelHead-v is in a different subnet from the client and the server.
Without simplified routing, if SteelHead-v is installed in a different subnet from the client or server, you must define one router as the default gateway and static routes for the other routers so that traffic is not redirected back through the SteelHead. In some cases, even with the static routes defined, the Access Control List (ACL) on the default gateway can still drop traffic that should have gone through the other router. Enabling simplified routing eliminates this issue.
Simplified routing has the following constraints:
• You cannot enable WCCP.
• The default route must exist on each SteelHead in your network. (For detailed information, see the SteelHead Deployment Guide.)
To enable simplified routing
1. Choose Appliance > Connectivity > Simplified Routing to display the Simplified Routing page.
2. Under Mapping Data Collection Setting, complete the configuration as described in the following table.
Control | Description |
Collect Mappings From | Select one of the following options from the drop‑down list: • None - Do not collect mappings. • Destination Only - Collects destination MAC data. Use this option in connection-forwarding deployments. This is the default setting. • Destination and Source - Collect mappings from destination and source MAC data. Use this option in connection-forwarding deployments. • All - Collect mappings for destination, source, and inner MAC data. Also collect data for connections that are not translated using NAT. You cannot enable this option in connection-forwarding deployments. Riverbed recommends that you use this option to maximize the effects of simplified routing. |
3. Click Apply to save your settings to the running configuration.
4. Click Save to save your settings permanently.
Autodiscovery Failure
All traffic passes through the in-path (physical or logical) SteelHead-v due to autodiscovery failure.
Possible Causes
• Cisco PIX 7.x or Raptor firewalls
• Satellite
• Intrusion detection system (IDS) or intrusion prevention system (IPS)
Solutions
• Create a fixed-target rule on the client-side SteelHead:
– Specify the Target Appliance IP Address and its port as 7800 on the opposite SteelHead (in-path without autodiscovery).
• Configure end nodes (firewalls) to allow your probe to pass through.
• Configure the SteelHead IP address as the friendly IP address for IDS or IPS.
• Cisco PIX Firewall IOS 7.0 might block the autodiscovery probe. Some firewall configurations strip TCP options or drop packets with these options. You can keep this configuration and switch to fixed-target rules or change the configuration on the firewall.
Protocol Optimization Errors
SteelHead-v fails to optimize expected protocols.
Solutions
To resolve protocol optimization errors, check:
• that connections have been successfully established.
• that SteelHeads on the other side of a connection are turned on.
• for secure or interactive ports that are preventing protocol optimization.
• for any pass-through rules that could be causing some protocols to pass through the SteelHeads unoptimized.
Resetting a Lost Password
To reset your password, you must have access to the serial console or monitor and must be able to see the entire boot process to perform this procedure:
To reset a lost password
1. Start or reboot the SteelHead.
2. When prompted, press any key to continue.
3. Immediately press E.
A GNU GRUB menu appears.
4. Press the V or ^ keys to select the disk image to boot.
5. Press E.
A GRUB menu appears, with options similar to the following:
-----------------
0: root (hd0,1)
1: kernel /vmlinuz ro root=/dev/sda5 console=tty0 console=ttyS0,9600n8
-----------------
6. Press V or ^ to select the kernel boot parameters entry.
7. Press E to edit the kernel boot parameters.
The CLI displays a partially completed line of text similar to the following:
kernel /vmlinuz ro root=/dev/sda5 console=tty0 console=ttyS0,9600n8
8. The line of text contains two console= entries. Modify this line as follows:
– If you are accessing the SteelHead remotely, delete the console=tty0 entry.
– If you are accessing the SteelHead directly (through a keyboard and monitor connected to the appliance), delete the console=ttyS0 entry.
– At the end of the line, type a space and append with single fastboot. It is important to type a space before single.
Tip: Use the arrow keys to access the entire command line.
9. Press Enter.
10. Press the B key to continue booting.
The system starts.
11. At the command prompt, enter /sbin/resetpw.sh.
There is no password.
12. Type reboot and press Enter to reboot the appliance.
Bypass NIC Log Messages
If you have deployed SteelHead-v with a Riverbed NIC, you can view log messages related to NIC card performance.
Note: The Microsoft Hyper-V hypervisor does not currently support deployment with a Riverbed NIC.
To view log messages regarding the Riverbed NIC
1. Log in to the Management Console using an account with permission to read logs, such as root.
2. Select the Logging tab.
3. On the System Log page, enter vsh_bypass in the Filter dialog box.
4. Click Filter.
Only log messages regarding the NIC appear. For example:
Feb 17 00:56:51 localhost rbtkmod: vsh_bypass_init: probing for hardware bypass
Feb 17 00:56:54 localhost rbtkmod: vsh_bypass_init: control interface lan0_0 mac 00:0C:29:05:5A:3F devnum 5 slave 6 esx_nic vmnic6 esx_mac 00:0E:B6:87:13:E8
Feb 17 00:56:54 localhost rbtkmod: vsh_bypass_init: slave interface wan0_0 mac 00:0C:29:05:5A:49 devnum 6 esx_nic vmnic7 esx_mac 00:0E:B6:87:13:E9
Feb 17 00:56:54 localhost rbtkmod: vsh_bypass_init: management interface bpvm0 mac 00:0C:29:05:5A:67 driver_version 2.0.1.12 device_count 8
Feb 17 00:56:54 localhost rbtkmod: vsh_bypass_init: success, hardware bypass enabled on [ inpath0_0 : lan0_0 ESX nic vmnic6 ESX mac 00:0E:B6:87:13:E8 ; wan0_0 ESX nic vmnic7 ESX mac 00:0E:B6:87:13:E9 ]
Feb 17 00:57:37 amnesiac wdt[6317]: [wdt.NOTICE]: vsh_bypass watchdog init: control interface "lan0_0" (guest mac 00:0C:29:05:5A:3F ESX nic vmnic6 ESX mac 00:0E:B6:87:13:E8 devnum 5) slave interface "wan0_0" (guest mac 00:0C:29:05:5A:49 ESX nic vmnic7 ESX mac 00:0E:B6:87:13:E9 devnum 6), interval "1000", bypass_interval 7000, disable_on_timeout "false”