This topic provides information about visibility tools and troubleshooting methods for SteelConnect. It includes these sections:

The dashboard map offers a unified at-a-glance view of your network topology, including a summary of the established tunnels, tunnels that need attention, sites under construction, and new events. It also provides continuous automatic monitoring of network events and site status.

The Google Map displays the sites and tunnels within an organization. Double-click to zoom in.

The dashboard map provides a graphic view of a single full-duplex tunnel. Each tunnel on the dashboard represents both directions.

The dashboard displays the tunnels in the full-mesh network for networks with less than 16 sites. Click a tunnel to view tunnel status. For networks with 16 or more sites, the dashboard displays only problematic tunnels, such as tunnels that are down or congested. For networks with 16 or more sites, the tunnels appear after you click a site marker.

The Tunnel Status tab reports on the tunnel status, shows details about the tunnel source and destination, and includes throughput statistics by QoS class.

The Health Check > Overlay Routes and Health Check > Tunnel Status pages provide more information. For details, see Checking overlay routes by site and Checking tunnel health by tunnel.

The Path Quality tab reports metrics on the path quality, including statistics on latency, jitter, and packet loss by QoS class. For more details on these metrics, see Monitoring path quality.

The Destination Networks tab displays the number of destination networks between the local and remote sites, including both reachable and unreachable networks. You can filter the destination networks by IP address. For more details, see Destination Networks tab.

You can monitor the real-time status of your sites along with the appliance status from the dashboard.

You can also monitor path quality metrics if you’re using path selection. For details, see Monitoring path quality.

This section provides an overview of the SteelConnect monitoring tools, such as tunnel uptime and downtime indicators, NetFlow and SNMP integration, event logging, connectivity tests, and syslog message export to a remote server.

SteelConnect 2.12 includes tools that monitor all SCM events, the underlay routing and reachability, and provide the collection, visualization, analysis and long-term storage of network statistics.

 

 

 

 

This table lists several troubleshooting tools provided in SteelConnect 2.12.

The Show Path tool displays the potential paths for a given destination prefix. It also shows the current connections on the appliance using the paths. Use the show path tool in SCM. See Show Path tool.

Beyond basic routing, these configurations and settings influence the actual path the traffic takes.

The Show Path tool displays the potential paths for a given destination prefix. It also shows the current connections on the appliance using the paths. For details, see the Show Path tool.

The troubleshooting command for SteelHead SD identifies all interfaces reaching SCM. For details, see troubleshoot.

Choose Health Check > Overlay Routes and Health Check > Tunnel Health to view a summary of the overlay tunnels and specific tunnels health. For details, see Checking network overlays.

The Show Path tool displays the underlay routing. For details, see the Show Path tool.

From one of the SteelHead SD appliance partners in a pair, enter this CLI command:

For details, see show ha_info.

Choose Health Check > Routing Tables. The Routing Tables page provides BGP, OSPF, and Forwarding Information Base (FIB) information for each appliance that is managed by SCM. For details, see Displaying underlay ARP tables in the SteelHead SD User Guide.

This section describes three useful ways to monitor your SD-WAN network: SNMP reporting for appliances, the syslog service that network devices use to send syslog messages to an external syslog server, and NetFlow to export flow information to a collector.

The Organization SNMP tab configures an SCM appliance to send SNMP data directly to a network management system (NMS) and export messages to an external server. When you configure these settings at the organization level, you override the global realm settings.

For details on enabling SNMP and configuring direct reporting or remote logging at the realm level, see Enabling SNMP reporting and logging. For a list of SNMP events, see SNMP traps.

When direct SNMP reporting is enabled, all gateways in an organization send SNMP data directly to your network management system. You can also enable direct SNMP reporting for all gateways in a realm.

SCM forwards the configuration to the SteelConnect gateways, and the gateways use this configuration to set up the SNMP agent daemon. The SNMP agent services the GET and WALK requests from the NMS and also generates traps and sends them to the NMS.

To enable SNMP operations (such as SNMP GET and WALK requests or asynchronous traps) from a SteelConnect appliance, you need to provide the NMS interface IP address through which the appliance is reachable.

For mid to large enterprises that require log exporting to a central server while keeping security, compliance, and audit requirements intact, SCM provides remote logging. With remote logging, all system logs from different appliances can be aggregated and stored in a single remote server located within the same enterprise network.

External syslog servers can be modified at the realm or the organization level. Organization-level configuration takes precedence over a realm-level configuration.

When an event occurs, SteelConnect generates a syslog message and places it in a syslog file in the on the local device and, if configured, on a remote device.

After all logs are aggregated on one server, using third-party monitoring tools such as Splunk can make log analysis and reporting simpler. This log aggregation is also useful for archiving logs. You can also forward real-time system messages from appliances to the remote server.

SCM pushes the configuration data to the SteelConnect appliances, and the appliances use this configuration to transport syslog messages to the remote syslog server.

Complete these steps on the local or remote syslog server when using TLS as the authentication process.

When logging is disabled at the organization level, SCM displays the realm level remote server and status.

NetFlow is a network protocol for the collection and monitoring of network traffic flow data generated by NetFlow-enabled appliances. Appliances running SteelConnect 2.12 can be enabled as flow exporters to export network flow information to a flow collector. In the flow exporter role, the appliances aggregate packet information into flows, and then export the flow records to one or more flow collectors using the IPFIX (IP Flow Information Export) protocol. For details on IPFIX, see RFC5101.

The flow collector receives, stores, and preprocesses the flow data.

NetFlow monitors all traffic flows that enter and exit the appliances, including flows on the underlay, overlay, and local flows. These flow records provide a central monitor with the SteelConnect view of the network.

You can enable NetFlow data export to a flow collector at the realm, organization, or appliance level. When NetFlow is enabled for an organization, the organization settings take precedence over the realm settings. For details on configuring NetFlow at the realm level, see Exporting NetFlow data.

SteelConnect supports NetFlow exporting on the SDI-130, SDI-330, SDI-1030, virtual gateways, SDI-2030, SDI-5030, and SteelHead SD appliances.

For details on NetFlow configuration levels, see Exporting NetFlow data.

NetProfiler or third-party collectors can be configured using an IPv4 address or fully qualified domain name. You can configure a maximum of four NetFlow collectors (three collectors from either a realm or an organization and one from an appliance).

You can also configure one NetFlow collector on a single SDI gateway or SteelHead SD appliance. Typically, NetFlow export is enabled at the realm or organization level and you don't need to configure individual appliances. By default, NetFlow is enabled on individual appliances and becomes active when it is also enabled at the realm or organization level.

SteelConnect pushes the NetFlow configuration to all of the appliances in the organization. If NetFlow monitoring is disabled on an appliance, or if an appliance is running SteelConnect 2.11.x or earlier, it will not export NetFlow data.

SCM displays the configured NetFlow collectors, the port the collector is listening on, and the default IPFIX export protocol. Any collectors configured at the appliance level do not appear.

This section describes how to generate, upload, and manage system dump files.

A system dump file contains diagnostic information about the common hypervisor, Control VM (CVM), Routing VM (RVM), and Service VM (SVM) nodes of a SteelHead SD appliance and an SDI-2030 gateway. On an SDI-5030 gateway, the file also contains information about the Host VM (HVM).

This information is helpful when troubleshooting. For more information on SteelConnect components, see SteelHead SD Installation Guide.

The appliance uses secure FTP (SFTP) to upload system dumps securely to the external server.

By default, when you request a system dump, it is uploaded to support.riverbed.com. You can specify an external server to receive the system dump uploads instead of sending them to Riverbed Support.

We recommend that you generate system dumps at your convenience during off-peak hours to minimize demands on your computing environment.

SCM generates the system dump, which can take up to ten minutes depending on the appliance model. The bottom of the page displays statistical data about the latest systen dumps generated for the appliance.

An appliance can store up to five system dumps at a time, but it’s unlikely to ever reach that limit. By default, when you create a system dump through SCM, it uploads to riverbed.support.com or a configured system dump server and deletes it from the appliance. The system dumps stored on the appliance include only those for which a previous upload and delete attempt had failed and those triggered without the options for upload and delete (by entering the sysdump CLI command). For details, see sysdump.

To view more details such as case number (if provided), sysdump name, file size, and more, click View details.

If there are already five system dumps stored on the appliance and you request a new system dump, SCM will attempt to overwrite the existing system dumps. First, it determines if any of the existing system dumps have already been uploaded to a URL and are stored on the appliance only because the subsequent attempt to delete the system dump failed. If it finds a system dump that has already been uploaded, it will delete it and then trigger the new system dump.

If SCM doesn’t find any existing system dumps that have already been uploaded, it displays a message asking you to upload or delete one.

When a SteelConnect gateway or SteelHead SD appliance is unable to reach SCM, we recommend that you access the appliance through its serial console port and enter the troubleshoot command from the Control VM (CVM), which is the default setting, or the hypervisor.

Entering the troubleshoot command from the hypervisor provides status such as CVM image presence, hypervisor bootup, flow switch status, and so on. In the case of a failure, the output includes next steps. For example, if the core resolution DNS test fails, the output suggests that you “Check DNS server health and verify if it is listening on port 53” as the next step. For details, see troubleshoot.

After entering the troubleshoot command, you can write the results into a troubleshooting log and then export the log to a USB drive for analysis by Riverbed support.

The Show Path tool displays the potential paths for a given destination prefix. For active flows, it shows which path a flow is currently using.

The Show Path tool answers these questions:

The Show Path tool is supported on SteelHead SD 570-SD, 770-SD, and 3070-SD appliances and SDI gateways.

The Show Path test displays details for up to 50 active flows. When there are more than 50 flows for this filter, a message tells you that they can’t all be shown. To view up to 100 flows, enter this CLI command (SteelHead SD, SDI-2030 and SDI-5030 gateways only):

For CLI command details, see show path.

For active flows, the output shows the active paths and their possible paths, including:

For an overlay tunnel, the output shows:

For the underlay, the output shows:

The settings from the previous test remain in place the next time you return to the Tools tab.

The Ping and Traceroute tests are useful to determine if a destination is reachable.

If the destination is unreachable, the tests report where the connection failed.

We recommend using the ping or traceroute tests to verify connectivity, followed by a packet capture or an echo test to dive deeper into troubleshooting. See Packet capture and Echo test.

You can request an on-demand packet capture for a gateway and download the capture file from the Tools tab. Capture files contain summary information for every internet packet received or transmitted on the appliance interface to help diagnose problems in the system.

The echo test dives deeper into troubleshooting by testing network connectivity between a client-side and a server-side zone using a TCP port number and the TCP protocol. The echo test tells you if the communication from appliance A to appliance B is crossing the network overlay.

A support package offers appliance status information in a single JavaScript Object Notation (JSON) file that you can view in SCM or download.

Under the Config column, “Up-to-date” indicates that the latest configuration was applied successfully.

Check for firewalls. In most cases, the first provisioning fails due to local firewall restrictions for TCP port 3900 and 3902.

For a gateway, make sure that internet connectivity and a DHCP server are available on the WAN port of the gateway to allow the first provisioning to succeed. While booting up, one green LED of the gateway will glow as long as a connection to SCM was established successfully. Glowing will stop in normal operation mode.

An access point appliance attempts to receive an IP address through DHCP from the zone automatically. Make sure that you have added your SSIDs (choose Wi-Fi) and then assign the SSIDs to sites (choose Wi-Fi > Broadcasts). All access points in a site will broadcast the SSIDs as configured in the Wi-Fi section.

While booting up, two LEDs (green and orange) on the access point blink as long as a connection to SCM was established successfully. Blinking stops in normal operation mode.

If the appliance comes online but the VPN tunnels aren’t established, make sure UDP/4500 inbound is allowed for the appliance.

For troubleshooting solutions, see Troubleshooting Zscaler tunnels.

Separate the servers by a space: for example, 208.67.222. 123 8.8.8.8.

The uplink subnet mask format is xxx.xx.xxx.xx/xx. For example: 121.73.131.14/16.

AutoVPN is enabled by default and provides connectivity between sites on the same organization. In each site there can be only one appliance acting as an AutoVPN endpoint.

The VPN appliance should ideally be the gateway for this site or placed in-path to pass all traffic entering and exiting the site.

If AutoVPN is down, check the uplink status and cabling, and check the tunnel.

Roaming behavior depends on the wireless client’s network adapter. Some manufacturers such as Intel provide documentation regarding roaming behavior:

If the wireless client is roaming inefficiently, check the signal strength and quality first and verify if the issue is persistent on other devices as well.

Yes, you just need to ensure that there is a DHCP service running in the configured access point zone to provide IPs to the wireless client services.

When no gateway is present, internal servers need to provide network services like DHCP.

Recovery mode is a mechanism used by the appliance in an attempt to restore communication to its SCM controller when an uplink is not working, usually due to a misconfiguration.

The WAN ports on the SDI gateway go into recovery mode approximately five minutes after communication is lost to its SCM controller. In recovery mode, the WAN port receives an IP address through DHCP and then uses this IP address to reach SCM. The last working WAN configuration is restored after 15 minutes if a configuration change has broken the SCM connectivity.

If recovery mode does not work, you can also restore the gateway using an off-line USB method. For details, see How do I provision a gateway offline?.

Yes, there are subtle differences. When a SteelHead SD appliance is in recovery mode, it doesn’t automatically restore the configuration. It stays in recovery mode until there’s a configuration change on SCM.

A SteelHead SD appliance in recovery mode attempts to connect to the SCM with the last working configuration and might use a site-level or default DNS server to resolve SCM.