This chapter provides several best practices to deploy NetIM on scale, including:

For network environment infrastructure that spans multiple geographic regions (GEOs), the deployment strategy for NetIM varies from a single instance. Geographic distribution is a consideration for data collection and use of NetIM.

Geographic dispersion of network devices introduces additional factors. The most common factors in determining the number of NetIM instances and deployment size are:

When operations are continuous (24x7) and systems are accessed from multiple points, deployment of NetIM per region (GEO, principality, country, and so on) provides a more localized system. Where other regions access extra-regional instances, the NetIM systems can be configured and tuned to provide consistent access and performance. One such method is described in the “Supplemental scheduled jobs” section below.

WANs can have variations in performance that impact data collection. Regional data collection can help manage the variations. NetIM scheduled jobs can be adjusted to accommodate the need for longer timeout values or collection retries due to failure or delay. This adjustment is more common in satellite-based WAN connections. Placing a NetIM instance closer to the network devices, interspersed between regions segregated by long-haul WAN, improves the manageability of data collection.

In cases where WAN connectivity is lower bandwidth and higher latency, the network can be supported using polling profiles adjusted to suit the constraints of the network. After initial discovery and polling, the network devices can be classified by polling duration. This classification allows the devices to be separated into groups based on polling duration. The groups can then be assigned individual polling profiles suitable for their average response time and polling duration.

The specifications in the Alluvio NetIM System and Deployment Requirements Guide provide a baseline for NetIM deployment. As the deployment matures or scope increases, the systems may require adjustment or tuning to continue providing optimum performance. The goal in tuning a NetIM deployment is to enhance the completeness of operation, correctness and consistency of configuration, and reliable operation.

The areas of NetIM where configuration refinement (tuning) improves performance commonly include system resources, distribution of processing workload, and meaningful organization of the collected data.

Correct allocation, reservation, priority, and performance of host system and storage resources are components integral to NetIM performance. NetIM relies on rapid communication between its member systems (swarm) to execute functions. Inadequate resources can induce system performance degradation. Non-reserved resources can produce a similar effect where delay in allocation at the host level occurs.

Initial allocation of resources occurs during the deployment of the systems. The resources are specified as part of the virtual appliance image(s). After the initial deployment, the same resources should be reserved at the host level.

Using ESXi as an example, the fields to reserve resources are in the VM settings. The values used in the reservation fields should follow the Deployment Guidelines table in the Alluvio NetIM System and Deployment Requirements Guide.

The Master Schedule job is scheduled to run once per day by default. The Master Schedule job in its default configuration performs data collection, maintenance, topology inference, and other functions.

As a NetIM deployment increases in size and scope, the length of time to execute the Master Schedule job increases. In extreme cases, the volume of primary data collection, parsing, and import can induce a significant load on the NetIM Core and Workers. The duration of the Master Schedule job should be monitored for duration and impact. The statistics for the duration of the jobs are contained in the service logs and require some analysis. The analysis includes the operations of multiple service components of the Master Schedule Job.

Additional jobs can be added to provide specific functions. The jobs can be scheduled to operate automatically or unscheduled. Unscheduled jobs are initiated manually using the Run button for a respective entry in the Scheduled Jobs table.

The primary data collection load can be distributed by adding and configuring supplemental scheduled jobs. The supplemental jobs allow primary data collection (configuration from the command line, WMI, and SNMP —separate from metric polling) to be scheduled to provide incremental network device configuration data resulting in a lower impact on networks and network devices.

This method of incremental collection uses supplemental scheduled jobs and automated grouping rules.

The Grouping Rules capability provides a method to organize device information into Groups. The groups can be used to define the scope of scheduled jobs. Combining the scheduled job(s) and grouping enables the scheduling of primary (configuration) data collection. Using this method, the primary data collection load can be distributed or segmented. The specific method and schedule may vary. General use would employ groups to define scope for jobs, scheduling to define the starting time of job runs, and selection of job functions providing options to manage job actions or duration.

Grouping Rules can utilize a wide variety of information to organize device data. Each type of data can also be matched or excluded based on rule logic or regular expressions. A NetIM system can support multiple rules determined by the organizational requirements, integration with other Riverbed products, or other management criteria.

This example is a rule that creates a group for devices that have the state of New York, USA, in their location data as determined by the Scope filter. The rule is also selected to run with the scheduled job to maintain the group membership. The overwrite option clears the group with each rule execution.

The group created by the rule is then used to scope the activity of a supplemental job. Multiple jobs can be added and scheduled, and the functions of a job can be enabled or disabled according to the required functions.

This job list includes the Master Schedule as well as supplemental jobs dividing data collection into five separate times and two separate jobs for device access maintenance.

In the example the collection of configuration data is distributed over a longer time frame per day. Additional settings in each job select the needed functions.

This example shows the basic settings of one supplemental job. The active state, schedule, time zone, recurrence, frequency, and scope (Group) are noted.

Data Collection is enabled with lower concurrent connections for the command-line interface (CLI) and SNMP. The default number of concurrent connections is 200. In this case, the number of connections is lowered to 50. This value imposes less load on the network (lower operating silhouette) and provides adequate connections to the target devices.

The job also has Pre-Collected Files set as inactive, Data Import is enabled, Auto Group is disabled (this runs under the Master Schedule), and Topology Inference is disabled (this runs under the Master Schedule).

The result is that the same daily data collection is completed while distributing the activity across a broader time frame.

Bear in mind this method manages network device command line and SNMP configuration data. SNMP polling for metric collection is managed separately using polling profiles.

In summary, this example collects configuration data from a subset of network devices while being scheduled to occur before the Master Schedule. No discovery or access configuration testing is done. The data is imported but topology inference is performed by the Master Schedule job.

This approach can be used to distribute and schedule job functions to meet scheduling, geographic, or other operational requirements.

For NetIM to discover a device, it must be SNMP accessible and must respond to SNMP ping as defined in NetIM.

The ping response must include:

If the device responds to the ping, additional information about the node is collected to create a lightweight model that contains system, interfaces, memory, and CPU information.

If the device responds to the ping, additional information about the node is collected to create a lightweight model containing system, interfaces, memory, and CPU information.

This section details how to access the discovery results reports, which generates each time the discovery process runs. Use this report to identify the reasons for the polling failure. This report informs how you proceed to the rest of the steps noted in this section.

Did discovery fail, or was the device merged with an existing device? If so, check these:

If the device is accessible and you have confirmed that the device requires SNMP Object Identifiers (OIDs), enable debug on the Adapter Server Process and rediscover the device.

Devices merge if 80% of their IP addresses match a previously discovered device. NetIM considers a device that matches while merging the same device. Therefore, there is no need to add them to its device management information records again.

The information used to determine which access address to set and if there are common addresses for merging is contained in this file:

The Equivalent IP Addresses example below shows the addresses discovered in the devices ipAddrTable. Valid IP Addresses/Ordered IP’s are the accessible addresses that could be used as potential access addresses. The order for accessing addresses selection is Loopback, Highest, and Lowest. If a device has many equivalent IP addresses that match, it can merge them.

Printing Addresses for Node : BETH-CS-MET01.nclab.nbttech.com( Device ID :1626833899430)

IP Address Table Collection Status (Complete, Empty)

s:1626833899430 ---> (true,false)

Access IP Addresses for Node : BETH-CS-MET01.nclab.nbttech.com, Count : 1

a:1626833899430 ---> 10.47.17.16

Equivalent IP Addresses for Node : BETH-CS-MET01.nclab.nbttech.com, Count : 6

e:1626833899430 ---> 10.1.128.14

e:1626833899430 ---> 10.2.10.2

e:1626833899430 ---> 10.44.133.81

e:1626833899430 ---> 10.47.17.16

e:1626833899430 ---> 192.168.90.1

e:1626833899430 ---> 192.168.100.1

Valid IP Addresses for Node : BETH-CS-MET01.nclab.nbttech.com, Count : 3

o:1626833899430 ---> 10.47.17.16

o:1626833899430 ---> 10.1.128.14

o:1626833899430 ---> 10.2.10.2

 

To prevent merging, all identified matching equivalent addresses can be added to "<install_dir>/input/AutoDiscovery/excludeFromDpiSelection.txt".

 

## Addresses specified in this file will not be chosen as an access address for

## a device during Device Discovery.

## Format :

## IP addresses and subnets should be defined in a single line.

## To specify an IP address to be excluded:

## xxx.xxx.xxx.xxx, e.g.:- 192.168.1.1

## To specify a subnet to be excluded:

## xxx.xxx.xxx.xxx/yy, e.g. :- this excludes all IP addresses from 192.168.1.1 to 192.168.1.255

## 192.168.1.1/24

## Additionally, you can also specify ranges of IP Aaddresses to be excluded:

## xxx.xxx.xxx.xxx-xxx, e.g.:- 192.168.1.1-5 excludes all IP Addresses from 192.168.1.1 to 192.168.1.5

 

## By default, we exclude the loop back addresses and all destination address (0.0.0.0)

127.0.0.0/8

0.0.0.0

An alternate solution would be to delete the contents of orderedIP.txt and rediscover the specific devices that were merged. Do not include the device that is getting merged within the rediscovery attempts, or the merge occurs again.

Other key discovery files are found in these locations as well:

Data collection is how NetIM collects command-line interface (CLI), SNMP, WMI-based configuration, and operational data from devices. This information is used to create a virtual representation (a model) of the device in the NetIM database. This model contains the information necessary to build a fully connected virtual network topology.

NetIM models include:

Check the Collected Files view to ensure that all files are collected.

Check data1/riverbed/NetIM/op_admin/tmp/vne/XML for XML generated from parsed CLI data.

Check <install_dir>/log/AdapterLogs. These logs show the line skipped from the CLI output files. If you note a config line that the client claims Riverbed should be parsing, then the command line might have changed and NetIM needs to be updated to parse the new command.

Check the legacy network browser to see if command-line data is modeled:

Check data1/Riverbed/NetIM/op_admin/tmp/vne/Device_MIB_Collection

See the example below.

If key data is missing from the XML, use the MIB Browser to check for the presence of this data on the device. See the example below.

Use the legacy network browser to validate the model.

If you need to look at the device login and CLI transcripts (failures only), look at this directory:

Transcript logging can be enabled for both successes and failures by enabling it in:

Device MIB Configuration Import Errors Report (cause SNMP Collection failure):

Device Connection Diagnosis (cause of CLI Connection failure):

SNMP Engine ID Report (generated by running the command app.sh ENGINEIDREP):

Adapter Logs (show command output lines skipped during command parsing):

NetIM Core contains a legacy graphical console feature that can be launched from the remote desktop or SSH client that contains an X-Windows Server. Run the command app.sh CP to launch it.

Click Event to launch the Event Viewer.

A summarization of these events is also available from the System Event Dashboard in the NetIM UI. Choose Configure > All Settings > (Administer) System Event Dashboard.

Topology Inference is how NetIM uses the collected IP address, neighbor protocol, forwarding, routing, and VPN configuration to infer the interconnectivity between devices. The Link and Connection Inference (LCI) adapter process facilitates these inferences, which runs at the end of the scheduled job.

This list provides the connectivity inferred by Link and Connection Inference:

Key points to remember:

If you have missing links, extra links, and stale links:

Check these logs if you have link issues:

After applying your workaround, you must delete the current links to remove any incorrect links. Link deletion can be performed from the General Settings page or the command line on the core node. The app.sh CLEAN_LINKS command needs to be run from an RDP session or an SSH client supporting X-Windows Server. See the example below.

To use the Legacy Advanced Management Interface, go to:

When attempting to access the Advanced Management screen, a form displays requiring you to enter a valid Support Access Token.

Open a case with Riverbed Support to request a token. After a valid token is entered into the present form and you click Verify Token, you will have full access to the Advanced Management Interface.

Once a valid token is verified within the application, access will be available for one hour. After that time, a new token will need to be obtained.

You can check the SNMP poller schedules in detail by logging in to the NetIM UI and accessing this hidden URL:

Click the icon under the Details column to view the schedule details.

In the schedule task details window that pops up, you can review the device management details and all the metrics scheduled for polling on the device. You can compare this information to the information set in the polling profile and the device manager to confirm whether the polling configuration was pushed to the job service in the swarm.

When troubleshooting an SNMP or WMI polling issue, you may need to capture the traffic. Captures should be performed if you contact Riverbed Support. This information helps debug SNMP/WMI connection and SNMP/WMI response issues.

In addition to the standard logging, if you need to see additional WMI logging information, enable the Debug mode for the PowerShell CLI. In the debug logs, you should see details about the WMI initialization process, the specific WMI commands being called, and potential delays between the command requests and responses.

A component is a pollable object on a device. Check the integrity of device components if you observe lapses in polling unrelated to the device's lack of reachability.

{

{

"id": 36324,

"modelComponentTypeId": 391,

"fieldsValues": {

"metricClass": "IFC_UTIL"

},

"modelComponentParentId": 34005,

"children": null

If there is an issue with some interfaces not polling, search the JSON file for the interface by name. If the interface is present in the file and the metric class, then NetIM has initialized polling on the interface.

Polling initialization does not occur for a metric if the corresponding interface or any other device-level components like CPU, memory, or disk are not present. Perform a packet capture to identify if the device is responding to requests for the metric or use the MIB Browser to query the device to determine if there are any interface or device-level statistics for the failing metric.

If searching the files shows duplicate components, this can lead to metrics not being available in the UI. To clear duplicate components, run these commands from the core node command line to check or delete all duplicates or components from a device.

We strongly recommend that the swarm service be restarted after the components have been deleted. The restart speeds up the update of the component mappings.

Manager

Data Manager

Worker

During deployment of the Swarm Open Virtualization Appliance (OVA), the administrator chooses what role the node being installed into the swarm should have. As described in the system requirements, the number of nodes performing each role varies with the scale of the deployment.

The Alluvio NetIM System and Deployment Requirements Guide document provides the baseline for initial deployment. Often the requirements are adequate for ongoing operation(s). An increase or tuning of resources allows a NetIM deployment to be configured for optimal performance in a wide variety of network sizes, topologies, and types.

When increasing resources for the NetIM VMs, the performance of the resources is as important as the amount of resources added. If NetIM benefits from reserved resources, in cases of host systems where resource contention may be high, NetIM VMs will have high-priority resources available. An increase from Normal to High (or to a custom value) is most effective for memory and CPU resources.

In addition to scaling considerations related to deployment size, in cases where NetIM will have many concurrent users of the web user interface, NetIM operation can benefit from additional Data Managers. Additional Data Managers can provide scaling support for consistent user interface access. Adding a Data Manager increases the processing capacity of NetIM for database and messaging operations.

Determining the need for additional Data Managers can be assessed through two methods:

Workers

The initial number of workers is determined in Alluvio NetIM System and Deployment Requirements Guide. The initial number accounts for the overall scaling of the deployment. Additional considerations include the type of data collected, the number of elements polled, and the frequency of collection. The collection of high-volume data, such as Class of Service metrics, increases the data collection and processing requirement. Where user-defined metrics (custom or specific) are implemented, the type, number, and frequency should also be considered. In each case, Workers can be added to meet growing or changing processing requirements as a deployment develops.

Determining the need for additional Workers is usually indicated in one of two ways:

The information presented in the interfaces addresses customary, well-known values such as CPU and memory. It also provides information on a wide range of operational metrics for the services.