SteelHead™ Deployment Guide : Storage Area Network Replication : Best Practices for SAN Replication Using Cisco MDS FCIP
  
Best Practices for SAN Replication Using Cisco MDS FCIP
This section describes the key concepts and recommended settings in the MDS. This section includes the following topics:
  • FCIP Profiles
  • FCIP Tunnels
  • Configuring a Cisco MDS FCIP Deployment
  • Best Practices for a RiOS v5.5.3 or Later with Cisco MDS FCIP Configuration
  • FCIP Profiles
    An FCIP profile defines characteristics of FCIP tunnels that are defined through a particular MDS Gigabit Ethernet interface. Profile characteristics include the:
  • IP address of the MDS Gigabit Ethernet interface that is originating the tunnel.
  • TCP port number.
  • bandwidth and latency characteristics of the WAN link.
  • advanced settings that are typically left to their default values.
  • The MDS enables you to define up to three FCIP profiles per physical MDS Gigabit Ethernet interface. Because a tunnel can be created for each profile, a Cisco MDS switch with two physical Gigabit Ethernet ports can have up to six profiles. Most configurations have only one profile per Gigabit Ethernet interface. Riverbed recommends maximizing the number of profiles configured for each GigE port to increase the total number of TCP connections.
    In the profile setting, the default maximum and minimum bandwidth settings per FCIP profile are 1000 Mbps and 500 Mbps, respectively. You can achieve better performance for unoptimized and optimized traffic using 1000 Mbps and 800 Mbps. This is the rate of the LAN-side TCP entering the SteelHead, so that setting it aggressively high does not have any downside, because the SteelHead terminates TCP locally on the LAN side and the MDS can slow down if it tries to go too fast by advertising a smaller TCP window.
    Similarly, leave the round-trip setting at its default (1000 ms in the Management Console, 1 ms in the CLI), because the network in this context is effectively the LAN connection between the MDS and the SteelHead.
    If you are doing unoptimized runs, configure the bandwidth and latency settings in the MDS to reflect the actual network conditions of the WAN link. These settings improve performance in terms of enabling the MDS to fill-the-pipe with unoptimized runs in the presence of latency.
    FCIP Tunnels
    An FCIP tunnel configuration is attached to a profile and defines the IP address and TCP port number of a far-side MDS to which an FCIP connection is established. You can keep the tunnel configuration default settings, with the following key exceptions:
  • In the Advanced tab of the MDS GUI:
  • Turn on the Write Accelerator option. Always use this option when testing with SteelHeads in the presence of latency. This is an optimization in the MDS (and similar features exist in other FCIP/iFCP products) to reduce round trips.
  • Set the FCIP configuration for each tunnel to Passive on one of the MDS switches. By default, when first establishing FCIP connectivity, each MDS normally tries to constantly initiate new connections in both directions, and it is difficult to determine which side ends up with the well-known destination port (for example, 3225). This behavior can make it difficult to interpret SteelHead reports. When you set one side to Passive, the nonpassive side always initiates connections, hence the behavior is deterministic.
  • FCIP settings allow you to specify the number of TCP connections associated with each FCIP tunnel. By default, this setting is 2: one for Control traffic, and one for the Data traffic. Do not change the default value. The single-TCP mode only exists to maintain compatibility with older FCIP implementations. Separating the Control and Data traffic has performance implications because FC is highly jitter sensitive.
    You can set whether the MDS compresses the FCIP data within the FCIP tunnel configuration. You must disable it when the SteelHead is optimizing. On the MDS the default setting is off. The best practices of common SAN replication vendors (for example, EMC) recommend turning on this setting when there are no WAN optimization controller (WOC) systems present. However, when adding SteelHeads to an existing environment, it should be disabled.
    Configuring a Cisco MDS FCIP Deployment
    The following example shows a Cisco MDS FCIP gateway configuration. Cisco-style configurations typically do not show the default values (for example, compression is off by default, and is not present in this configuration dump). Also, this configuration does not show any non-FCIP elements (such as the FC ports that connect to the SAN storage array and VSANs). This example shows a standard and basic topology that includes an MDS FCIP gateway at each end of a WAN link, MDS1, and MDS2.
    To configure a standard and basic topology that includes an MDS FCIP gateway
    Configure MDS1.
    fcip profile 1
    ip address 10.12.254.15
    tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 800 round-trip-time-ms 1
    fcip profile 2
    ip address 10.12.254.145
    tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 800 round-trip-time-ms 1
    interface fcip1
    use-profile 1
    peer-info ipaddr 10.12.254.45
    write-accelerator
    no shutdown
    interface fcip2
    use-profile 2
    peer-info ipaddr 10.12.254.245
    write-accelerator
    no shutdown
    ip route 10.12.254.32 255.255.255.224 10.12.254.30
    ip route 10.12.254.224 255.255.255.224 10.12.254.130
    interface GigabitEthernet1/1
    ip address 10.12.254.15 255.255.255.224
    switchport description LAN side of mv-emcsh1
    no shutdown
    interface GigabitEthernet1/2
    ip address 10.12.254.145 255.255.255.224
    switchport description LAN side of mv-emcsh1
    no shutdown
    Configure MDS2.
    fcip profile 1
    ip address 10.12.254.45
    tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 800 round-trip-time-ms 1
    fcip profile 2
    ip address 10.12.254.245
    tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 800 round-trip-time-ms 1
    interface fcip1
    use-profile 1
    passive-mode
    peer-info ipaddr 10.12.254.15
    write-accelerator
    no shutdown
    interface fcip2
    use-profile 2
    passive-mode
    peer-info ipaddr 10.12.254.145
    write-accelerator
    no shutdown
    ip route 10.12.254.0 255.255.255.224 10.12.254.60
    ip route 10.12.254.128 255.255.255.224 10.12.254.230
    interface GigabitEthernet1/1
    ip address 10.12.254.45 255.255.255.224
    switchport description LAN side of mv-emcsh2
    no shutdown
    interface GigabitEthernet1/2
    ip address 10.12.254.245 255.255.255.224
    switchport description LAN side of mv-emcsh2
    no shutdown
    Best Practices for a RiOS v5.5.3 or Later with Cisco MDS FCIP Configuration
    Riverbed recommends the following best practices regarding a Cisco MDS FCIP configuration:
  • Enable the RiOS v5.5 or later multicore balancing feature due to the small number of data connections.
  • Use an in-path rule to specify the neural-mode as never for FCIP traffic.
  • Set the always-probe port to 3225 to ensure that MDS aggressive SYN-sending behavior does not result in unwanted pass-through connections.
  • The following table summarizes the CLI commands RiOS v5.5.3 or later with Cisco MDS FCIP.
    Feature
    CLI Commands
    Enable RiOS SDR-M
    When using SteelHead DX, select SDR Turbo for higher throughput or SDR Classic for higher data reduction.
    datastore sdr-policy sdr-m
    Set compression level (LZ1)
    datastore codec compression level 1
    Multicore Balancing
    datastore codec multi-core-bal
    Turn Off Nagle
    in-path rule auto-discover srcaddr all-ip dstaddr all-ip
    dstport “3225” preoptimization “none” optimization
    “normal” latency-opt “normal” vlan -1 neural-mode
    “never” wan-visibility “correct” description “”
    rulenum start
    MX-TCP class covering FCIP traffic
    qos classification class add class-name “blast”
    priority realtime min-pct 99.0000000 link-share
    100.0000000 upper-limit-pct 100.0000000 queue-type
    mxtcp queue-length 100 parent “root”
    qos classification rule add class-name “blast”
    traffic-type optimized destination port 3225
    rulenum 1
    qos classification rule add class-name “blast”
    traffic-type optimized source port 3225 rulenum 1
    Set WAN TCP buffers
    protocol connection wan receive def-buf-size <2*BDP>
    protocol connection wan send def-buf-size <2*BDP>
    Set LAN TCP buffers
    protocol connection lan send buf-size 1048576
    tcp adv-win-scale -1
    Note: tcp adv-win-scale -1 is for RiOS v5.5.6c or later.
    Reset existing connections on startup
    in-path kickoff
    in-path kickoff-resume
    Note: in-path kickoff-resume is for RiOS v6.0.1a or later.
    Never pass-through SYN packets
    in-path always-probe enable
    Change always-probe port to FCIP
    in-path always-probe port 3225
    Increase encoder buffer sizes
    datastore codec multi-codec encoder max-ackqlen 20
    datastore codec multi-codec encoder global-txn-max 128
    FCIP optimization
    protocol fcip enable
    Note: Use only with RiOS v6.0.1 or later.
    DIF header optimization
    protocol fcip rule src-ip <x.x.x.x> dst-ip <y.y.y.y> dif enable
    Replace <x.x.x.x> and <y.y.y.y> with IP address pairs for MDS Gigabit Ethernet ports. For details, see Storage Optimization Modules.
    Note: Use only with EMC VMAX and RiOS v6.0.1 or later.
    Restart the optimization service
    restart
    If you increase the number of FCIP profiles, you must also create separate in-path rules to disable Nagle for other TCP ports (for example, 3226 and 3227).
    Similarly, if you decide to set QoS rules to focus on port 3225 to drive traffic into a particular class, you must create rules for both ports 3226 and 3227.