Pins in the fabric

The Nexus series of data center switches by Cisco allow for an extended fabric configuration. By utilising the notion of remote line cards, an engineer can deploy Nexus 2200 series switches as Top of Rack. These top of rack switches allow for data center access for servers. Nexus 5000 series (or 7000) for that matter act as the master. These are what the Nexus 2000 series pin back to.

With the mindset of remote line card now in your head let’s get on the job about pinning one. This exercise will show you how to pin a Nexus 2248TP to a Nexus 5548UP switch. The term FEX is used and is short for Fabric Extenders, referring to the Nexus 2200 series.

Screen Shot 2013-02-16 at 2.40.35 PM

The above diagram shows you how we are going to pin our access layer devices to each switch. Below we will pin our device with the following configuration. I have two 10Gb Twinax cable links connected from Nexus 2248TP up to the 5548UP. I am going to single home each FEX to an upstream Nexus 5548. Aligning even-numbered FEX and odd-numbered FEX to their own upstream 5548 allows for rack redundancy in the situation of a failure. Servers has a choice to be connected to both FEX’s allowing diverse paths. The above deployment would suit three Racks worth of switches for fabric redundancy. You can dual home your FEX devices and Tony Mattke goes in depth about design considerations.

Ensure your Nexus 5000 device is powered on and the Nexus 2200 device is not before continuing.

LAB-N5548-1(config)# int e1/5-6
LAB-N5548-1(config-if-range)# channel-group 101
LAB-N5548-1(config-if-range)# int po101
LAB-N5548-1(config-if)# switchport mode fex
LAB-N5548-1(config-if)# fex ass 101
LAB-N5548-1(config-if)# exit

The commands are easy enough. Add ethernet 1/5 and 1/6 to channel-group 101. Under the interface of port-channel 101 the switchport mode is not access or trunk but something unique to the NX-OS. Fex. Fex ports tell NXOS to expect a remote line card on this port. Then associate and identify that FEX number 101. Ahh, I have a thing with consistency in my deployments. Port group 101, Fex 101…. You get the idea! 🙂 So let us confirm what is happening now.

LAB-N5548-1# sh fex
  FEX         FEX           FEX                       FEX
Number    Description      State            Model            Serial
------------------------------------------------------------------------
---       --------             Connected     N2K-C2224TP-1GE   SSI163504DR

Okay so a downstream device is connected. This is good. The serial number matches up which is handy. Let us check the detailed output.

LAB-N5548-1# sh fex det
FEX: 101 Description: FEX0101   state: Image Download
  FEX version: 4.2(1)N2(1) [Switch version: 5.2(1)N1(1)]
  FEX Interim version: 4.2(1)N2(0.51)
  Switch Interim version: 5.2(1)N1(1)
  Module Sw Gen: 12594  [Switch Sw Gen: 21]
  post level: complete
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/5
  FCoE Admin: false
  FCoE Oper: true
  FCoE FEX AA Configured: false
  Fabric interface state:
    Po101 - Interface Up. State: Active
    Eth1/5 - Interface Up. State: Active
    Eth1/6 - Interface Up. State: Active
Logs:
02/04/2009 02:58:28.251005: Module register received
02/04/2009 02:58:28.251699: Image Version Mismatch
02/04/2009 02:58:28.252053: Registration response sent
02/04/2009 02:58:28.252224: Requesting satellite to download image

Look at that. The FEX is downloading and installing a newer software from the Nexus 5000. I didn’t expect this to be running its own software by itself. Due to the nature of the Nexus 2000 requiring an upstream 5000 I naturally through it would be “boot from SAN” styled power on. Note that the 4.2 code is moving on up the parents 5.2. Upgrades! Time to grab a coffee whilst I wait.

LAB-N5548-1# sh fex det
FEX: 101 Description: THO-EST-FEX-P2-101 state: Online
FEX version: 5.2(1)N1(1) [Switch version: 5.2(1)N1(1)]
FEX Interim version: 5.2(1)N1(1)
Switch Interim version: 5.2(1)N1(1)
Extender Serial: SSI163504DR
Extender Model: N2K-C2224TP-1GE, Part No: 73-13373-01
Card Id: 132, Mac Addr: 88:75:56:c0:68:02, Num Macs: 64
Module Sw Gen: 12594 [Switch Sw Gen: 21]
post level: complete
pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/5
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po101 - Interface Up. State: Active
Eth1/5 - Interface Up. State: Active
Eth1/6 - Interface Up. State: Active
Fex Port State Fabric Port
Eth101/1/1 Down Po101
Eth101/1/2 Down Po101
Eth101/1/3 Down Po101
Eth101/1/4 Down Po101
Eth101/1/5 Down Po101
<snip>
Logs:
02/04/2009 02:58:28.251005: Module register received
02/04/2009 02:58:28.251699: Image Version Mismatch
02/04/2009 02:58:28.252053: Registration response sent
02/04/2009 02:58:28.252224: Requesting satellite to download image
02/04/2009 03:07:32.676640: Image preload successful.
02/04/2009 03:07:33.816624: Deleting route to FEX
02/04/2009 03:07:33.825731: Module disconnected
02/04/2009 03:07:33.826383: Module Offline
02/04/2009 03:07:33.846159: Deleting route to FEX
02/04/2009 03:07:33.853588: Module disconnected
02/04/2009 03:07:33.854835: Offlining Module
02/04/2009 03:08:28.252593: Module timed out
02/04/2009 03:08:51.563834: Module register received
02/04/2009 03:08:51.565068: Registration response sent
02/04/2009 03:08:51.782832: Module Online Sequence
02/04/2009 03:08:56.416394: Module Online

LAB-N5548-1#

Note the state has change from Image Download to Online. This is noted also by the FEX log at the bottom. Looking also at the output the FEX has added interfaces with a slot aligned with the FEX number. I have cut some out for brevity but Eth 101/1/1-24 now exist! You can pin-up to 12 FEX’s to a Nexus 5000 series. You can appreciate the flexibility this gives opposed to chassis switches. It is important to take into consideration failure domains and what affects this on uplink bandwidth. This allow for great expansion, flexibility with different access layer mediums, and ease of management.

 

Leave a Reply

Your email address will not be published. Required fields are marked *


*