The Nexus series of data center switches by Cisco allow for an extended fabric configuration. By utilising the notion of remote line cards, an engineer can deploy Nexus 2200 series switches as Top of Rack. These top of rack switches allow for data center access for servers. Nexus 5000 series (or 7000) for that matter act as the master. These are what the Nexus 2000 series pin back to.
With the mindset of remote line card now in your head let’s get on the job about pinning one. This exercise will show you how to pin a Nexus 2248TP to a Nexus 5548UP switch. The term FEX is used and is short for Fabric Extenders, referring to the Nexus 2200 series.
The above diagram shows you how we are going to pin our access layer devices to each switch. Below we will pin our device with the following configuration. I have two 10Gb Twinax cable links connected from Nexus 2248TP up to the 5548UP. I am going to single home each FEX to an upstream Nexus 5548. Aligning even-numbered FEX and odd-numbered FEX to their own upstream 5548 allows for rack redundancy in the situation of a failure. Servers has a choice to be connected to both FEX’s allowing diverse paths. The above deployment would suit three Racks worth of switches for fabric redundancy. You can dual home your FEX devices and Tony Mattke goes in depth about design considerations.
Ensure your Nexus 5000 device is powered on and the Nexus 2200 device is not before continuing.
LAB-N5548-1(config)# int e1/5-6 LAB-N5548-1(config-if-range)# channel-group 101 LAB-N5548-1(config-if-range)# int po101 LAB-N5548-1(config-if)# switchport mode fex LAB-N5548-1(config-if)# fex ass 101 LAB-N5548-1(config-if)# exit
The commands are easy enough. Add ethernet 1/5 and 1/6 to channel-group 101. Under the interface of port-channel 101 the switchport mode is not access or trunk but something unique to the NX-OS. Fex. Fex ports tell NXOS to expect a remote line card on this port. Then associate and identify that FEX number 101. Ahh, I have a thing with consistency in my deployments. Port group 101, Fex 101…. You get the idea! 🙂 So let us confirm what is happening now.
LAB-N5548-1# sh fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------ --- -------- Connected N2K-C2224TP-1GE SSI163504DR
Okay so a downstream device is connected. This is good. The serial number matches up which is handy. Let us check the detailed output.
LAB-N5548-1# sh fex det
FEX: 101 Description: FEX0101 state: Image Download
FEX version: 4.2(1)N2(1) [Switch version: 5.2(1)N1(1)]
FEX Interim version: 4.2(1)N2(0.51)
Switch Interim version: 5.2(1)N1(1)
Module Sw Gen: 12594 [Switch Sw Gen: 21]
post level: complete
pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/5
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po101 - Interface Up. State: Active
Eth1/5 - Interface Up. State: Active
Eth1/6 - Interface Up. State: Active
Logs:
02/04/2009 02:58:28.251005: Module register received
02/04/2009 02:58:28.251699: Image Version Mismatch
02/04/2009 02:58:28.252053: Registration response sent
02/04/2009 02:58:28.252224: Requesting satellite to download image
Look at that. The FEX is downloading and installing a newer software from the Nexus 5000. I didn’t expect this to be running its own software by itself. Due to the nature of the Nexus 2000 requiring an upstream 5000 I naturally through it would be “boot from SAN” styled power on. Note that the 4.2 code is moving on up the parents 5.2. Upgrades! Time to grab a coffee whilst I wait.
LAB-N5548-1# sh fex det FEX: 101 Description: THO-EST-FEX-P2-101 state: Online FEX version: 5.2(1)N1(1) [Switch version: 5.2(1)N1(1)] FEX Interim version: 5.2(1)N1(1) Switch Interim version: 5.2(1)N1(1) Extender Serial: SSI163504DR Extender Model: N2K-C2224TP-1GE, Part No: 73-13373-01 Card Id: 132, Mac Addr: 88:75:56:c0:68:02, Num Macs: 64 Module Sw Gen: 12594 [Switch Sw Gen: 21] post level: complete pinning-mode: static Max-links: 1 Fabric port for control traffic: Eth1/5 FCoE Admin: false FCoE Oper: true FCoE FEX AA Configured: false Fabric interface state: Po101 - Interface Up. State: Active Eth1/5 - Interface Up. State: Active Eth1/6 - Interface Up. State: Active Fex Port State Fabric Port Eth101/1/1 Down Po101 Eth101/1/2 Down Po101 Eth101/1/3 Down Po101 Eth101/1/4 Down Po101 Eth101/1/5 Down Po101 <snip> Logs: 02/04/2009 02:58:28.251005: Module register received 02/04/2009 02:58:28.251699: Image Version Mismatch 02/04/2009 02:58:28.252053: Registration response sent 02/04/2009 02:58:28.252224: Requesting satellite to download image 02/04/2009 03:07:32.676640: Image preload successful. 02/04/2009 03:07:33.816624: Deleting route to FEX 02/04/2009 03:07:33.825731: Module disconnected 02/04/2009 03:07:33.826383: Module Offline 02/04/2009 03:07:33.846159: Deleting route to FEX 02/04/2009 03:07:33.853588: Module disconnected 02/04/2009 03:07:33.854835: Offlining Module 02/04/2009 03:08:28.252593: Module timed out 02/04/2009 03:08:51.563834: Module register received 02/04/2009 03:08:51.565068: Registration response sent 02/04/2009 03:08:51.782832: Module Online Sequence 02/04/2009 03:08:56.416394: Module Online LAB-N5548-1#
Note the state has change from Image Download to Online. This is noted also by the FEX log at the bottom. Looking also at the output the FEX has added interfaces with a slot aligned with the FEX number. I have cut some out for brevity but Eth 101/1/1-24 now exist! You can pin-up to 12 FEX’s to a Nexus 5000 series. You can appreciate the flexibility this gives opposed to chassis switches. It is important to take into consideration failure domains and what affects this on uplink bandwidth. This allow for great expansion, flexibility with different access layer mediums, and ease of management.