I have been working on deploying some Nexus fabric over the last few months. The standard life cycle comes and passes and as I was sating my want to know about everything I was reading some interesting information. It is well-known the Nexus 2000 devices act as remote line cards for their older 5000 or 7000 brothers. Any layer 2 lookup or transaction is passed upwards to the master device before being forwarded on. This is a standard action you might say; I don’t think you expected this on neighbouring ports.
On neighbouring ports a normal switch such as a 3750 with a standard architecture would perform the lookup and forwarding decision on the switch and forward out another port on the device. With the distributed architecture of the Nexus family the brains of the platform reside in the 5000 or 7000. With the 2000 acting as ToR switch it must pass traffic up towards a 5000 or 7000 to perform lookup and forwarding decisions. This requires some thought. If you are placing a series of VMware hosts onto the access layer fabric some traffic considerations need to be made.
In the example below a VMware device may have two hosts of different clusters. A guest on cluster A may communicate to a guest on cluster B. This would mean traffic would traverse up and down a single uplink port channel. From a design consideration your East West traffic needs to be taken into consideration. The traffic that is intraDC such as vMotion between cluster hosts, Database backups, storage vMotion or DRS.
The above picture shows a bit more balance. Across two 5000’s traffic patterns can be balanced. This would be when VM load distribution is optimal. If an evacuation of a cluster host occurs or LUN abandoning is in progress such patterns need to be catered for. Sometimes you cannot always optimise but situations can arise where a sub optimal situation are for a short period of time.
Understanding the dependencies of your application architecture is paramount to a good East West traffic management. Ask good questions of your applications team. Speak to your storage team. iSCSI and FCoE users in particular. Teamwork is key here. The network is the fabric that binds us and we need to share it. Don’t be a jerk and accuse or belittle. Respect their application and they will respect your network. Just remember, vMotion/Storage vMotion/Unified vMotion respects no one and will use ALL THE BANDWIDTH!