“..the first thing you do when you get NSX is .. do not deploy NSX”

I was recording a PacketPushers podcast this morning with Ethan Banks, Stephen Skinner, and Chris Wahl which was great. It was talking the tips and war stories associated with VMware NSX and deploying it. There was a quote that was fired off early in the podcast and it was as follows

“..the first thing you do when you get NSX is .. do not deploy NSX”.

This is something that I wholeheartedly agree with. VMware espouse that NSX is quick to install, easy to integrate, and can happily work in a brownfield environment. All of that is true. Deploy NSX Manager OVA to your management cluster, point it towards vCenter, and happy days I now have a new inventory item in vCenter called Networking and Security.

The problem lies when people don’t have a design. Like a bull in a china shop and clicking buttons, deploying VTEPs, next next next and random policy application is something that becomes unruly.

What are you building that you will throw over the fence to ops? What was the design? Did you have a design?

I’ve walked into a couple of environments that have seen customers deploy:
* mind-boggling complex security policy (taking the hardware mentality of security) and applying it to NSX not uses
* complex routing architectures uses NSX edges but not distributed routing
* crazy L2 bridging and routing scenarios
* Did I mention security rule crazy ness?

The end result is a customer deploying something very different to what they expected it to be. The end result is the customer not getting full benefits from something they own. The end result is no net benefit moving the function to the hypervisor because they’ve just pushed

Security Policy

Policy is quite the buzzword. It is almost the new ‘SDN’ in terms of hype. Policy this and policy that. Policy is very personal and very much a word that holds most meaning to the individual.

In my world I term policy in the context of security as ‘desired state’. I want to attest a set of rules upon my infrastructure and ensure that they are applied. I want to know that what I express against ‘Web Policy’ is enforced. This has been a dilemma in the past where rules have for a long time been based on 5-tuple matching – the old SRC/DST IP SRC/DST Port and protocol type. The firewall has no idea about the platform or what is running on it other than can talk to on port TCP 3389.

So where does one start thinking about security in the scope of design? With software like VMware NSX you can enforce security rules at the vNIC. This is great. Every VM has a firewall filter protecting it. Think that last statement through. If every virtual machine has a firewall then I need to move beyond 5-tuple matching.

There are three mindsets I discuss with people with regards to security and NSX. They are grouped into the following:

  • Networking – Rules based on IPSet & MACSet
  • Infrastructure – Rules based on vCenter objects (Datastore, VM Name, VM type, Network port (VXLAN/VLAN)
  • Application – Service orientated focusing on Security group membership via Security tags.

Looking at this as a pyramid with networking rule on the bottom this is where we find most of the thinking happens initially. The need to define subnets, enforce them against VM’s and the thought of that alone is overwhelming. Network rules work for ingress and egress for elements that are not within the domain of NSX or vCenter. This could indulge physical objects such as SCADA or end user desktops. If you take the mindset that the industry as awhile has used for other 20 years then you will end up with thousands of firewall rules, overlap, and potential holes in your firewall environment.

Infrastructure rules look to bring administrators and security architects up a level and start expressing their security attestation using relevant objects. Building rules based on networks a workload is connected to is a start. Web-VLAN to App-VLAN. App-VLAN to Shared services-Logical-Switch . Assign a Service Group of protocols. There is a rule that is based on the virtual infrastructure.

You may use the vCenter object ‘Cluster’ and state Deny Cluster-PCI to All other Clusters. This would stop communication to any components on PCI cluster to All other clusters. You may chose to exempt AD-VM and DNS-VM with a Service Group matching AD and DNS ports to PCI cluster. This is one way to allow explicit communication from Cluster-PCI to AD and DNS VM.

Evolving from this customers and architects eventually iterate their environment to the final, Application centric or focused, security model. This is going ‘all in’ on the tools available within NSX. Security Tags, Service Policy, and Security Groups. The anatomy of a service architecture is as follows.

Security Group SG.Internet.Proxy that matches membership based on the Security Tag ST.Internet.Proxy. Applied to SG.Internet.Proxy is a Security Policy named SP.Internet.Proxy. This Security Policy states ‘Source security group’ to destination Proxy VIP (Squid is load balanced!) is allowed on 8080.

This means if a workload is allowed internet access it is tagged ST.Internet.Proxy and gains the ability to connect to the internet via the Squid Proxy. A workload can be the member of numerous tags at any given time. This allows a service based architecture that ensures key-security policies are based on broader groups such as ‘Users’ ‘Admins’ ’HR’ and then smaller service policies can be used between these larger groups.

This way of thinking does take time. This is a fundamental shift in the way the industry as a whole, irrespective of product, is looking at security.

I have built policy out and explored grouping mechanisms here.

So coming back onto design. How do you foresee your ‘micro segmentation’ plan? How do you feel you will carve your applications up? A mix of infrastructure and network rules? What about a mix of infrastructure and application rules? Or swim in the deep (and very awesome end) of Application rules?

Design. Whiteboard. Draw it out. Talk it out. Revise.

Earlier it was discussed there was some horror stories. In each instance a more refined, scalable and automated policy was applied with a high weight – same source and destination but using a more efficient matching and object criteria. This was applied inline, on a per application basis which allowed the removal of the network centric rules.

War Stories

There have been horror stories but we are changing the way we approach many things. After all – you do not know what you don’t know. Luckily there are many was to skin a cat. It is just simply one way results in a fur coat and the other a bloody mess and Dim Sims.

Like any other networking or IT project or work there is strong value in understanding what your desired goal is, the outcomes that are tied to it, and what each milestone along the way means. Just because it is done in software and you can install it by simply clicking next next next doesn’t mean you will get a desirable outcome.

Remember – Prior preparation prevents piss poor performance!

CISCO VIRL : Terminal Multiplexing

Cisco VIRL : Terminal Mutliplexing

There is a built-in console with VIRL. It allows access to the CLI of the network devices that you deploy so you can administer them. In previous build there were some clicky click features. Due to the dynamic shared nature of VIRL when you deploy a topology there are large number of ports in a pool it chooses from. If you are constantly changing, restarting, deploying or tweaking your VIRL platform these ports can change quite a bit.

Previously you had to individually open each console and type telnet : from your or right-click on each router and open a terminal in VM Maestro. If you have a large topology this can be frustrating. Let alone the fact you miss out on great features like SecureCRT’s send to all tabs. It is great for running show ip route eigrp on all ten tabs at once!

Feedback requested and accepted

I submitted a feature request email in to the team and I do not think I was alone. The most recent build has brought a feature called Terminal Multiplexing. It allows the ability with one click to open all terminal windows on all devices in the topology.

Screenshot 2014-08-07 12.59.31Open the Terminal Multiplexer.

Screenshot 2014-08-07 12.59.05

Select the routers you want to issue a command on and then type the command in the window.

Screenshot 2014-08-07 13.01.01Confirm the output.

Screenshot 2014-08-07 13.01.01(2)

As you can see it did not send anything to a device it should not have.


The dynamic nature of port bindings (whilst sequential in my topology) made it too time consuming to adjust my SecureCRT ports all the time. Using the maestro client is fine for something like this. More time learning new things than preparing the lab. It is great to see feedback incorporated into software quickly. Keeps customers happy and developers have feedback from the field. The joy of software is that changes can be easily made! Good work team VIRL.