The other week I presented at both Melbourne and Sydney VMware User Group conferences. This was a one day conference that consists of vendors, community, and partner presentation. The focus is on technical content for users by users. I submitted a presentation on Log Insight and NSX which was accepted.
The fact is that most people have seen VMware marketing around NSX – distributed firewall hat provides state based protection around the workload. Wherever it resides the workload has a policy that follows it. This is a shift of enforcement pint from where traditionally the edge of a broad as domain or zone was the first point of enforcement. It is known as micro segmentation and it something that we do pretty well.
As a recovering ASA administrator and one with a curious mind the first thing that springs to mind is the following. If I have an enforcement point on the vNIC around the VM then I need to know what it speaks? What flows have been seen by my traditional firewall? What about those that weren’t seen? This post breaks down the problem space, the distributed firewall architecture, using logs to create security policy, and through a framework based approach provide a repeatable process for the creation of Security Policy.
Logs. Logs, logs, and more logs.
Logs for a long time have been created by area boundaries. These have been on Firewalls that reside between zones or at the Data Centre edge. This DC Firewall provides the ability to see and filter traffic based on what traverses between zones. It can log and act on what is seen.
If someone needed to provide the ability to block traffic east and west then there was a black and white solution. Block everything with private VLANS or permit everything. Isolated ports talking to promiscuous ports or community to promiscuous. The reality is there are many scenarios that require some communication east west.
With the Distributed Firewall it is possible to provide vNIC level enforcement. This enforcement gives administrators the ability to apply rule sets on the vNIC around the workload itself. That provides the ability to see traffic as it both ingresses and egresses the Virtual Machine. Logs can be generated to DFWPKT.log in /var/log on each vSphere host.
Before defining an approach or looking to centralise logs it is worth digging a little deeper into how the Distributed Firewall is architected. Rules are created by vCenter Server or the NSX Manager API and stored on NSX Manager. They are then pushed down and filtered based on the Applied To field by VSFWD (vShield Firewall Daemon) on each vSphere host. Based on the applied to the object is resolved back to VM ID to IP addresses. Based on the VM-ID the rule is applied to all vNICs for that VM. If there are multiple vNICs – it is possible to isolate a particular vNIC. The VSIP module will push the rules into what is known as an IOChain. The particular IOchain or slot of focus is slot 2 – VMware-SFW. VMware Stateful Firewall is where enforcement occurs.
- Slot 0 is Esxi-firewall. It is the stateless ACL module which provides rules at the VDS level. It is programmed via vCenter APIs.
- Slot 1 is the SW-SEC module. Switch Security provides Layer 2 learning for VM IP addresses through ARP and DHCP mechanisms. Its learning provides the ability to deliver Spoofguard functions with and without VMtools operating.
- Slot 2 is the VMware-SFW module. This is what the VSIP process will pass rules into when it learns rules from NSX Manager.
There are 16 IOchains or slots on the vNIC. VMware reserve the first and last four for their own use. The slots 4-11 are for partners integrated into the NetX ecosystem.
With an understanding of how the vNIC DFW works the next step it to begin segmenting. Using Security Groups to group ‘like’ workloads together it is possible to start seeing where all this is going. By using security groups around like workloads it is now feasible to see what is both leaving a security group and what is traversing a security group. This is the same for SGTSAPP and DB.
Using a rule that states any source, any destination, any services, allow and log, and finally applying to a Security Group allows us to see all traffic regardless of destination.
The SGTSBOOKS Security Group will include WEB, APP and DB. This inclusion means that an individual security group and represent the entire application opposed to three individuals. It also will be used later for a special rule!
Distributed Firewall Tags are an arbitrary string that can be applied to any DFW rule. This string is appended to any log generated by that rule. This provides a human friendly context to what could be constituted as meaningless information in the logs. Some of the MO-REF ID’s that vCenter uses are not overly friendly. This string can also be searched against in any log aggregation platform. This will be used for the context of searching for logs later on.
The tags used for each Security Group are the Security groups name. This SGTSWEB, SGTSAPP, SGTSDB, and SGTSBOOKS.
Visualising rules is part of the ongoing learning of the applications rules. It provides insight based on what is talking to the destination but also what is originating from it. In this example there are a number of RegEx’s being used to create a query. This query is using Source, firewall protocol, and destination IP and port fields within a log and building a Pie Chart.
You can see the source 172.16.1.1 talking to both 10.0.1.11 and 12 on port 80. There is also communication from 10.0.1.11 and 12 to 172.16.1.6 on port 80 too!
That means the Web-VIP which is on 192.168.100.193 actually translates is traffic to the Web tier to 172.16.1.1. The Web VMs are talking to the App VIP on 172.16.1.6.
These logs are pulled from what is seen on the vNIC. As a result we know what is exactly being spoken by the application. That means traffic flow should be outside translated to VIP – VIP to Web tier, Web Tier to App Tier VIP and onwards through to the App Tier and DB. This means for objects that are not a vCenter constructs such as a load balancer VIP it can be represented by an IPset. The destination for the first rule will be a Security Group which represents the Web Tier. We know port 80 is being used and HTTP is this service. The Allow and log is allowed and the applied to field used to restrict the scope of the rule to just the relevant workloads. This is repeated for all logs seen under each respective tag. A deny all is used against the SGTSBOOKS to ensure per application micro-segmentation. This ensures that if mistakes are made then the failure domain is relevant to the application. This is due to the members of SGTSBOOKS being nested Security Groups – Web, App, and Db!
This dashboard represents all the logs and we are seeing across the Security Groups Web, App, DB and Books. It shows the hits we are expected against the relevant tiers. We also see spurious attempts by addresses in the 192.168.100.x address space attempting to reach the application. It is possible to generate emails based on events over a period of time, vCenter alarms which can be acted on, or reports. This helps when monitoring an application to ensure that the application is permitting all legitimate flows.
This represents a sound and repeatable method to applying Micro Segmentation to any workload. I did not cover it all off here but if you want to learn more flick through my entire VMUG user conference 2016 presentation below on slide share or watch the video. My dulcet Dingo tones should help whittle away the minutes.
You can view the presentation below via SlideShare.
You can view the recording of the presentation embedded below via YouTube.