I have access to a rather large OpenStack deployment. This environment I am a tenant and have standard rights assigned to me by my Administrator. For the purpose of the exercise I am going to show how to enable mac-learning for the Open vSwitch under NSX-MH. It will also demonstrate how one can nest workloads on an OpenStack powered environment. This environment runs OpenStack across KVM and vSphere. It is powered by NSX-MH.
There are three things that I will talk about in this post for ground floor switching success with Nested workloads on NSX-MH driven by OpenStack.
Neutron
When creating and attaching an Instance to a Network, Neutron is responsible for creating the MAC address and assigning the IP address. Order of operations is Neutron generates MAC address and assigns it. Neutron subsequently creates and uses an IP address from the subnet ranged assigned to the network the Instance is being attached to. After all, Neutron is the source of truth when it comes to IP address and MAC creation. It takes precedence. During this process a chosen security-group is associated to the interface, lest default be assigned.
Security-Groups
These are a set of rules that are applied to the OVS logical port. Neutron Security groups maps to an NSX Security Policy which is assigned to a Logical Port on OVS. These have rule sets and permit and deny based on 5-tuple matching – SRC/DST IP, SRC/DST port, and protocol.
Port-Security
Oh, good old Port-Security. Port-security is equivalent to Spoofguard. When Port-Security is enabled it is not possible to change the IP address of an Instance. When an Instance has its IP address changed it will cease its ability to communicate on that network. You cannot remove Port-Security without first removing the Security-Group from a port.
MAC-Learning
Open vSwitch is has a learning ability. It can learn multiple MAC addresses assigned to a logical port. This allows capabilities such as nested virtualisation. This is what is used to utilise resources for lab environments. This has to be enabled on a per interface level. If your instance has been connected to multiple networks then you need to execute the command for each interface. MAC-Learning required for nesting environments. VMkernel interfaces have their own MAC’s as does nested VMs. Learning tables are enabled in OVS for the particular port. This allows the software switch to act and behave like a normal switch.
I’m just burning doing the Neutron dance
With what we know, I am going to prepare my Instances within my Project for MAC-Learning, disable security-groups and port-security while I am at it. First cab off the rank is my Storage appliance.
I am going to list all my workloads.
I can expose just what I want with a simple grep.
The output I have is as follows
- Instance ID – a81c8bbe-ae5c-4227-bbba-d3c60f2684a2
- Instance Name – storage-a81c8bbe-ae5c-4227-bbba-d3c60f2684a2
- Status – ACTIVE
- Power State – Running
- Network/IP = Backend_Network=10.0.1.3
With this information I can further run commands to get the interface_ID. By using the Instance ID I can extract further information.
The ID is what we are after for this instance. We can see here also the mac-address assigned by Neutron to the Instance.
Port updated. To explain what is going on here – no-security-groups <id> is a Neutron command. Neutron processes the removal of the Security Group. Everything after the — such as –port-security_enabled=false –mac_learning_enabled=true are commands that are ignored by Neutron. They are received by NSX and processed accordingly.
The Result
The images stored within my Glance repository having static IP addresses there is a problem. There is a disconnect to what IP an Instance has to what Neutron programmed the OVS port. Passing the ability to disable Port-Security allows Neutron to learn what IP address an Instance has, irrespective of what was initially programmed by Neutron.
The subsequent command of enabling Mac-learning allows me to have a nested ESX host.
Rinse and Repeat
So Nick Bradford, my Solution Architect locally showed me a neat way to do this faster.
[email protected]:~$ export INST=9d66ed34-1076-4f71-9300-e98ac29a8f78 [email protected]:~$ for port in `neutron port-list --device-id=$INST | awk '{print $2}' | grep -v id`; do neutron port-update --no-security-groups $port -- --port-security_enabled=false --mac_learning_enabled=true; done Updated port: 2f567a54-7522-4de7-9cc9-50ceb29dc8bd Updated port: d860c3a2-ad09-45e1-8a17-360db3347da8 Updated port: eeb71acf-7f3c-4a13-9fea-b298b08c0228
This for loop allows me to take an Instance ID and loop it through all active interfaces for the instance. All I need to do is go through my required Instances to find out the relevant workloads to have the above for loop applied to. After I find the Instance ID, I update the INST environment variable and re-run the loop.