Learning by doing: Adding routes to Neutron

This post outlines how to add routes to a Neutron router. The outcome of this post will allow the jumphost to access VMs and networks advertised behind the SRX. Working on my lab environment I have some server infrastructure and jump hosts in the network Due to Neutron routing being very plain I could not dynamically peer the SRX with the Neutron gateway.

First time to list my routers in my project

[email protected]:~$ neutron router-list
| id                                   | name            | external_gateway_info                                                       |
| 27d89917-bb77-46c3-95d5-250a259ba304 | public_router   | {"network_id": "083ad060-d6dd-4e49-84e1-c8a2259982ff", "enable_snat": true} |
| 60aefbeb-d2f2-4daf-91b2-6f59391bfee5 | external_router | {"network_id": "083ad060-d6dd-4e49-84e1-c8a2259982ff", "enable_snat": true} |
| a41a761d-9ee1-449d-80be-3ea0f599c4f9 | isolated_router | {"network_id": "083ad060-d6dd-4e49-84e1-c8a2259982ff", "enable_snat": true} |

The router I want to use is the isolated_router. The ID is a41a761d-9ee1-449d-80be-3ea0f599c4f9.

The attached image below shows the rough network environment.



The three networks attached to the Distributed Logical Router are unknown beyond the edge of the SRX. WIN-MGT on the network has no idea of it. It can only see the interface of the SRX in the network. We need to teach the Neutron Router that routes between these two networks about

This can be done with updating the neutron router.

[email protected]:~$ neutron router-update a41a761d-9ee1-449d-80be-3ea0f599c4f9 --routes type=dict list=true destination=,nexthop= destination=,nexthop= destination=,nexthop=
destination=,nexthop= destination=,nexthop=
Updated router: a41a761d-9ee1-449d-80be-3ea0f599c4f9

The result when we look at the Neutron router again is much better.

[email protected]:~$ neutron router-show a41a761d-9ee1-449d-80be-3ea0f599c4f9
| Field                 | Value                                                                       |
| admin_state_up        | True                                                                        |
| distributed           | False                                                                       |
| external_gateway_info | {"network_id": "083ad060-d6dd-4e49-84e1-c8a2259982ff", "enable_snat": true} |
| id                    | a41a761d-9ee1-449d-80be-3ea0f599c4f9                                        |
| name                  | isolated_router                                                             |
| routes                | {"destination": "", "nexthop": ""}            |
|                       | {"destination": "", "nexthop": ""}            |
|                       | {"destination": "", "nexthop": ""}            |
|                       | {"destination": "", "nexthop": ""}            |
|                       | {"destination": "", "nexthop": ""}             |
| status                | ACTIVE                                                                      |
| tenant_id             | c3485cfe92be4f47852db87ca06b4383                                            |

As you can see there is a new field that includes the routes that I have programmed into my Neutron router. I now have connectivity from the network into my networks advertised off the DLR. Between the SRX and the DLR is an ECMP fabric.

mgt-lnxjump (                   Tue Jul 28 00:32:10 2015
Keys:  Help   Display mode   Restart statistics   Order of fields
   quit                 Packets               Pings
 Host                 Loss%   Snt   Last   Avg  Best  Wrst StDev
 1.      0.0%   173    0.5   0.3   0.2   4.7   0.3
 2.    0.0%   173    6.6   8.0   1.2  11.6   2.3
 3.        0.0%   173    3.9   4.0   1.1  23.4   2.3
 4. ???
 5.      0.0%   172    7.9   8.7   5.9  22.9   2.1

End to end connectivity. We can see at hop three is the E3 currently passing traffic. If this drops or turns off this hop will be updated with,2 or 4. ECMP is great!

Gotcha: A gotcha of this is that neutron doesn’t add additional static routes each time you execute the command. It will refresh the list. Ensure you don’t forget any else you may have some connectivity issues!

The alternative is to assign host routes under a DHCP scope. This is pretty easy. A host route is a DHCP option passed to an instance on boot that would allow an allocation of pre-defined static routes. This would do that but in my case my instance had spawned and the other instance accessing this environment was actually not a nova instance and therefore not in scope for an IP from my Neutron DHCP Client.

There you are. Connectivity to my remote network. Openstack is pretty powerful!

Enable MAC learning on OVS under OpenStack

I have access to a rather large OpenStack deployment. This environment I am a tenant and have standard rights assigned to me by my Administrator. For the purpose of the exercise I am going to show how to enable mac-learning for the Open vSwitch under NSX-MH. It will also demonstrate how one can nest workloads on an OpenStack powered environment. This environment runs OpenStack across KVM and vSphere. It is powered by NSX-MH.

Screen Shot 2015-05-12 at 8.35.06 am

There are three things that I will talk about in this post for ground floor switching success with Nested workloads on NSX-MH driven by OpenStack.


When creating and attaching an Instance to a Network, Neutron is responsible for creating the MAC address and assigning the IP address. Order of operations is Neutron generates MAC address and assigns it. Neutron subsequently creates and uses an IP address from the subnet ranged assigned to the network the Instance is being attached to. After all, Neutron is the source of truth when it comes to IP address and MAC creation. It takes precedence. During this process a chosen security-group is associated to the interface, lest default be assigned.


These are a set of rules that are applied to the OVS  logical port. Neutron Security groups maps to an NSX Security Policy which is assigned to a Logical Port on OVS. These have rule sets and permit and deny based on 5-tuple matching – SRC/DST IP, SRC/DST port, and protocol.


Oh, good old Port-Security. Port-security is equivalent to Spoofguard. When Port-Security is enabled it is not possible to change the IP address of an Instance. When an Instance has its IP address changed it will cease its ability to communicate on that network. You cannot remove Port-Security without first removing the Security-Group from a port.


Open vSwitch is has a learning ability. It can learn multiple MAC addresses assigned to a logical port. This allows capabilities such as nested virtualisation. This is what is used to utilise resources for lab environments. This has to be enabled on a per interface level. If your instance has been connected to multiple networks then you need to execute the command for each interface. MAC-Learning required for nesting environments. VMkernel interfaces have their own MAC’s as does nested VMs. Learning tables are enabled in OVS for the particular port. This allows the software switch to act and behave like a normal switch.

I’m just burning doing the Neutron dance

With what we know, I am going to prepare my Instances within my Project for MAC-Learning, disable security-groups and port-security while I am at it. First cab off the rank is my Storage appliance.

I am going to list all my workloads.

Screen Shot 2015-05-08 at 2.41.51 pm

I can expose just what I want with a simple grep.

Screen Shot 2015-05-08 at 2.41.37 pm

The output I have is as follows

  • Instance ID – a81c8bbe-ae5c-4227-bbba-d3c60f2684a2
  • Instance Name –  storage-a81c8bbe-ae5c-4227-bbba-d3c60f2684a2
  • Status – ACTIVE
  • Power State – Running
  • Network/IP = Backend_Network=

With this information I can further run commands to get the interface_ID. By using the Instance ID I can extract further information.

Screen Shot 2015-05-08 at 2.41.30 pm


The ID is what we are after for this instance. We can see here also the mac-address assigned by Neutron to the Instance.

Screen Shot 2015-05-08 at 3.16.40 pm

Port updated. To explain what is going on here – no-security-groups <id> is a Neutron command. Neutron processes the removal of the Security Group. Everything after the — such as –port-security_enabled=false –mac_learning_enabled=true are commands that are ignored by Neutron. They are received by NSX and processed accordingly.

The Result

The images stored within my Glance repository having static IP addresses there is a problem. There is a disconnect to what IP an Instance has to what Neutron programmed the OVS port. Passing the ability to disable Port-Security allows Neutron to learn what IP address an Instance has, irrespective of what was initially programmed by Neutron.

The subsequent command of enabling Mac-learning allows me to have a nested ESX host.

Rinse and Repeat

So Nick Bradford, my Solution Architect locally showed me a neat way to do this faster.

[email protected]:~$ export INST=9d66ed34-1076-4f71-9300-e98ac29a8f78
[email protected]:~$ for port in `neutron port-list --device-id=$INST | awk '{print $2}' | grep -v id`; do neutron port-update --no-security-groups $port -- --port-security_enabled=false --mac_learning_enabled=true; done
Updated port: 2f567a54-7522-4de7-9cc9-50ceb29dc8bd
Updated port: d860c3a2-ad09-45e1-8a17-360db3347da8
Updated port: eeb71acf-7f3c-4a13-9fea-b298b08c0228

This for loop allows me to take an Instance ID and loop it through all active interfaces for the instance. All I need to do is go through my required Instances to find out the relevant workloads to have the above for loop applied to. After I find the Instance ID, I update the INST environment variable and re-run the loop.