Skip to content

Neutron Networking

The focus of UnderStack is on on delivering bare metal systems while wanting to provide a cloud like experience there are a number of SDN functions that must be implemented. By utilizing OpenStack Neutron as the user facing API many of these features can be achieved.

To enable this we are using the following plugins/features of Neutron:

The physical network design for each site is a leaf/spine configuration so to best support this we use the VXLAN type driver with VLANs on all the leaves bound to the VXLAN VNI. In this configuration there is only one subnet across the network, in Neutron this is called L2 Adjacency. This model has been used by networking vendors such as Arista, Juniper, and Cisco in their own ML2 mechanism driver. However full support for this has been lacking upstream so we have developed our own mechanism driver as we explore the best approach. Traditionally Ironic (baremetal) based Neutron deployment have utilized the networking-generic-switch mechanism however as it stands today it only supports VLAN and it's templating capabilities are not sufficient for our needs. We hope to eventually create a generic mechanism which can be contributed back. Another limitation that we are aware of is in the VXLAN type driver itself, the Neutron team designed this type as an overlay style VXLAN system while we are focused on an underlay or EVPN-VXLAN style system. We are actively working with upstream on how to best include this use case.

Mapping Leaf/Spine to Neutron Networks

In a Leaf/Spine fabric, VXLAN VNIs are used to create virtual network segments that run over the IP underlay. On the leaf switches, traditional VLANs connected to physical assets are typically mapped to specific VNIs to provide connectivity across the fabric.

VNIs

An available pool of VNIs is defined by creating a VXLAN network segment range with the same name as the fabric on which the VNIs will reside.

Note

This is one of the places that the underlay vs overlay VXLAN disconnect rears its head. Neutron assumes that all VXLAN participants will be able to reach each other via the tunnel configuration. However it is possible to have multiple physically separate fabrics but Neutron does not allow the VXLAN type to have a physical_network value like VLAN networks.

Provider networks and self-serviced tenant networks are allocated VNIs from this range.

VLANs

For each leaf pair on the fabric a VLAN network segment range is created with the physical_network value set to their pair name. For example, the name of the rack they serve could be used.

Connecting a Server to a Network

When a server needs to establish a connection to a network, Ironic takes all the baremetal ports assigned to that server and compares each baremetal port's local_link_connection and physical_network attributes to the desired network to determine the correct port to use. This process is documented is documented in Ironic's Networking Guide as VIF Attachment. Changes to this process are coming in a forthcoming Ironic spec for dynamic port attachment.

If the VNI that is associated with the VXLAN network is not already mapped to a VLAN on the leaf pair where the server is being connected then there will not be a physical_network match to a segment in the network and one of the baremetal port's of the server. In this case we will not have a VLAN segment, so we will allocate a new VLAN in the correct leaf pair by utilizing the physical_network of one of the baremetal ports to create a VLAN segment and attach it to the VXLAN network. The mechanism is then responsible for then ensuring the switch configuration is applied. The code then re-attempts this operation and this time finds a match and is able to use it.

For more technical details on this operation see the Bind Port section.

Routing Network Traffic

Networks by default are isolated from each other and do not support any data ingress or egress by default. To route traffic between networks and to provide ingress and egress, Neutron provides router support. Neutron router support is implemented by a plugin. In the case of UnderStack, since OVN is being utilized the OVN L3 plugin is used. This plugin provides for the ability to define different router flavors so that more than just OVN can be used to route traffic. Router flavor plugins can be crafted to encompass physical devices or virtual devices upon which other features can be provided.

Routers operate the same as baremetal servers. A VLAN must exist on the leaf pair where the virtual or physical router is being served from so that traffic can be handled.

A View from the Neutron API/CLI

First we'll create a self-serviced tenant network with the following:

openstack network create milantest
# Fictional output because this network was already created and also
# had a subnet created for it. We've also got additional permissions
# to view the provider information to help with the explanation.
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2025-02-12T18:32:40Z                 |
| description               |                                      |
| dns_domain                | None                                 |
| id                        | 783b4239-7220-4a74-8253-415539469860 |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | None                                 |
| is_vlan_qinq              | None                                 |
| is_vlan_transparent       | None                                 |
| l2_adjacency              | True                                 |
| mtu                       | 9000                                 |
| name                      | milantest                            |
| port_security_enabled     | False                                |
| project_id                | d3c2c86bdbf24ff5843f323524b63768     |
| provider:network_type     | vxlan                                |
| provider:physical_network | None                                 |
| provider:segmentation_id  | 200004                               |
| qos_policy_id             | None                                 |
| revision_number           | 2                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 6f8e3a32-c7a7-4354-808f-75800b21efcf |
| tags                      |                                      |
| updated_at                | 2025-04-25T12:40:34Z                 |
+---------------------------+--------------------------------------+

From the northbound side of OVN we see the following about this network:

ovn-nbctl show
# snip extra output
switch 9077901b-2a7b-46ed-a012-59bcce9a4da3 (neutron-783b4239-7220-4a74-8253-415539469860) (aka milantest)
# snip extra output

You will see that a virtual switch is created with the same name and its ID matches the network ID.

Attaching servers to networks

Now there's a network that can be used to attach a server to. We'll go ahead and assume a server was built and attached to the network.

We can see how this server got connected to the network by looking at the segments.

openstack network segment list --network milantest
+--------------------------------------+------------------+--------------------------------------+--------------+---------+
| ID                                   | Name             | Network                              | Network Type | Segment |
+--------------------------------------+------------------+--------------------------------------+--------------+---------+
| 5ab3339d-ae44-4f45-9293-7b41a83bf473 | None             | 783b4239-7220-4a74-8253-415539469860 | vlan         |    1800 |
| 78be9792-cf21-4c5e-8432-bd83f0830763 | None             | 783b4239-7220-4a74-8253-415539469860 | vxlan        |  200004 |
+--------------------------------------+------------------+--------------------------------------+--------------+---------+

openstack network segment show 5ab3339d-ae44-4f45-9293-7b41a83bf473
+------------------+--------------------------------------+
| Field            | Value                                |
+------------------+--------------------------------------+
| created_at       | 2025-04-29T13:21:31Z                 |
| description      | None                                 |
| id               | 5ab3339d-ae44-4f45-9293-7b41a83bf473 |
| name             | None                                 |
| network_id       | 783b4239-7220-4a74-8253-415539469860 |
| network_type     | vlan                                 |
| physical_network | f20-2-network                        |
| revision_number  | 0                                    |
| segmentation_id  | 1800                                 |
| updated_at       | 2025-04-29T13:21:31Z                 |
+------------------+--------------------------------------+

Now we can check the ports to confirm that this segment exists to provide connectivity to this server.

openstack port list --network milantest
+--------------------------------------+-----------------+-------------------+--------------------------------------------------------------------------------+--------+
| ID                                   | Name            | MAC Address       | Fixed IP Addresses                                                             | Status |
+--------------------------------------+-----------------+-------------------+--------------------------------------------------------------------------------+--------+
| 47bb4c37-f60d-474f-8ce5-c7c1d9982585 | trunk_parent11  | 14:23:f3:f5:22:b0 | ip_address='192.168.100.170', subnet_id='6f8e3a32-c7a7-4354-808f-75800b21efcf' | ACTIVE |
+--------------------------------------+-----------------+-------------------+--------------------------------------------------------------------------------+--------+

openstack baremetal port list --address '14:23:f3:f5:22:b0' --fields physical_network internal_info
+------------------+----------------------------------------------------------------+
| Physical Network | Internal Info                                                  |
+------------------+----------------------------------------------------------------+
| f20-2-network    | {'tenant_vif_port_id': '47bb4c37-f60d-474f-8ce5-c7c1d9982585'} |
+------------------+----------------------------------------------------------------+

You will see in the last output that the tenant_vif_port_id matches the ID of the Neutron port when we shows the ports on the network.

This process will be repeated for every server that is connected to the network. If a server is connected to a leaf with an existing segment then an additional VLAN will not be consumed.

Similarly on the northbound of OVN we will see the port appear on the virtual switch.

ovn-nbctl show
# snip extra output
switch 9077901b-2a7b-46ed-a012-59bcce9a4da3 (neutron-783b4239-7220-4a74-8253-415539469860) (aka milantest)
    port 47bb4c37-f60d-474f-8ce5-c7c1d9982585 (aka trunk_parent11)
        type: external
        addresses: ["14:23:f3:f5:22:b0 192.168.100.170"]
# snip extra output

Once again the naming and the IDs match up with the port as it exists in Neutron to aid debugging.

Attaching routers to networks

Attaching a router to a network operates similarly to attaching servers except that the router port will be trunked to our Neutron Network node.

Firstly we'll create a router.

openstack router create puc-908
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2025-05-01T15:13:06Z                 |
| description               |                                      |
| enable_default_route_bfd  | False                                |
| enable_default_route_ecmp | False                                |
| enable_ndp_proxy          | None                                 |
| external_gateway_info     | null                                 |
| external_gateways         | []                                   |
| flavor_id                 | None                                 |
| ha                        | False                                |
| id                        | 85533c29-d1f1-42f8-a133-d15099318f3a |
| interfaces_info           | []                                   |
| name                      | puc-908                              |
| project_id                | d3c2c85bdbf24ff5843f323524b63768     |
| revision_number           | 1                                    |
| routes                    |                                      |
| status                    | ACTIVE                               |
| tags                      |                                      |
| tenant_id                 | d3c2c85bdbf24ff5843f323524b63768     |
| updated_at                | 2025-05-01T21:09:19Z                 |
+---------------------------+--------------------------------------+

We'll see this object get created in OVN as well.

ovn-nbctl show
# snip extra output
router 81a34be1-bbb3-4ae4-8d3e-d9b7bf3992b4 (neutron-85533c29-d1f1-42f8-a133-d15099318f3a) (aka puc-908)
# snip extra output

The name and the ID continue to match up with data inside of Neutron. Now we can attach our network's subnet to the router.

openstack router add subnet puc-908 6f8e3a32-c7a7-4354-808f-75800b21efcf
+---------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Field                     | Value                                                                                                                                     |
+---------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up            | UP                                                                                                                                        |
| availability_zone_hints   |                                                                                                                                           |
| availability_zones        |                                                                                                                                           |
| created_at                | 2025-05-01T15:13:06Z                                                                                                                      |
| description               |                                                                                                                                           |
| enable_default_route_bfd  | False                                                                                                                                     |
| enable_default_route_ecmp | False                                                                                                                                     |
| enable_ndp_proxy          | None                                                                                                                                      |
| external_gateway_info     | null                                                                                                                                      |
| external_gateways         | []                                                                                                                                        |
| flavor_id                 | None                                                                                                                                      |
| ha                        | True                                                                                                                                      |
| id                        | 85533c29-d1f1-42f8-a133-d15099318f3a                                                                                                      |
| interfaces_info           | [{"port_id": "10099d3c-0ade-41b9-8a1c-1d50ace4bf22", "ip_address": "192.168.100.1", "subnet_id": "6f8e3a32-c7a7-4354-808f-75800b21efcf"}] |
| name                      | puc-908                                                                                                                                   |
| project_id                | d3c2c85bdbf24ff5843f323524b63768                                                                                                          |
| revision_number           | 4                                                                                                                                         |
| routes                    |                                                                                                                                           |
| status                    | ACTIVE                                                                                                                                    |
| tags                      |                                                                                                                                           |
| tenant_id                 | d3c2c85bdbf24ff5843f323524b63768                                                                                                          |
| updated_at                | 2025-05-01T21:10:39Z                                                                                                                      |
+---------------------------+-------------------------------------------------------------------------------------------------------------------------------------------+

We'll look at the segments and how they're attached.

openstack network segment list --network milantest
+--------------------------------------+------------------+--------------------------------------+--------------+---------+
| ID                                   | Name             | Network                              | Network Type | Segment |
+--------------------------------------+------------------+--------------------------------------+--------------+---------+
| 059fd287-4fd1-446f-a506-e0ed9276f67d | None             | 783b4239-7220-4a74-8253-415539469860 | vlan         |    1801 |
| 5ab3339d-ae44-4f45-9293-7b41a83bf473 | None             | 783b4239-7220-4a74-8253-415539469860 | vlan         |    1800 |
| 78be9792-cf21-4c5e-8432-bd83f0830763 | None             | 783b4239-7220-4a74-8253-415539469860 | vxlan        |  200004 |
+--------------------------------------+------------------+--------------------------------------+--------------+---------+

openstack network segment show 059fd287-4fd1-446f-a506-e0ed9276f67d
+------------------+--------------------------------------+
| Field            | Value                                |
+------------------+--------------------------------------+
| created_at       | 2025-05-01T20:45:16Z                 |
| description      |                                      |
| id               | 059fd287-4fd1-446f-a506-e0ed9276f67d |
| name             | None                                 |
| network_id       | 783b4239-7220-4a74-8253-415539469860 |
| network_type     | vlan                                 |
| physical_network | f20-1-network                        |
| revision_number  | 2                                    |
| segmentation_id  | 1801                                 |
| updated_at       | 2025-05-01T20:55:24Z                 |
+------------------+--------------------------------------+

This time we can see a different VLAN is selected on a different leaf. We can confirm this via our ports.

openstack port list --network milantest
+--------------------------------------+-----------------+-------------------+--------------------------------------------------------------------------------+--------+
| ID                                   | Name            | MAC Address       | Fixed IP Addresses                                                             | Status |
+--------------------------------------+-----------------+-------------------+--------------------------------------------------------------------------------+--------+
| 10099d3c-0ade-41b9-8a1c-1d50ace4bf22 |                 | fa:16:3e:10:8f:f1 | ip_address='192.168.100.1', subnet_id='6f8e3a32-c7a7-4354-808f-75800b21efcf'   | ACTIVE |
| 47bb4c37-f60d-474f-8ce5-c7c1d9982585 | trunk_parent11  | 14:23:f3:f5:22:b0 | ip_address='192.168.100.170', subnet_id='6f8e3a32-c7a7-4354-808f-75800b21efcf' | ACTIVE |
+--------------------------------------+-----------------+-------------------+--------------------------------------------------------------------------------+--------+

Inside of OVN we see the following data:

ovn-nbctl show
# snip extra output
switch 9077901b-2a7b-46ed-a012-59bcce9a4da3 (neutron-783b4239-7220-4a74-8253-415539469860) (aka milantest)
    port 47bb4c37-f60d-474f-8ce5-c7c1d9982585 (aka trunk_parent11)
        type: external
        addresses: ["14:23:f3:f5:22:b0 192.168.100.170"]
    port 10099d3c-0ade-41b9-8a1c-1d50ace4bf22
        type: router
        router-port: lrp-10099d3c-0ade-41b9-8a1c-1d50ace4bf22
    port provnet-059fd287-4fd1-446f-a506-e0ed9276f67d
        type: localnet
        tag: 1801
        addresses: ["unknown"]
router 81a34be1-bbb3-4ae4-8d3e-d9b7bf3992b4 (neutron-85533c29-d1f1-42f8-a133-d15099318f3a) (aka puc-908)
    port lrp-10099d3c-0ade-41b9-8a1c-1d50ace4bf22
        mac: "fa:16:3e:10:8f:f1"
        networks: ["192.168.100.1/24"]
# snip extra output

The names and the IDs all match, along with the VLAN ID of the segment where the node running OVN resides.

ML2 Mechanism Operations

Our ML2 mechanism is responsible for the following:

  • creating dynamic VLAN segments on VXLAN networks via port binding operations via bind_port()
  • deleting dynamic VLAN segments on VXLAN networks when ports are removed via delete_port_postcommit()
  • triggering the actual operation to update the leaf/spine devices to provide the connectivity via update_port_postcommit()
sequenceDiagram
    participant Client
    participant NeutronAPI
    participant ML2Plugin
    participant MechanismManager
    participant MechanismDriver

    %% CREATE PORT FLOW
    Client->>NeutronAPI: POST /v2.0/ports
    NeutronAPI->>ML2Plugin: create_port(context)
    ML2Plugin->>ML2Plugin: _create_port_with_binding()
    ML2Plugin->>MechanismManager: bind_port(port_context)
    MechanismManager->>MechanismDriver: attempt to bind (e.g., OVS, SR-IOV)
    MechanismDriver-->>MechanismManager: binding result
    MechanismManager-->>ML2Plugin: port bound
    ML2Plugin-->>NeutronAPI: port with binding details
    NeutronAPI-->>Client: 201 Created + port info

    %% UPDATE PORT FLOW
    Client->>NeutronAPI: PUT /v2.0/ports/{id}
    NeutronAPI->>ML2Plugin: update_port(context)
    ML2Plugin->>ML2Plugin: _update_port_with_binding()
    ML2Plugin->>MechanismManager: bind_port(port_context)
    MechanismManager->>MechanismDriver: rebind or validate existing binding
    MechanismDriver-->>MechanismManager: binding result
    MechanismManager-->>ML2Plugin: updated port context
    ML2Plugin-->>NeutronAPI: port updated
    NeutronAPI-->>Client: 200 OK + updated port info

Bind Port

While bind_port() is a distinct method inside of an ML2 mechanism, there is no direct call for this via the Neutron API. This method is triggered by Neutron based on certain data provided to port creation and update API calls.

bind_port() will be triggered in the following situations:

  • the port has a binding host
  • the port is either unbound or has previously failed to bind