OVN / Open vSwitch
Debugging OVN / Open vSwitch
To troubleshoot OVN specific issues you'll want to refer to
the OVN manual pages.
To troubleshoot OVS specific issues you'll want to refer to
the OVS manual pages.
The ovn- prefixed commands can be run from the ovn- prefixed pods.
Specifically the north and south DB pods will have the most best data.
The ovs- prefixed commands should be run from the openvswitch pods.
Troubleshooting alert NeutronAgentDown
If you see this prometheus alert, the following steps may help to resolve the issue.
name: NeutronAgentDown
expr: openstack_neutron_agent_state != 1
labels:
severity: P4
annotations:
description: The service `{{$labels.exported_service}}` running on `{{$labels.hostname}}` is being reported as down.
summary: [`{{$labels.hostname}}`] `{{$labels.exported_service}}` down
First, check the neutron agent statuses:
openstack network agent list
+--------------------------------------+------------------------------+--------------------------------------+-------------------+-------+-------+----------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+------------------------------+--------------------------------------+-------------------+-------+-------+----------------------+
| 00a833c1-7395-438c-956d-76a68c363992 | Baremetal Node | b31cd289-c475-481d-b34b-ca52106a9222 | None | XXX | UP | ironic-neutron-agent |
| 0c292b05-73b8-4851-9fca-23343e51d75e | Baremetal Node | 3a4f592b-c5f2-4df2-85dd-a8a810856b55 | None | XXX | UP | ironic-neutron-agent |
| 136f56f2-fc81-482d-b461-261df5d1c59b | Baremetal Node | 113752b7-489a-4206-b2a1-e4fcbed8d6d7 | None | XXX | UP | ironic-neutron-agent |
| 1b2e3104-3596-4544-a535-35a64f2c61cb | Baremetal Node | 61694efc-0834-4b3f-b10d-e7534ece1d7c | None | XXX | UP | ironic-neutron-agent |
| 21ceeba6-4370-4bca-9b90-fc403b9dd325 | Baremetal Node | a1a61c02-7df2-4e5b-b8fe-b0fb115b2885 | None | XXX | UP | ironic-neutron-agent |
| 31c472d9-fde7-4354-95d4-d4a6b287a65f | Baremetal Node | 29fb8908-a225-4bda-a5c7-ea9c8d80df97 | None | XXX | UP | ironic-neutron-agent |
| 391f2b35-1904-4a51-a91a-0b94efbaeae1 | Baremetal Node | 5ae6258c-255e-4d5c-9a6e-2048b914e516 | None | XXX | UP | ironic-neutron-agent |
| 40228bd2-dbed-4efd-99fd-1b25f73c5486 | Baremetal Node | 063fcc5c-5d7d-42cb-8f53-20284eb6e553 | None | XXX | UP | ironic-neutron-agent |
| 483cb4c6-282e-45e0-a2c8-aab8d73251c0 | Baremetal Node | f6be9302-96b0-47e9-ad63-6056a5e9a8f5 | None | XXX | UP | ironic-neutron-agent |
| 67412731-8a9d-4045-9fb3-dc8c74739c0e | Baremetal Node | 6cc75fc1-756a-4b19-bbab-fe8e63eee45b | None | XXX | UP | ironic-neutron-agent |
| 6cb9a276-6d51-4d7f-81b3-75d4822ae9df | Baremetal Node | f6293c81-49ff-40ff-baed-9833f6bf7480 | None | XXX | UP | ironic-neutron-agent |
| 81abb06b-176c-4248-bbcf-690d794c837b | Baremetal Node | de6495fc-3df2-4724-8217-5745d679fad1 | None | XXX | UP | ironic-neutron-agent |
| 90b18cc3-ae2f-45f7-a558-9ac00ca5b280 | Baremetal Node | bfa06d8a-7d2e-4934-aeb5-c9b185b83548 | None | XXX | UP | ironic-neutron-agent |
| 944cf411-5c22-45d7-a057-8b4eff338e7b | Baremetal Node | 0572c9c4-8199-4d34-957f-4c3fb310d557 | None | XXX | UP | ironic-neutron-agent |
| 95807c35-d58c-4a40-bf59-61173a41dcc1 | Baremetal Node | b9f80d94-aa2b-4f0c-ae4a-2f5e5dfaad25 | None | XXX | UP | ironic-neutron-agent |
| a451c2bd-e2fb-4c24-8105-235eb40d8a48 | Baremetal Node | b68bd2fb-8670-4205-b9e9-737de89dfcba | None | XXX | UP | ironic-neutron-agent |
| aa2ce8d4-2f47-4e0a-8b94-428f65fec2e0 | Baremetal Node | 7102d86d-2f7b-4217-8653-d5a8e8957a7c | None | XXX | UP | ironic-neutron-agent |
| ab01ce9a-0b7e-49c8-8c85-ece612e30cbc | Baremetal Node | 048e3b73-9c5e-4727-a36b-a00406212aa0 | None | XXX | UP | ironic-neutron-agent |
| b43072a9-d20c-4800-a29f-d0f761c0cd31 | Baremetal Node | 90b75aae-bd7b-4ad8-98f0-230968738d2c | None | XXX | UP | ironic-neutron-agent |
| c794a5f1-fca9-4862-8841-4577eeaa0d1f | Baremetal Node | 2e32caa9-482a-4ca1-a16b-3dcc164e696c | None | XXX | UP | ironic-neutron-agent |
| e7c44284-7b02-4193-bd8e-ec37684dca57 | Baremetal Node | 61a7f7b9-ec6a-4250-a096-fea2b954d9be | None | XXX | UP | ironic-neutron-agent |
| ec9cf408-355c-4d13-acc3-fa86dc4a99af | Baremetal Node | 4933fb3d-aa7c-4569-ae25-0af879a11291 | None | XXX | UP | ironic-neutron-agent |
| f57317ec-8bb0-47f3-bfbc-31cb4c09a9b1 | Baremetal Node | ce27f4a5-9607-41f4-b48d-f6b5ae88da88 | None | XXX | UP | ironic-neutron-agent |
| 10669d82-03c7-450f-8b0d-32acc75fc987 | OVN Controller Gateway agent | 1327172-hp1 | | XXX | UP | ovn-controller |
+--------------------------------------+------------------------------+--------------------------------------+-------------------+-------+-------+----------------------+
All the agents are reporting as down with XXX in the Alive column. This likely means either the ovn controller is having an issue,
the ironic neutron agent is having an issue, or both of those services are having issues.
Looking at the neutron-ironic-agent logs we don't see any obvious errors:
kubectl logs -f neutron-ironic-agent-764fd4dcd6-szzd5
Defaulted container "neutron-ironic-agent" out of: neutron-ironic-agent, init (init)
+ COMMAND=start
+ start
+ exec ironic-neutron-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini
2025-04-14 15:50:20.721 1 INFO neutron.common.config [-] Logging enabled!
2025-04-14 15:50:20.721 1 INFO neutron.common.config [-] /var/lib/openstack/bin/ironic-neutron-agent version 24.1.1.dev41
But looking at the ovn-controller logs shows a problem:
2025-04-14T15:31:35.834Z|00313|ovsdb_idl|WARN|transaction error: {"details":"Transaction causes multiple rows in \"Encap\" table to have identical values (geneve and \"10.46.100.84\") for index on columns \"type\" and \"ip\". First row, with UUID 859dad94-3322-41ae-af23-893cdc474aba, existed in the database before this transaction and was not modified by the transaction. Second row, with UUID 37a9d58a-4848-4578-bf8b-58da6ab81972, was inserted by this transaction.","error":"constraint violation"}
2025-04-14T15:31:35.834Z|00314|main|INFO|OVNSB commit failed, force recompute next time.
The transaction fails due to duplicate rows, which can occur if the hostname of the ovn-controller changes, amongst other reasons.
We can fix this issue by deleting the old chassis from ovn, restarting the ovn-controller service, and then restarting the
neutron ironic agent service.
Fix ovn data:
kubectl exec -it ovn-ovsdb-sb-0 -- bash
Defaulted container "ovsdb" out of: ovsdb, init (init)
root@ovn-ovsdb-sb-0:/# ovn-sbctl list encap
_uuid : 859dad94-3322-41ae-af23-893cdc474aba
chassis_name : "10669d82-03c7-450f-8b0d-32acc75fc987"
ip : "10.46.100.84"
options : {csum="true"}
type : geneve
root@ovn-ovsdb-sb-0:/# ovn-sbctl chassis-del 10669d82-03c7-450f-8b0d-32acc75fc987
Restart ovn-controller:
Check the ovn-controller logs and the duplicate key error should be gone now.
Once ovn-controller has finished restarting, then restart the neutron ironic agent:
kubectl rollout restart deployment neutron-ironic-agent
deployment.apps/neutron-ironic-agent restarted
Give it a moment to restart and perform re-checks of the agents, and we should see healthy network agents now:
openstack network agent list
+--------------------------------------+------------------------------+--------------------------------------+-------------------+-------+-------+----------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+------------------------------+--------------------------------------+-------------------+-------+-------+----------------------+
| 00a833c1-7395-438c-956d-76a68c363992 | Baremetal Node | b31cd289-c475-481d-b34b-ca52106a9222 | None | :-) | UP | ironic-neutron-agent |
| 0c292b05-73b8-4851-9fca-23343e51d75e | Baremetal Node | 3a4f592b-c5f2-4df2-85dd-a8a810856b55 | None | :-) | UP | ironic-neutron-agent |
| 136f56f2-fc81-482d-b461-261df5d1c59b | Baremetal Node | 113752b7-489a-4206-b2a1-e4fcbed8d6d7 | None | :-) | UP | ironic-neutron-agent |
| 1b2e3104-3596-4544-a535-35a64f2c61cb | Baremetal Node | 61694efc-0834-4b3f-b10d-e7534ece1d7c | None | :-) | UP | ironic-neutron-agent |
| 21ceeba6-4370-4bca-9b90-fc403b9dd325 | Baremetal Node | a1a61c02-7df2-4e5b-b8fe-b0fb115b2885 | None | :-) | UP | ironic-neutron-agent |
| 31c472d9-fde7-4354-95d4-d4a6b287a65f | Baremetal Node | 29fb8908-a225-4bda-a5c7-ea9c8d80df97 | None | :-) | UP | ironic-neutron-agent |
| 391f2b35-1904-4a51-a91a-0b94efbaeae1 | Baremetal Node | 5ae6258c-255e-4d5c-9a6e-2048b914e516 | None | :-) | UP | ironic-neutron-agent |
| 40228bd2-dbed-4efd-99fd-1b25f73c5486 | Baremetal Node | 063fcc5c-5d7d-42cb-8f53-20284eb6e553 | None | :-) | UP | ironic-neutron-agent |
| 483cb4c6-282e-45e0-a2c8-aab8d73251c0 | Baremetal Node | f6be9302-96b0-47e9-ad63-6056a5e9a8f5 | None | :-) | UP | ironic-neutron-agent |
| 67412731-8a9d-4045-9fb3-dc8c74739c0e | Baremetal Node | 6cc75fc1-756a-4b19-bbab-fe8e63eee45b | None | :-) | UP | ironic-neutron-agent |
| 6cb9a276-6d51-4d7f-81b3-75d4822ae9df | Baremetal Node | f6293c81-49ff-40ff-baed-9833f6bf7480 | None | :-) | UP | ironic-neutron-agent |
| 81abb06b-176c-4248-bbcf-690d794c837b | Baremetal Node | de6495fc-3df2-4724-8217-5745d679fad1 | None | :-) | UP | ironic-neutron-agent |
| 90b18cc3-ae2f-45f7-a558-9ac00ca5b280 | Baremetal Node | bfa06d8a-7d2e-4934-aeb5-c9b185b83548 | None | :-) | UP | ironic-neutron-agent |
| 944cf411-5c22-45d7-a057-8b4eff338e7b | Baremetal Node | 0572c9c4-8199-4d34-957f-4c3fb310d557 | None | :-) | UP | ironic-neutron-agent |
| 95807c35-d58c-4a40-bf59-61173a41dcc1 | Baremetal Node | b9f80d94-aa2b-4f0c-ae4a-2f5e5dfaad25 | None | :-) | UP | ironic-neutron-agent |
| a451c2bd-e2fb-4c24-8105-235eb40d8a48 | Baremetal Node | b68bd2fb-8670-4205-b9e9-737de89dfcba | None | :-) | UP | ironic-neutron-agent |
| aa2ce8d4-2f47-4e0a-8b94-428f65fec2e0 | Baremetal Node | 7102d86d-2f7b-4217-8653-d5a8e8957a7c | None | :-) | UP | ironic-neutron-agent |
| ab01ce9a-0b7e-49c8-8c85-ece612e30cbc | Baremetal Node | 048e3b73-9c5e-4727-a36b-a00406212aa0 | None | :-) | UP | ironic-neutron-agent |
| b43072a9-d20c-4800-a29f-d0f761c0cd31 | Baremetal Node | 90b75aae-bd7b-4ad8-98f0-230968738d2c | None | :-) | UP | ironic-neutron-agent |
| c794a5f1-fca9-4862-8841-4577eeaa0d1f | Baremetal Node | 2e32caa9-482a-4ca1-a16b-3dcc164e696c | None | :-) | UP | ironic-neutron-agent |
| e7c44284-7b02-4193-bd8e-ec37684dca57 | Baremetal Node | 61a7f7b9-ec6a-4250-a096-fea2b954d9be | None | :-) | UP | ironic-neutron-agent |
| ec9cf408-355c-4d13-acc3-fa86dc4a99af | Baremetal Node | 4933fb3d-aa7c-4569-ae25-0af879a11291 | None | :-) | UP | ironic-neutron-agent |
| f57317ec-8bb0-47f3-bfbc-31cb4c09a9b1 | Baremetal Node | ce27f4a5-9607-41f4-b48d-f6b5ae88da88 | None | :-) | UP | ironic-neutron-agent |
| 10669d82-03c7-450f-8b0d-32acc75fc987 | OVN Controller Gateway agent | 1327172-hp1 | | XXX | UP | ovn-controller |
| a2172b59-5558-4018-858d-88947e2d9adf | OVN Controller Gateway agent | 1327172-hp1 | | :-) | UP | ovn-controller |
+--------------------------------------+------------------------------+--------------------------------------+-------------------+-------+-------+----------------------+
We can delete the old ovn controller network agent to clean it up:
Now everything is back to normal and the prometheus alert should clear:
openstack network agent list
+--------------------------------------+------------------------------+--------------------------------------+-------------------+-------+-------+----------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+------------------------------+--------------------------------------+-------------------+-------+-------+----------------------+
| 00a833c1-7395-438c-956d-76a68c363992 | Baremetal Node | b31cd289-c475-481d-b34b-ca52106a9222 | None | :-) | UP | ironic-neutron-agent |
| 0c292b05-73b8-4851-9fca-23343e51d75e | Baremetal Node | 3a4f592b-c5f2-4df2-85dd-a8a810856b55 | None | :-) | UP | ironic-neutron-agent |
| 136f56f2-fc81-482d-b461-261df5d1c59b | Baremetal Node | 113752b7-489a-4206-b2a1-e4fcbed8d6d7 | None | :-) | UP | ironic-neutron-agent |
| 1b2e3104-3596-4544-a535-35a64f2c61cb | Baremetal Node | 61694efc-0834-4b3f-b10d-e7534ece1d7c | None | :-) | UP | ironic-neutron-agent |
| 21ceeba6-4370-4bca-9b90-fc403b9dd325 | Baremetal Node | a1a61c02-7df2-4e5b-b8fe-b0fb115b2885 | None | :-) | UP | ironic-neutron-agent |
| 31c472d9-fde7-4354-95d4-d4a6b287a65f | Baremetal Node | 29fb8908-a225-4bda-a5c7-ea9c8d80df97 | None | :-) | UP | ironic-neutron-agent |
| 391f2b35-1904-4a51-a91a-0b94efbaeae1 | Baremetal Node | 5ae6258c-255e-4d5c-9a6e-2048b914e516 | None | :-) | UP | ironic-neutron-agent |
| 40228bd2-dbed-4efd-99fd-1b25f73c5486 | Baremetal Node | 063fcc5c-5d7d-42cb-8f53-20284eb6e553 | None | :-) | UP | ironic-neutron-agent |
| 483cb4c6-282e-45e0-a2c8-aab8d73251c0 | Baremetal Node | f6be9302-96b0-47e9-ad63-6056a5e9a8f5 | None | :-) | UP | ironic-neutron-agent |
| 67412731-8a9d-4045-9fb3-dc8c74739c0e | Baremetal Node | 6cc75fc1-756a-4b19-bbab-fe8e63eee45b | None | :-) | UP | ironic-neutron-agent |
| 6cb9a276-6d51-4d7f-81b3-75d4822ae9df | Baremetal Node | f6293c81-49ff-40ff-baed-9833f6bf7480 | None | :-) | UP | ironic-neutron-agent |
| 81abb06b-176c-4248-bbcf-690d794c837b | Baremetal Node | de6495fc-3df2-4724-8217-5745d679fad1 | None | :-) | UP | ironic-neutron-agent |
| 90b18cc3-ae2f-45f7-a558-9ac00ca5b280 | Baremetal Node | bfa06d8a-7d2e-4934-aeb5-c9b185b83548 | None | :-) | UP | ironic-neutron-agent |
| 944cf411-5c22-45d7-a057-8b4eff338e7b | Baremetal Node | 0572c9c4-8199-4d34-957f-4c3fb310d557 | None | :-) | UP | ironic-neutron-agent |
| 95807c35-d58c-4a40-bf59-61173a41dcc1 | Baremetal Node | b9f80d94-aa2b-4f0c-ae4a-2f5e5dfaad25 | None | :-) | UP | ironic-neutron-agent |
| a451c2bd-e2fb-4c24-8105-235eb40d8a48 | Baremetal Node | b68bd2fb-8670-4205-b9e9-737de89dfcba | None | :-) | UP | ironic-neutron-agent |
| aa2ce8d4-2f47-4e0a-8b94-428f65fec2e0 | Baremetal Node | 7102d86d-2f7b-4217-8653-d5a8e8957a7c | None | :-) | UP | ironic-neutron-agent |
| ab01ce9a-0b7e-49c8-8c85-ece612e30cbc | Baremetal Node | 048e3b73-9c5e-4727-a36b-a00406212aa0 | None | :-) | UP | ironic-neutron-agent |
| b43072a9-d20c-4800-a29f-d0f761c0cd31 | Baremetal Node | 90b75aae-bd7b-4ad8-98f0-230968738d2c | None | :-) | UP | ironic-neutron-agent |
| c794a5f1-fca9-4862-8841-4577eeaa0d1f | Baremetal Node | 2e32caa9-482a-4ca1-a16b-3dcc164e696c | None | :-) | UP | ironic-neutron-agent |
| e7c44284-7b02-4193-bd8e-ec37684dca57 | Baremetal Node | 61a7f7b9-ec6a-4250-a096-fea2b954d9be | None | :-) | UP | ironic-neutron-agent |
| ec9cf408-355c-4d13-acc3-fa86dc4a99af | Baremetal Node | 4933fb3d-aa7c-4569-ae25-0af879a11291 | None | :-) | UP | ironic-neutron-agent |
| f57317ec-8bb0-47f3-bfbc-31cb4c09a9b1 | Baremetal Node | ce27f4a5-9607-41f4-b48d-f6b5ae88da88 | None | :-) | UP | ironic-neutron-agent |
| a2172b59-5558-4018-858d-88947e2d9adf | OVN Controller Gateway agent | 1327172-hp1 | | :-) | UP | ovn-controller |
+--------------------------------------+------------------------------+--------------------------------------+-------------------+-------+-------+----------------------+
Verifying a router port is bound to an HA_Chassis_Group
UnderStack tenant networks are VXLAN. Bare-metal nodes reach them through the
leaf switches, which map each network's VNI to a VLAN on the trunk facing the
neutron network node — so OVN has to tag traffic leaving through that network
node's uplink with the correct VLAN for the leaf to translate back onto the
right VNI (see create_uplink_port/fetch_or_create_router_segment in
neutron_understack/routers.py for how that per-network dynamic VLAN segment
gets allocated today). This is the same VNI-to-VLAN bridging problem the
upstream
L2VNI mechanism driver
(baremetal_l2vni) solves generically — allocate a dynamic VLAN per overlay
network, create an OVN localnet port for it, then let the switch-management
driver program that VLAN on the physical trunk — and what UnderStack is
moving toward in place of the bespoke logic in routers.py. Both a router's
external gateway port
(network:router_gateway) and its internal interface ports
(network:router_interface) rely on an OVN HA_Chassis_Group (HCG) to know
which chassis should actually own that tagging/forwarding for their traffic:
- The gateway port's HCG (named
neutron-<router_id>) determines which
chassis runs SNAT/DNAT — including floating IPs. If it's not linked,
north/south NAT traffic is silently dropped even though everything else
(ports, NAT rules, chassis registration) looks correct. - An internal interface port's HCG (named
neutron-<network_id>) is what
lets bare-metal nodes on that VXLAN network ARP/route through the router.
If it's not linked, those nodes can't reach their gateway.
The Logical_Router_Port.ha_chassis_group field is the thing that must be
set — if it's empty ([]), the port has no chassis to run on, regardless of
whether an HCG exists elsewhere in the database.
Note
A No hosting information found for port <id> WARNING in the neutron
logs is usually not related to this — that message just means the
Neutron ML2 port binding for that port has no host set, which is normal
for router ports. Don't chase it; check the HCG directly instead.
Process
-
Find the router port IDs (gateway and/or interface):
-
Check whether each port's Logical_Router_Port has an HCG linked:
kubectl -n openstack exec -i ovn-ovsdb-nb-0 -c ovsdb -- \ ovn-nbctl get Logical_Router_Port lrp-<port_id> ha_chassis_groupAn empty result (
[]) means the port isn't bound to any chassis — this
is the bug. -
If it's empty, check whether the HCG it should be pointing at even
exists and has a chassis in it. For a gateway port, look for
neutron-<router_id>; for an internal interface port, look for
neutron-<network_id>:- If the group doesn't exist at all, or exists with an empty
ha_chassislist, nothing has scheduled a gateway chassis for that
router/network yet. - If the group exists and has a chassis, but the LRP still isn't
linked to it, the group was created/synced but never attached to the
port — this is exactly the gapneutron_understack.routers's
link_router_gateway_ha_chassis_groupand
link_vxlan_network_ha_chassis_groupwork around (see
python/neutron-understack/neutron_understack/routers.py).
- If the group doesn't exist at all, or exists with an empty
-
Cross-check the chassis itself is live and gateway-capable:
Look for
ovn-cms-options=enable-chassis-as-gwinother_config, and
confirm the chassis UUID referenced by the HCG'sHA_Chassisrows
matches a chassis that's actually present here.
If the HCG and chassis both look correct but the LRP is still unlinked,
scripts/cleanup_dead_ovn_ha_chassis.py (dry-run by default) can repopulate
empty per-network HCGs and re-link the LRP in one pass — run it before making
any manual ovn-nbctl changes.
Resync Neutron with OVN
Resync neutron data with OVN and repair the ovn database by running the
neutron-ovn-db-sync-util command in a neutron-server pod: