kubectl-us

kubectl-us is a kubectl plugin for operating and troubleshooting
UnderStack. Its commands are grouped by concern:

kubectl us net ... -- Neutron/OVN data-plane troubleshooting. Wraps
kubectl exec into the OVN NB/SB pods (and, for ovs-vsctl/ovs-appctl,
whichever pod is running on a given node) alongside OpenStack API calls, so
you don't have to remember pod names, container names, or the
neutron-<uuid> naming convention OVN uses for objects it syncs from
Neutron.
kubectl us backup ... -- pull local backups of the platform databases
(MariaDB and the OVN NB/SB databases).

The net commands are a companion to the manual debugging steps in
OVN / Open vSwitch -- for example, the walkthrough in
Verifying a router port is bound to an HA_Chassis_Group
is exactly what net router show (below) automates in one command.

This is intentionally a plain Python CLI for now (no compiled binary, no
krew packaging) so the command surface and output can be iterated on
quickly. Distributing it via krew is planned once the behavior settles.

Installation

cd python/kubectl-us
uv sync
export PATH="$PWD/.venv/bin:$PATH"   # so `kubectl us ...` finds it

Prerequisites

kubectl, pointed at the target cluster (via --context or your current
context)
OpenStack credentials available on your machine -- OS_CLOUD env var /
clouds.yaml, or pass --os-cloud explicitly

This tool doesn't manage either of those; it only reads them.

Global options

Option	Default	Description
`--context`	current kubectl context	kubectl context to target
`--namespace`, `-n`	`openstack`	Namespace hosting the OVN NB/SB pods
`--nb-pod`	`ovn-ovsdb-nb-0`	Northbound OVSDB pod name
`--sb-pod`	`ovn-ovsdb-sb-0`	Southbound OVSDB pod name
`--os-cloud`	`OS_CLOUD` env / clouds.yaml default	OpenStack cloud name

Every command starts by printing a banner showing the kube context, OVN
namespace/pod names, and (for OpenStack-backed commands) the OpenStack cloud
target, so it's always clear what you're actually talking to.

Global options go before the command group, e.g.
kubectl us --context my-cluster net router list.

Commands

net -- raw passthrough

kubectl us net nbctl -- show
kubectl us net sbctl -- list Chassis
kubectl us net vsctl --node <nodename> -- show
kubectl us net appctl --node <nodename> -- version

Thin wrappers around ovn-nbctl, ovn-sbctl, ovs-vsctl, and
ovs-appctl that resolve the right pod for you and stream native output
straight through -- use these for anything not covered by a higher-level
command below.

vsctl/appctl resolve the target pod by node name. On UnderStack's OVN
deployment, OVS is co-located inside the ovn-controller DaemonSet pod (no
separate openvswitch pod), so both commands default to the
ovn-controller name-prefix. If your cluster's pod naming differs, pass
--pod <name> (and, if needed, --container <name>) to target it
directly, or --target <prefix> on appctl to change the discovery
prefix.

net router list

kubectl us net router list

A table of every router seen in OpenStack and/or OVN, so you can spot
mismatches (present on only one side) before drilling into one with
net router show. Flavored routers (e.g. VRF) are handled by a different L3
backend and never get an OVN Logical_Router, so they're marked
n/a (flavored) in the OVN column rather than a false "NO".

$ kubectl us net router list
NAME             ID                                    OPENSTACK  OVN
---------------------------------------------------------------------
tenant-router    769be712-d084-4846-bd21-a85f6494f3b6  yes        yes
patch-router     94c6e0ee-3959-49e2-a9f2-e5f3a304bb77  yes        yes
vrf-router       df9746b3-d1da-484c-bf4c-86248279dddb  yes        n/a (flavored)
vrf-router-2     8f3492e8-d4d9-4f27-817c-b53a19b91f67  yes        n/a (flavored)

net router show

kubectl us net router show <router-name-or-id>
kubectl us net router show <router-name-or-id> --flows   # also dump SB logical flows

Resolves the router in OpenStack, maps it to its OVN Logical_Router
(neutron-<router_id>), and prints:

Router ports -- each Logical_Router_Port (gateway vs. internal), its
Neutron port ID, OVN-side networks and Neutron-side fixed IPs, the VLAN
tag(s) of its network's localnet/uplink port(s), and its chassis binding:
- a linked HA_Chassis_Group, with each chassis's liveness and physical
  networks (via ovn-bridge-mappings), highest priority first;
- or, for a VLAN/FLAT distributed gateway, its Gateway_Chassis binding
  (OVN's own L3-scheduler mechanism, distinct from ha_chassis_group);
- or, for a centralized router, the chassis it's pinned to via
  options:chassis;
- and only flags a port as "likely bug" when none of the above apply --
  the exact bug class scripts/cleanup_dead_ovn_ha_chassis.py repairs.
NAT rules -- each rule's type/external IP/logical IP, and the
OpenStack port it resolves to (if any).
Ports -- for each NAT-resolved port: its fixed IPs, owner (server name
for compute-owned ports), a cross-check against its OVN Logical_Switch_Port
(type/up/addresses), and that port's own HA_Chassis_Group binding (the
per-network unified HCG referenced by external/baremetal ports, distinct
from the router-port-level HCG shown above).
Optionally, southbound logical flows (ovn-sbctl lflow-list), with
--flows.

Note

As of this writing, every router in UnderStack is centralized --
there are no VLAN/FLAT distributed-gateway routers deployed, so the
Gateway_Chassis path above is exercised only by unit tests, not live
traffic. A router port should show up as "Pinned chassis"
(options:chassis), not a router-port-level HA_Chassis_Group. The
per-network HA_Chassis_Group is expected on external ports bound to
physical baremetal servers instead (the Ports section below) -- that
is a different, per-network HCG used to steer traffic to the chassis
wired to that server's physical network, and is unrelated to router
centralization. If a router port ever shows a linked HCG of its own, or
a baremetal external port has none, treat it as a red flag worth
investigating with net router show's "likely bug" diagnostics.

Example output (IPs and hostnames below are illustrative, not real):

$ kubectl us --context my-cluster-dev --os-cloud example-cloud net router show patch-router
================================================================
kubectl-us -- target environment
================================================================
  Kubernetes context : my-cluster-dev
  OVN namespace/pods : openstack (nb=ovn-ovsdb-nb-0, sb=ovn-ovsdb-sb-0)
  OpenStack cloud    : example-cloud
    auth URL         : https://keystone.dev.undercloud.example.com/v3
    project          : baremetal
================================================================


Router patch-router (94c6e0ee-3959-49e2-a9f2-e5f3a304bb77)
OVN Logical_Router: neutron-94c6e0ee-3959-49e2-a9f2-e5f3a304bb77
Type: centralized (options:chassis=a2172b59-5558-4018-858d-88947e2d9adf)

Router ports:
  - lrp-6aeee225-e410-4a7a-891d-edc2a3f3831f [gateway]
      Neutron port      : 6aeee225-e410-4a7a-891d-edc2a3f3831f
      OVN networks      : 203.0.113.41/26
      Neutron fixed IPs : 203.0.113.41
      Network VLAN tag  : 1804, 1800
      Pinned chassis    : a2172b59-5558-4018-858d-88947e2d9adf (alive, physnets=physnet1)
  - lrp-c0a41495-4cb5-4913-a7e7-9a2047f79e7d [internal]
      Neutron port      : c0a41495-4cb5-4913-a7e7-9a2047f79e7d
      OVN networks      : 192.0.2.1/24
      Neutron fixed IPs : 192.0.2.1
      Network VLAN tag  : 1802
      Pinned chassis    : a2172b59-5558-4018-858d-88947e2d9adf (alive, physnets=physnet1)

NAT rules:
  dnat_and_snat  external=203.0.113.19   logical=192.0.2.156  -> port d0075095-6232-43a6-b0c0-7fd7a07716a3
  snat           external=203.0.113.41   logical=192.0.2.0/24

Ports:
  d0075095-6232-43a6-b0c0-7fd7a07716a3 ((unnamed))
      Fixed IPs        : 192.0.2.156
      Owner            : server 14d4e13f-fef5-4676-a8e2-a5bdf63c2b6f (web-server-01)
      OVN LSP          : type=external, up, addresses=d4:04:e6:4f:7e:cc 192.0.2.156
      HA_Chassis_Group : neutron-5af67e57-7052-49fc-9bad-253275b39986 -> a2172b59-5558-4018-858d-88947e2d9adf (priority=32767, alive, physnets=physnet1)

Here the router port shows a pinned chassis (expected for a centralized
router), while the baremetal server's external port shows its own
HA_Chassis_Group -- exactly the split described in the note above.

net router audit

kubectl us net router audit <router-name-or-id>

Compares Neutron's native OVN router and port state with its realized state.
The command exits nonzero when any check fails, so it can also be used as a
pre/post-upgrade check. Flavored routers are skipped because their realization
is outside this command's scope.

For a native OVN router, the audit verifies:

the Logical_Router exists;
every Neutron gateway and interface port has an attached
Logical_Router_Port and a peer Logical_Switch_Port on the expected
switch. Gateway ports must use neutron-<network-id>. Router interfaces
currently use the same base-network switch because ML2/OVN does not
host-bind them. The audit defensively accepts a future segment-stamped
interface on its neutron-<segment-id> switch;
every LRP attached to the logical router, and its peer LSP when present,
corresponds to a current Neutron router port;
each router network has exactly one shared Neutron uplink-* port, with a
matching localnet LSP attached to neutron-<network-id> and
addresses=unknown, and exactly one VLAN tag;
the peer LSP has type=router, addresses=router, and the matching
options:router-port;
gateway LSPs additionally have options:nat-addresses=router and
options:exclude-lb-vips-from-garp=true; and
each router-owned per-network HA_Chassis_Group has at least one
HA_Chassis member whose chassis is live in the Southbound database. This
reports empty and stale-only groups as requiring per-network HA chassis
group repopulation; and
when repopulation is required, a live options:chassis value or an
unambiguous router-level HA chassis group is available as its source.

Example of the corruption detected in issue 2330:

Router tenant-router (7a4b...)
Backend: native OVN
  PASS  logical router: neutron-7a4b...
  FAIL  gateway 6718...: LSP type: expected router, found (empty)
  FAIL  gateway 6718...: LSP addresses: expected ['router'], found []
  FAIL  gateway 6718...: router-port option: expected lrp-6718..., found (missing)
  FAIL  gateway 6718...: nat-addresses option: expected router, found (missing)

net router repair

# Plan only; this never writes to OVN.
kubectl us net router repair <router-name-or-id>

# Apply the displayed plan and verify the result.
kubectl us net router repair <router-name-or-id> --apply

router repair is deliberately narrower than Neutron's fleet-wide OVN DB
sync repair mode. It repairs only the selected native-OVN router. Existing
gateway/interface peer LSP rows must already be attached to both the expected
logical router port and logical switch. Its LSP writes are:

type=router;
addresses=router;
options:router-port=lrp-<port-id>; and, for gateway ports only,
options:nat-addresses=router and
options:exclude-lb-vips-from-garp=true.

The command updates option keys individually, preserving unrelated options
such as requested-chassis.

Repair also repopulates router-owned per-network HA chassis groups that have
no live members. It selects the target chassis from the logical router's live
options:chassis value. When that option is absent, it uses the router's own
HA chassis group only when that group resolves to exactly one live chassis.
For each affected per-network group, repair removes its non-live member
references, creates a replacement HA_Chassis at priority 32767, and adds
the replacement to the group.

This per-router repair overlaps the stale-member cleanup and per-network group
repopulation performed fleet-wide by
scripts/cleanup_dead_ovn_ha_chassis.py.
The script also has a separate, opt-in orphan-network teardown mode that this
command does not provide. The command is preferable when diagnosing and
repairing one known router; use the script when intentionally auditing the
fleet-wide HA chassis state.

Shared uplink-* localnet LSPs are audit-only. router repair does not
recreate them because doing so also requires coordinating Neutron port and
trunk state.

With --apply, all planned LSP and HA chassis group updates are sent in one
ovn-nbctl transaction and the affected state is read back for verification.
HA chassis group verification checks the exact plan: the selected chassis is
present at priority 32767 and the stale member references are absent.

Repair refuses a router with a flavor entirely. For native OVN routers, it
refuses the affected repair domain when:

a logical router, router port, or switch port is missing; or
an existing port is attached to the wrong logical router or switch; or
an LRP is attached to the logical router without a corresponding Neutron
router port; or
an affected per-network HA chassis group has no unambiguous live target
chassis.

Those conditions require investigation rather than field-level repair. Run
net router audit after resolving them.

LSP repair and HA chassis group repair are independent safety domains. A
refusal in one domain does not block safe work in the other. When --apply
performs only the safe portion, the command verifies that work and exits 2
to report that part of the router remains unresolved.

backup local

kubectl us backup local                          # into the current directory
kubectl us backup local --directory ./backups    # into a specific directory
kubectl us backup local --skip-mariadb           # OVN NB/SB only
kubectl us backup local --skip-ovn               # MariaDB only

Downloads a consistent set of backups to your local machine, wrapping the
manual runbook steps into one command so you don't have to hand-copy pod
names, container names, and socket paths before an upgrade. It writes three
files into --directory (default: the current directory):

mariadb_backup_<context>_<epoch>.sql -- a logical dump of every database
(mariadb-dump --single-transaction --routines --triggers --all-databases)
taken against mariadb-0's mariadb container. The root password is never
read locally: the dump runs via sh -c and expands the pod's own
$MARIADB_ROOT_PASSWORD (injected by the mariadb-operator) into MYSQL_PWD
inside the container, so the secret never touches the local process, the
kubectl argv, or the exec request.
ovnnb_backup_<context>_<epoch>.db -- ovsdb-client backup of the OVN
Northbound DB from ovn-ovsdb-nb-0.
ovnsb_backup_<context>_<epoch>.db -- ovsdb-client backup of the OVN
Southbound DB from ovn-ovsdb-sb-0.

<context> is the kube context (sanitized for filenames, e.g.
rax_prod_iad3_rxdb_mt -> rax-prod-iad3-rxdb-mt) and all files in one run
share a single <epoch> timestamp so they sort together as a set. These are
the same ovsdb-client backup / mariadb-dump operations documented in the
RXDB and MariaDB upgrade runbooks; as there, the exec deliberately uses -i
without -t (a TTY corrupts the backup byte stream). --skip-mariadb and
--skip-ovn narrow the set; skipping both is an error.

Development

Each command is a self-contained module under us_cli/commands/ that
registers into the typer app (us_cli/cli.py), where the net commands
are grouped under a net sub-app and backup is mounted at the top level.
Low-level OVSDB access lives in us_cli/ovn.py (JSON unwrap ported from
scripts/cleanup_dead_ovn_ha_chassis.py), kubectl exec plumbing in
us_cli/kube.py, and OpenStack SDK connection setup in
us_cli/osclient.py. Run tests and linting from python/kubectl-us/:

uv run pytest
uv run ruff check
uv run ruff format

Contributing

If you find any issues or have suggestions for improvements, please open an
issue on the GitHub repository.