Skip to content

MariaDB Operator Upgrade Runbook

This runbook describes the steps to upgrade the MariaDB Operator and perform backup/restore procedures in Kubernetes.

Pre-Upgrade Steps

1. Backup current MariaDB configuration

kubectl get mariadb -A -o yaml > current-mariadb-config-backup-$(date +%Y%m%d).yaml

2. Retrieve root password

export MARIADB_ROOT_PASSWORD=$(kubectl get secret mariadb -n openstack -o jsonpath='{.data.root-password}' | base64 -d)

3. Assess database sizes and usage

kubectl exec -n openstack -it mariadb-0 -- mariadb -u root -p"$MARIADB_ROOT_PASSWORD"

Run inside MariaDB shell:

SELECT
  table_schema AS 'Database',
  ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS 'Size_MB'
FROM information_schema.tables
WHERE table_schema NOT IN ('information_schema', 'performance_schema', 'mysql', 'sys')
GROUP BY table_schema
ORDER BY Size_MB DESC;

4. Backup all databases

kubectl exec -n openstack -it mariadb-0 -- mariadb-dump -u root -p"$MARIADB_ROOT_PASSWORD" \
  --single-transaction \
  --routines \
  --triggers \
  --all-databases > full-backup-$(date +%Y%m%d-%H%M).sql

5. Verify storage classes

  • StorageClass specified for MariaDB should exists
kubectl get storageclass

Upgrade MariaDB Operator

6. Update Mariadb Operator version

  • Edit operator reference in:
    kustomization.yaml
  • Also bump targetRevision in the CRDs Application,
    charts/argocd-understack/templates/application-mariadb-operator-crds.yaml,
    to the same version.
  • Do not skip intermediate versions. Upgrade one hop at a time
    (25.8.x -> 25.10.4 -> 26.3.0 -> 26.6.0) and wait for the Galera cluster to
    be Synced and the MariaDB resource Ready before the next hop.
    Avoid 25.10.0 and 25.10.1, which have a regression.
  • Our MariaDB resource sets updateStrategy.autoUpdateDataPlane: true, so
    every operator upgrade rolls the Galera pods to update the data plane
    (init and agent containers). Schedule a maintenance window.

7. Perform the release

Release Process Documentation

Validate Upgrade

8. Check applications in ArgoCD

Applications:

  • <deploy-cluster>-mariadb-operator (ex: charlie-uc-iad3-staging-mariadb-operator)
  • <deploy-cluster>-openstack (ex: charlie-uc-iad3-staging-openstack)

9. Check logs in MariaDB StatefulSet

kubectl logs -n openstack statefulset/mariadb -f

10. Check logs in MariaDB Operator

kubectl logs -n mariadb-operator deployment/mariadb-operator -f

Emergency Cleanup

If you encounter volume mount issues:

kubectl patch mariadb mariadb -n openstack --type='merge' -p='{"metadata":{"finalizers":[]}}'
kubectl delete mariadb mariadb -n openstack
kubectl delete pvc -n openstack -l app.kubernetes.io/name=mariadb --wait=false

Restore Procedure

11. Get new root password

export MARIADB_ROOT_PASSWORD=$(kubectl get secret mariadb -n openstack -o jsonpath='{.data.root-password}' | base64 -d)

12. Restore from backup if volumes has been deleted

kubectl exec -i -n openstack mariadb-0 -- mariadb -u root -p"$MARIADB_ROOT_PASSWORD" < full-backup-production-20250910-0735.sql

13. Re-check database sizes

kubectl exec -n openstack -it mariadb-0 -- mariadb -u root -p"$MARIADB_ROOT_PASSWORD"

Run SQL again:

SELECT
  table_schema AS 'Database',
  ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS 'Size_MB'
FROM information_schema.tables
WHERE table_schema NOT IN ('information_schema', 'performance_schema', 'mysql', 'sys')
GROUP BY table_schema
ORDER BY Size_MB DESC;

Post-Upgrade Connectivity Check

14. Test Galera peer connectivity

kubectl exec mariadb-0 -n openstack -- timeout 5 bash -c "echo > /dev/tcp/mariadb-1.mariadb-internal.openstack.svc.cluster.local/4567" && echo "mariadb-1 reachable" || echo "mariadb-1 not reachable"

kubectl exec mariadb-0 -n openstack -- timeout 5 bash -c "echo > /dev/tcp/mariadb-2.mariadb-internal.openstack.svc.cluster.local/4567" && echo "mariadb-2 reachable" || echo "mariadb-2 not reachable"

End of Runbook