Upgrading Issues Past 2.10.0

Upgrading from 2.10.0 to 2.19.1+¶

There is a known issue with upgrading / install AWX Operator beyond version 2.10.0, because of how the PostgreSQL database upgrades from 13.0 to 15.0, and has changed permissions. The following workflow will help get past that and adjust the permissions in such a way that allows the upgrade to proceed successfully. If this is a clean installation, you can also perform this step if the fresh install of 2.19.1 is not working yet. (It wont work out of the box because of this bug). `The developers of AWX seem to just not care about this issue, and have not implemented an official fix themselves at this time).

Create a Temporary Pod to Adjust Permissions¶

We need to create a pod that will mount the PostgreSQL PVC, make changes to permissions, then destroy the v15.0 pod to have the AWX Operator automatically regenerate it.

/awx/temp-pod.yml

apiVersion: v1
kind: Pod
metadata:
  name: temp-pod
  namespace: awx
spec:
  containers:
  - name: temp-container
    image: busybox
    command: ['sh', '-c', 'sleep 3600']
    volumeMounts:
    - mountPath: /var/lib/pgsql/data
      name: postgres-data
  volumes:
  - name: postgres-data
    persistentVolumeClaim:
      claimName: postgres-15-awx-postgres-15-0
  restartPolicy: Never

# Deploy Temporary Pod
kubectl apply -f /awx/temp-pod.yaml

# Open a Shell in the Temporary Pod
kubectl exec -it temp-pod -n awx -- sh

# Adjust Permissions of the PostgreSQL 15.0 Database Folder
chown -R 26:root /var/lib/pgsql/data
exit

# Delete the Temporary Pod
kubectl delete pod temp-pod -n awx

# Delete the Crashlooped PostgreSQL 15.0 Pod to Regenerate It
kubectl delete pod awx-postgres-15-0 -n awx

# Track the Migration
kubectl get pods -n awx
kubectl logs -n awx awx-postgres-15-0

Be Patient

This upgrade may take a few minutes depending on the speed of the node it is running on. Be patient and wait until the output looks something similar to this:

root@awx:/awx# kubectl get pods -n awx
NAME                                               READY   STATUS      RESTARTS   AGE
awx-migration-24.6.1-bh5vb                         0/1     Completed   0          9m55s
awx-operator-controller-manager-745b55d94b-2dhvx   2/2     Running     0          25m
awx-postgres-15-0                                  1/1     Running     0          12m
awx-task-7946b46dd6-7z9jm                          4/4     Running     0          10m
awx-web-9497647b4-s4gmj                            3/3     Running     0          10m

If you see a migration pod, like seen in the above example, you can feel free to delete it with the following command: kubectl delete pod awx-migration-24.6.1-bh5vb -n awx.