I have a lab environment running Ansible Automation Platform 2.3 (AAP) that I use for my customer demos.
At the end of 2022 I stopped the VM for a couple of days.
Today I start my AAP VM as usual. Everything seemed to work except when I try to run a simple job template.
I get an error “cannot re-exec process to join the existing user namespace” in the job output.
I try to sync the project and get the same error. I sync other projects and they all fail.
A quick Google search takes me to a Podman issue https://github.com/containers/podman/issues/9137.
Podman indeed is the foundation of AAP under the hood, with the concept called Execution Environments (really, those are just containers).
Podman runs as the unprivileged user
I could verify the behavior from the command line:
[root@aap23 etc]# su - awx -c "podman ps -a" Error: cannot re-exec process to join the existing user namespace
The link above mentions a file
pause.pid that fixes the problem when deleted.
I find a folder
/tmp/podman-run-988/libpod/tmp but no
It turns out the temporary folder is overriden by AAP.
You can find the actual temp folder with this:
[root@aap23 etc] su awx - [awx@aap23 ~]$ podman --log-level debug ps INFO podman filtering at log level debug DEBU Called ps.PersistentPreRunE(podman --log-level debug ps) DEBU Merged system config "/usr/share/containers/containers.conf" DEBU Merged system config "/var/lib/awx/.config/containers/containers.conf" DEBU Using conmon: "/usr/bin/conmon" DEBU Initializing boltdb state at /var/lib/awx/.local/share/containers/storage/libpod/bolt_state.db DEBU Overriding run root "/tmp/podman-run-988/containers" with "/tmp/ansible.d1xwha4l/containers" from database DEBU Overriding tmp dir "/tmp/podman-run-988/libpod/tmp" with "/tmp/ansible.d1xwha4l/libpod/tmp" from database DEBU systemd-logind: Unknown object '/'.
Simple delete the file:
cd /tmp/ansible.d1xwha4l/libpod/tmp rm pause.pid
I can now sync my projects again with no restart needed.