Awx: upgrade from awx 13 to 14, postgres failed to restart

[linuxphone]

ISSUE TYPE

• Bug Report

SUMMARY

postgres failed to restart from upgrade of awx v13 to v14

ENVIRONMENT

• AWX version: 14
• AWX install method: docker
• Ansible version: 2.8.3-1
• Operating System: RHEL 8.2
• Web Browser: Chrome/Firefox

STEPS TO REPRODUCE

docker stop ansible/awx:14.0.0 ansible/awx:14.0.0 redis postgres:10
rm rf /root/awx
git clone https://github.com/ansible/awx.git
cd awx/installer/
ansible-playbook -i inventory install.yml
docker ps

EXPECTED RESULTS

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e0effec6e2ab ansible/awx:14.0.0 "tini -- /usr/bin/la…" 31 minutes ago Up About a minute 8052/tcp awx_task
a3a328a5ffc3 ansible/awx:14.0.0 "tini -- /bin/sh -c …" 31 minutes ago Up About a minute 0.0.0.0:80->8052/tcp awx_web
8dd7063400a8 postgres:10 "docker-entrypoint.s…" 32 minutes ago Restarting (1) 30 seconds ago awx_postgres
e0eb72177943 redis "docker-entrypoint.s…" 2 weeks ago Up About a minute 6379/tcp awx_redis

ACTUAL RESULTS

postgres stuck in restart mode

ADDITIONAL INFORMATION

No additional details collected

[ryanpetrello]

docker logs 8dd7063400a8 ?

[shanemcd]

Did you copy over your old inventory file with your passwords?

[ryanpetrello]

What @shanemcd said:

https://github.com/ansible/awx/blob/devel/INSTALL.md#upgrading-from-previous-versions

If you blew away your inventory file, upgrading is unlikely to work.

[agrisvv]

centos 8 from awx 13 to 14 after upgrade as usual with old inventory settings.
postgres docker restarting..

````

Data page checksums are disabled.

initdb: directory "/var/lib/postgresql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data" or run initdb
with an argument other than "/var/lib/postgresql/data".
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

initdb: directory "/var/lib/postgresql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data" or run initdb
with an argument other than "/var/lib/postgresql/data".

[linuxphone]

I did both, with and without old inventoy file....the outcome was the same.
Postgres not starting.

On Thu, Aug 6, 2020, 12:47 PM Shane McDonald [e-mail]
wrote:

Did you copy over your old inventory file with your passwords?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ansible/awx/issues/7854#issuecomment-670157610, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AD7QQLFV3OJZMEPPECVMPITR7MCGNANCNFSM4PW5EJ5A
.

[linuxphone]

Hi agris....looks like some of us are in the same boat as you after
upgrade.

On Thu, Aug 6, 2020, 1:11 PM Agris [e-mail] wrote:

centos 8 from awx 13 to 14 after upgrade as usual with old inventory
settings.
postgres docker restarting..

Data page checksums are disabled.

initdb: directory "/var/lib/postgresql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data" or run initdb
with an argument other than "/var/lib/postgresql/data".
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

initdb: directory "/var/lib/postgresql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data" or run initdb
with an argument other than "/var/lib/postgresql/data".

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ansible/awx/issues/7854#issuecomment-670168121, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AD7QQLH5NXF4OC4BIQIC3LDR7ME7HANCNFSM4PW5EJ5A
.

[egmar]

@linuxphone see this change https://github.com/ansible/awx/commit/06fd1316371c9a1e6482c8e38a6bf7bfc6b6e7f4 , perhaps its the cause

[agrisvv]

@linuxphone see this change 06fd131 , perhaps its the cause

modified installer reverted changes as in commit and installed again, but now it started with empty db.. will try to check where it gone..

[linuxphone]

Hi Agris,

That entry is already committed to the latest code, so this is not the solution….the issue remains.

From: Agris [e-mail]
Sent: Thursday, August 6, 2020 2:02 PM
To: ansible/awx [e-mail]
Cc: linuxphone [e-mail]; Mention [e-mail]
Subject: Re: [ansible/awx] upgrade from awx 13 to 14, postgres failed to restart (#7854)

@linuxphone https://github.com/linuxphone see this change 06fd131 https://github.com/ansible/awx/commit/06fd1316371c9a1e6482c8e38a6bf7bfc6b6e7f4 , perhaps its the cause

modified installer reverted changes as in commit and installed again, but now it started with empty db.. will try to check where it gone..

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/ansible/awx/issues/7854#issuecomment-670189748 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7QQLFHZ7LP6GUV5YCSK73R7MK3BANCNFSM4PW5EJ5A . https://github.com/notifications/beacon/AD7QQLDJYSRIPIXRITGCECTR7MK3BA5CNFSM4PW5EJ5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOE7ZERNA.gif

[amtilghman]

I tested a devel build 2 days ago and had the same issue, was thinking it would be fixed before 14.0.0 (or i would have made an issue :) )

Just tried 12.0.0 > 14.0.0

Error was identical to person above:

initdb: directory "/var/lib/postgresql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data" or run initdb
with an argument other than "/var/lib/postgresql/data".
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

[linuxphone]

Hi Amtilghman,

It looks like we need a small patch for this….hopefully not too many users start upgrading and face the same issue.

From: amtilghman [e-mail]
Sent: Thursday, August 6, 2020 3:57 PM
To: ansible/awx [e-mail]
Cc: linuxphone [e-mail]; Mention [e-mail]
Subject: Re: [ansible/awx] upgrade from awx 13 to 14, postgres failed to restart (#7854)

I tested a devel build 2 days ago and had the same issue, was hoping it would be fixed before 14.0.0

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/ansible/awx/issues/7854#issuecomment-670230849 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AD7QQLHXH2DEVXNNR2NELOTR7MYMFANCNFSM4PW5EJ5A . https://github.com/notifications/beacon/AD7QQLDQQUETWVCB6WESZVTR7MYMFA5CNFSM4PW5EJ5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOE7ZOSQI.gif

[EkinOf]

Same with me, I just tried to upgrade from 12.0.0 to 14.0.0, my postgres container doesn't work. I downgrade to 13.0.0 and now it works.

[kenchrcum]

Hello,
I had the same issue, I solved it by using
docker-compose down && docker-compose up -d
It seemed to me the problem happened because the old AWX was still running when I started the installer.

[Goeger]

@linuxphone see this change 06fd131 , perhaps its the cause

had the same issue and to change this file to the old version solved it for me

[dotslashcmd]

It seems that in the docker-compose generated by the V14 installer, the env variable PGDATA for the postgres container is missing.

Docker-compose v 13 :

  postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - /var/lib/pgdocker/10/data/:/var/lib/postgresql/data/pgdata:Z
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      PGDATA: /var/lib/postgresql/data/pgdata
      http_proxy: 
      https_proxy: 
      no_proxy:

Docker-compose v 14 :

postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - /var/lib/pgdocker/10/data/:/var/lib/postgresql/data:Z
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      http_proxy:
      https_proxy:
      no_proxy:

[linuxphone]

@linuxphone see this change 06fd131 , perhaps its the cause

had the same issue and to change this file to the old version solved it for me

This didnt fix it for me, I tried putting back the old entries and reinstalling....same issue with postgress...it wont restart normally.

[linuxphone]

It seems that in the docker-compose generated by the V14 installer, the env variable PGDATA for the postgres container is missing.

Docker-compose v 13 :

  postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - /var/lib/pgdocker/10/data/:/var/lib/postgresql/data/pgdata:Z
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      PGDATA: /var/lib/postgresql/data/pgdata
      http_proxy: 
      https_proxy: 
      no_proxy:

Docker-compose v 14 :

postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - /var/lib/pgdocker/10/data/:/var/lib/postgresql/data:Z
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      http_proxy:
      https_proxy:
      no_proxy:

Putting the old entries back, appears to fix it....I will try again in my VAL environment.....for those having the issues, ensure you put back the old entries from released v13 and rerun the installer. Will report back shortly with my VAL results.

[linuxphone]

It seems that in the docker-compose generated by the V14 installer, the env variable PGDATA for the postgres container is missing.

Docker-compose v 13 :

  postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - /var/lib/pgdocker/10/data/:/var/lib/postgresql/data/pgdata:Z
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      PGDATA: /var/lib/postgresql/data/pgdata
      http_proxy: 
      https_proxy: 
      no_proxy:

Docker-compose v 14 :

postgres:
    image: postgres:10
    container_name: awx_postgres
    restart: unless-stopped
    volumes:
      - /var/lib/pgdocker/10/data/:/var/lib/postgresql/data:Z
    environment:
      POSTGRES_USER: awx
      POSTGRES_PASSWORD: awxpass
      POSTGRES_DB: awx
      http_proxy:
      https_proxy:
      no_proxy:

Team, I was able to fix this under my VAL environment...its working as expected....thank you all for the support.

[bile0026]

Same issue here upgrading from 12.0.0 > 14.0.0. Postgres won't start.

cd into ~/.awx/awxcompose and running docker-compose down && docker-compose up -d did solve the issue though, so looks like I'm back in business.

[shanemcd]

Hi folks. Apologies for this, the breakage was not intentional. It does seem like this broke in https://github.com/ansible/awx/commit/06fd1316371c9a1e6482c8e38a6bf7bfc6b6e7f4. Since fixing this would lead to more complications for future upgrades, I am inclined to roll forward and direct people towards the workarounds described above. Sorry again and thank you for living on the bleeding edge of a fast-moving project. :smile:

[nvonwolf]

@shanemcd Why is this not specified in the upgrade documentation or at least the release notes?

If the upgrade is no longer seamless it needs to be mentioned somewhere that going from < v14 to v14 or higher requires additional manual intervention.

[ryanpetrello]

@nvonwolf this isn't an issue we noticed until users reported it on upgrade (which is after we created the changelog/release notes)

Does this seem reasonable cc @shanemcd?

https://github.com/ansible/awx/pull/8054

[nvonwolf]

Thanks for the quick response. Issue is solved for me after finding this thread but other may not be so lucky.

[danielbostock]

Same issue here upgrading from 12.0.0 > 14.0.0. Postgres won't start.

cd into ~/.awx/awxcompose and running docker-compose down && docker-compose up -d did solve the issue though, so looks like I'm back in business.

This also fixed my issue. Thanks.