Charts: Bitnami Airflow Postgres password lost after upgrade

[estenssoros]

Which chart:

bitnami/airflow

Description

pods cannot connect to postgres after running helm update

Cannot connect to PostgreSQL server:
psql: FATAL: password authentication failed for user "bn_airflow"

Steps to reproduce the issue:

Following the instructions in the README, I ran:

export APP_HOST=127.0.0.1
export APP_PORT=8080
export APP_PASSWORD=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-password}" | base64 --decode)
export APP_DATABASE_PASSWORD=$(kubectl get secret --namespace default airflow-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
export APP_REDIS_PASSWORD=$(kubectl get secret --namespace default airflow-redis -o jsonpath="{.data.redis-password}" | base64 --decode)

helm upgrade airflow bitnami/airflow \
    --set service.type=ClusterIP,airflow.baseUrl=http://$APP_HOST:$APP_PORT,airflow.auth.password=$APP_PASSWORD,postgresql.postgresqlPassword=$APP_DATABASE_PASSWORD,redis.password=$APP_REDIS_PASSWORD 

everything seemed to be working correctly verified by kubectl port-forward --namespace default svc/airflow 8080:8080

I then attempted to set cloneDagFilesFromGit

helm upgrade airflow bitnami/airflow \
    --set airflow.cloneDagFilesFromGit.enabled=true,airflow.cloneDagFilesFromGit.repository=https://github.com/estenssoros/test_dags,airflow.cloneDagFilesFromGit.branch=master,service.type=ClusterIP,airflow.baseUrl=http://$APP_HOST:$APP_PORT,airflow.auth.password=$APP_PASSWORD,postgresql.postgresqlPassword=$APP_DATABASE_PASSWORD,redis.password=$APP_REDIS_PASSWORD

I noticed in the logs, that the git connection is successful and pulling updates however, pods cannot connect to Postgres given the password in the earliest steps. inspecting the logs for pods labeled CrashLoopBackOff I found: FATAL: password authentication failed for user "bn_airflow"

I deleted the release via helm del --purge airflow and brought it back up via following all the steps prior to setting cloneDagFilesFromGit.

Now the error is persisting that pods cannot connect to Postgres

Describe the results you received:

FATAL: password authentication failed for user "bn_airflow"

Describe the results you expected:

Pods are able to connect to Postgres with the APP_DATABASE_PASSWORD

Additional information you deem important (e.g. issue happens only occasionally):

I also ran kubectl port-forward --namespace default svc/airflow-postgresql 5432:5432
and tried to connect with psql -U bn_airflow -h 127.0.0.1 and the $APP_DATABASE_PASSWORD
psql: FATAL: password authentication failed for user "bn_airflow"

I have also tried helm install -f values.yaml bitnami/airflow --name airflow where I overrode the Postgres password with the same results

Version of Helm and Kubernetes:

• Output of helm version:

Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-502bfb", GitCommit:"502bfb383169b124d87848f89e17a04b9fc1f6f0", GitTreeState:"clean", BuildDate:"2020-02-07T01:31:02Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

[dani8art]

Hi @estenssoros thank for opening the issue.

I would say the first problem could be fixed using "" when you provide passwords to the helm command, could you try this?

helm upgrade airflow bitnami/airflow \
  --set service.type=ClusterIP \
  --set airflow.baseUrl=http://$APP_HOST:$APP_PORT \
  --set airflow.auth.password="$APP_PASSWORD" \
  --set postgresql.postgresqlPassword="$APP_DATABASE_PASSWORD" \
  --set redis.password="$APP_REDIS_PASSWORD"

Remaining the PostgreSQL connection error after helm del --purge airflow, you must be sure you were removing PVC since Helm is not doing that for you.

Regards

[estenssoros]

Thanks @dani8art,

I am still running into the same issue using quotations. I did find an interesting error just now where the airflow webserver cannot find /opt/bitnami/airflow/nss_passwd

k logs airflow-web-8596f4d645-8pjcp    
NWRAP_ERROR(1) - nwrap_files_cache_reload: Unable to open '/opt/bitnami/airflow/nss_passwd' readonly -1:No such file or directory
NWRAP_ERROR(1) - nwrap_files_getpwuid: Error loading passwd file

Welcome to the Bitnami airflow container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-airflow
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-airflow/issues
Send us your feedback at [email protected]

nami    INFO  Initializing airflow
airflow INFO  ==> No injected configuration files found. Creating default config file...
airflow INFO  ==> Setting Airflow base URL...
airflow INFO  ==> Enabling Webserver authentication...
airflow INFO  ==> Disabling examples...
airflow INFO  ==> Configuring Airflow database...
airflow INFO  ==> Setting Airflow Fernet key...
airflow INFO  ==> Configuring Airflow Celery Executor...
airflow INFO  ==> Deploying Airflow from scratch...
postgre INFO  Trying to connect to PostgreSQL server
postgre INFO  Found PostgreSQL server listening at airflow-postgresql:5432
postgre ERROR [canConnect] Connection with 'bn_airflow' user is unsuccessful
postgre ERROR [canConnect] Connection with 'bn_airflow' user is unsuccessful
postgre ERROR [canConnect] Connection with 'bn_airflow' user is unsuccessful
postgre ERROR [canConnect] Connection with 'bn_airflow' user is unsuccessful
postgre ERROR [canConnect] Connection with 'bn_airflow' user is unsuccessful
Error executing 'postInstallation': Cannot connect to PostgreSQL server:
psql: FATAL:  password authentication failed for user "bn_airflow"

output of kubectl get pods:

```
NAME READY STATUS RESTARTS AGE
airflow-postgresql-0 1/1 Running 0 8m20s
airflow-redis-master-0 1/1 Running 0 6m6s
airflow-redis-slave-0 1/1 Running 0 5m38s
airflow-redis-slave-1 1/1 Running 0 6m6s
airflow-scheduler-66b4df786f-p9pjm 0/1 CrashLoopBackOff 5 8m20s
airflow-web-7ccd47fbc-c8plk 0/1 CrashLoopBackOff 5 6m13s
airflow-web-8596f4d645-8pjcp 0/1 CrashLoopBackOff 5 8m20s
airflow-worker-0 0/1 CrashLoopBackOff 5 8m20s

[estenssoros]

hey all, I was able to fix this by running helm del --purge airflow, then deleting the pvc that had been created for postgres, redis etc k delete pvc -n default {value}, and restarting from the beginning, including quotations around my passwords