Docker-airflow: Example utilizing RDS or other external Postgres instance?

[maxcountryman]

Hi, I'm wondering if there's some example of using this configuration with an RDS instance of Postgres? It seems that standing up Postgres inside of the Docker Compose configuration means that there is possibly no persistence between deployments. As such it seems a better option might be to point to Postgres running in a different context.

[chribsen]

You can connect to any externally hosted PostgreSQL database by injecting a connection string into the Airflow containers using the AIRFLOW__CORE__SQL_ALCHEMY_CONN environment variable.

[roirodriguez]

@chribsen: I'm trying exactly this to connect to mysql instead of postgres and it's not working. My compose file

version: '3'

services:
  mysql:
    image: mysql/mysql-server:5.7
    environment:
      - MYSQL_ROOT_PASSWORD=secret
      - MYSQL_DATABASE=airflow
      - MYSQL_USER=airflow
      - MYSQL_PASSWORD=airflow
    volumes:
      - mysqlairflowdata:/var/lib/mysql
  airflow:
    image: puckel/docker-airflow:1.8.2
    depends_on:
      - mysql
    environment:
      - LOAD_EX=n
      - EXECUTOR=Local
      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=mysql://airflow:airflow@mysql:3306/airflow
    ports:
      - "8080:8080"
    volumes:
      - ./airflow-dags:/usr/local/airflow/dags
    command: webserver
    healthcheck:
      test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3

volumes:
  mysqlairflowdata:

If i run this:

$ docker-compose up -d
$ docker-compose logs -f airflow
airflow_1  | Tue Feb 20 14:09:38 UTC 2018 - waiting for postgres:5432... 1/20

Anybody have some working sample with config env variables passed through docker-compose.yml? I can't make this to work.

[roirodriguez]

In fact at line 67 of scripts/entrypoint.sh in this repo:

AIRFLOW__CORE__SQL_ALCHEMY_CONN="postgresql+psycopg2://$POSTGRES_USER:$POSTGRES_PASSWORD@$POSTGRES_HOST:$POSTGRES_PORT/$POSTGRES_DB"

So this variable seems to get overriden at entrypoint.sh. I'm afraid it'd be necessary to provide your own entrypoint for this to work. Maybe a documentation issue? Or maybe surrounding the above line with an if statement if the variable is unset.

[chribsen]

Aah. That explains it: The entrypoint.sh script constructs the connection string based on multiple environment variables. So instead of using AIRFLOW__CORE__SQL_ALCHEMY_CONN directly you should pass in the following environment variables:

export POSTGRES_USER=<postgres-user>
export POSTGRES_PASSWORD=<password-to-postgres>
export POSTGRES_HOST=<database-host>
export POSTGRES_PORT=<database-port>
export POSTGRES_DB=<name-of-database>

If you don't pass in these environment variables, the entrypoint.sh script will choose the following default settings (as of commit 7a99885):

: "${POSTGRES_HOST:="postgres"}"
: "${POSTGRES_PORT:="5432"}"
: "${POSTGRES_USER:="airflow"}"
: "${POSTGRES_PASSWORD:="airflow"}"
: "${POSTGRES_DB:="airflow"}"

I'm using a custom fork that doesn't rely on this particular version of entrypoint.sh, so that is why I didn't spot this before.

[roirodriguez]

Oh, i see... I think i overlooked that. So it's not possible to use other database engine, as the SQLAlchemy driver is set to postgresql+psycopg2 hardcoded at entrypoint.sh.

[chribsen]

Unfortunately not without forking the repo and removing that part of entrypoint.sh

[7yl4r]

Relevant PR: #253