Docker-airflow: airflow CLI commands using `docker exec -it` ignore the config in compose files

[cmardiros]

I have pulled this repo and changed postgres details in docker-compose-LocalExecutor.yml to the following (nothing else changed). Note that load examples is set to False and executor is set to LocalExecutor

version: '2.1'
services:
    postgres:
        image: postgres:9.6
        environment:
            - POSTGRES_USER=w63kIwod7gae
            - POSTGRES_PASSWORD=OioE9068995S
            - POSTGRES_DB=airdb

    webserver:
        image: puckel/docker-airflow:1.9.0
        restart: always
        depends_on:
            - postgres
        environment:
            - LOAD_EX=n
            - EXECUTOR=Local
            - POSTGRES_USER=w63kIwod7gae
            - POSTGRES_PASSWORD=OioE9068995S
            - POSTGRES_DB=airdb
        volumes:
            - ./dags:/usr/local/airflow/dags
            # Uncomment to include custom plugins
            # - ./plugins:/usr/local/airflow/plugins
        ports:
            - "8080:8080"
        command: webserver
        healthcheck:
            test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
            interval: 30s
            timeout: 30s
            retries: 3

I then did:

docker-compose -f docker-compose-LocalExecutor.yml up --build

This starts the container correctly and I can see in the console that the configuration was picked up correctly:

webserver_1  | [2018-02-09 17:55:17,519] {{__init__.py:45}} INFO - Using executor LocalExecutor
webserver_1  | DB: postgresql+psycopg2://w63kIwod7gae:***@postgres:5432/airdb

I can access the admin ok and I can trigger tutorial dag ok.

Now, I then log into the container interactively:

docker exec -it $(docker-compose -f docker-compose-LocalExecutor.yml ps -q webserver) bash

If I do:

airflow connections
# brings up Using executor CeleryExecutor

airflow trigger_dag example_python_operator
# returns FATAL:  password authentication failed for user "airflow" (note that user defined in compose is w63kIwod7gae)

airflow list_dags
# lists all default examples (load examples is set to false in compose)

Location of airflow is:

airflow@b64d95a03a95:~$ which airflow
/usr/local/bin/airflow

*It seems that in interactive mode airflow uses the default configuration and completely ignores the contents of compose. *

I also tried assigning another name to the psql service i.e. changed postgres to scheduler-psql and aliased it appropriately

    webserver:
    links:
      - scheduler-psql:postgres

In this case running airflow trigger_dag tutorial from inside the container complains that postgres service is not running even though the DAGs trigger and register correctly through the UI (and if I expose port 5432 I can access the db with a sql client).

Any ideas? I'm not sure where to go next debugging this. It would be great to have the flexibility of using CLI commands inside of the container.

Thanks!

[jonwarghed]

Hi don't know if you solved this but i ran into the same thing. If I use my own config file with sql_alchemy_conn set then it works as the cli is reading that file.

[ena1106]

Same problem here. It seems that the airflow.cfg that you find inside the airflow container (located in the airflow home) is always the default one, no matter what you do in the yml file. I wonder why airflow starts with the correct configuration. From where does it read the config?

[ena1106]

Airflow reads the environment variables...

AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow

As stated here https://airflow.apache.org/configuration.html the environment variables take precedence over airflow.cfg

So all that the yml does is to export these env variables to override the default configuration.

[ena1106]

Now I understand what is happening.
Env variables are set by entrypoint.sh only for the current session, so airflow is started with the correct config.
However if you open a bash the env variables are not set.

echo $AIRFLOW__CORE__SQL_ALCHEMY_CONN

You need to export the variables in .bashrc file in order to make them available in all bash sessions.
Se also this interesting post on Airflow configuration:
http://www.the-efficient-programmer.com/programming/an-effective-airflow-setup.html

[y2k-shubham]

@ena1106 I believe L#19 is doing this very thing: _exporting environment variables_. I even tried adding explicit export statements around L#60-61, to no avail.

Just a quick confirmation: when you say

..However if you open a bash the env variables are not set...

you mean opening bash session by docker exec -it into an already running container that was launched via docker-compose / Dockerfile, right? If not, could you please elaborate a bit?

[sid88in]

@y2k-shubham @ena1106 Im facing the same issue and stumbled upon this post. When I ssh into the container I don't see AIRFLOW__CORE__SQL_ALCHEMY_CONN set but seems airflow initdb correctly created tables in postgres.
Can you please elaborate on _You need to export the variables in .bashrc file in order to make them available in all bash sessions._ ?

[lismoma]

Since you are using using docker-compose to initialize your environment then you should use docker-compose to list your dags or execute any other airflow command.
docker-compose -f docker-compose-CeleryExecutor.yml run --rm webserver airflow list_dags

Running the above will allow to airflow cli to take the same environment variables that are used in your docker component.

If you need to use docker instead of docker-compose (remember that docker cli is used to manage individual containers and docker-compose cli is used to manage a multi-container application), then you can go through entypoint.sh
docker exec -ti airflow_webserver_1 sh -c "/entrypoint.sh airflow list_dags"

[nathadfield]

This is a little bit of a hack but it does enable me to open a bash session into a Docker container and run all of the Airflow subcommands.

Basically I have created a custom entrypoint script (and a slightly modified Docker image) and added this little one liner.

if [ "$AIRFLOW__CORE__EXECUTOR" = "LocalExecutor" ]; then
    printenv | grep AIRFLOW | awk '{ print "export " $1 }' > ~/.bashrc
fi 

So, when I start Airflow under Local Executor I output all of the environment variables into a .bashrc file. This means I can then start a bash session into a running container using docker exec -it and then perform any airflow subcommand as normal.

[JonnyIncognito]

Here's a slightly different hack: ask the entrypoint to run the shell for you, so the environment variables are present:

docker exec -it airflow sh -c "/entrypoint.sh /bin/bash"