Docker-airflow: [Airflow KubernetesExecutor] Not working Entrypoint on worker pod

[tyg03485]

Hi, guys.

I'm trying to migrating airflow 1.10.4 for KubernetesExecutor.

I use same image for webserver, schedular, worker.

But, as some reason, with creating KubernetesExecutor pod, entrypoint.sh doesn't execute.
Of Course, webserver, schedular works fine.

I dig this problem for over 1 week, I can't find the reason why they don't execute entrypoint.sh.

Dockerfile

...
ARG AIRFLOW_VERSION=1.10.4
ARG AIRFLOW_USER_HOME=/usr/local/airflow
...
COPY ./entrypoint.sh /entrypoint.sh
COPY ./dags ${AIRFLOW_HOME}/dags

RUN chown -R airflow: ${AIRFLOW_USER_HOME}
RUN chmod +x /entrypoint.sh

USER airflow
WORKDIR ${AIRFLOW_USER_HOME}
ENTRYPOINT ["/entrypoint.sh"]
CMD ["--help"]

entrypoint.sh

#!/usr/bin/env bash

ECHO "START"
TRY_LOOP="10"
AIRFLOW_HOME="/usr/local/airflow"
...
# Defaults and back-compat
: "${AIRFLOW__CORE__FERNET_KEY:=${FERNET_KEY:=$(python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)")}}"
...
case "$1" in
  webserver)
    airflow initdb
    if [ "$AIRFLOW__CORE__EXECUTOR" = "LocalExecutor" ]; then
      # With the "Local" executor it should all run in one container.
      airflow scheduler &
    fi
    exec airflow webserver
    ;;
  worker|scheduler)
    # To give the webserver time to run initdb.
    sleep 10
    exec airflow "$@"
    ;;
  flower)
    sleep 10
    exec airflow "$@"
    ;;
  version)
    exec airflow "$@"
    ;;
  *)
    # The command is something like bash, not an airflow subcommand. Just run it in the right environment.
    exec "$@"
    ;;
esac

And FYI, I use airflow.cfg as k8s configmap

[ketikat]

I also have this same question. I was wondering if this line:
if [ "$AIRFLOW__CORE__EXECUTOR" = "LocalExecutor" ]
be:
if [ "$AIRFLOW__CORE__EXECUTOR" = "KubernetesExecutor" ]

with the kuEx you should only have 1 webserver pod that spins up tasks per dag as needed in other containers.

[AXington]

You need to run pip install apache-airflow[kubernetes]==${AIRFLOW_VERSION} in your dockerfile, it's not installed in this container by default.

Also are there any logs from the pods that are failing?

Also, how are you deploying it? Can you share details of your kube yamls or helm info?

[ketikat]

I have already pip installed the correct version of airflow with all the req'd airflow packages. I am successfully able to spin up pods with the Kubernetes Operator, however now my jobs fail because there is no log file. I am gettign this Error in airflow: *** Log file does not exist: /usr/local/airflow/logs/hw_load.load_and_validate_dimensions/exrt_load_dimension_HWLOAD_UNIT/2019-10-16T16:17:37.011639+00:00/1.log *** Fetching from: http://hwloadsystemloadandvalidatedimensions-790ab2287dc5454f97d70d03b:8793/log/hw_load.load_and_validate_dimensions/exrt_load_dimension_HWLOAD/2019-10-16T16:17:37.011639+00:00/1.log *** Failed to fetch log file from worker. HTTPConnectionPool(host='hwloadsystemloadandvalidatedimensions-790ab2287dc5454f97d70d03b', port=8793): Max retries exceeded with url: /log/hw_load.load_and_validate_dimensions/exrt_load_dimension_HWLOAD/2019-10-16T16:17:37.011639+00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0615ce4f28>: Failed to establish a new connection: [Errno -2] Name or service not known',))

[ketikat]

here is my airflow entrypoint.sh for reference:

``` #!/usr/bin/env bash

TRY_LOOP="20"
: "${AIRFLOW_HOME:="/usr/local/airflow"}"
: "${REDIS_HOST:="redis"}"
: "${REDIS_PORT:="6379"}"
: "${REDIS_PASSWORD:=""}"
: "${DB_HOST:="mysql-server"}"
: "${DB_PORT:="3306"}"

export
AIRFLOW__CORE__EXECUTOR="KubernetesExecutor"
AIRFLOW__CORE__FERNET_KEY=$FERNET_KEY
AIRFLOW_HOME="/usr/local/airflow"
AIRFLOW__CORE__DAGS_FOLDER="/usr/local/airflow/mydags"
DAGS_FOLDER="/usr/local/airflow/mydags"
AIRFLOW__WEBSERVER__RBAC="True"

wait_for_port() {
local name="$1" host="$2" port="$3"
local j=0
while ! nc -z "$host" "$port" >/dev/null 2>&1 < /dev/null; do
j=$((j+1))
if [ $j -ge $TRY_LOOP ]; then
echo >&2 "---- waiting for port: $(date) - $host:$port still not reachable, giving up ---- "
exit 1
fi
echo "---- waiting for port: $(date) - waiting for $name... $j/$TRY_LOOP ---- "
sleep 5
done
}

wait_for_redis() {
if [ "$AIRFLOW__CORE__EXECUTOR" = "CeleryExecutor" ]
echo "---- Entrypoint: Waiting for Redis to start ---- "
then
wait_for_port "Redis" "$REDIS_HOST" "$REDIS_PORT"
fi
}

wait_for_mysql() {
echo "---- Entrypoint: Waiting for mysql server to start ---- "
wait_for_port "mysql-server" "$DB_HOST" "$DB_PORT"
}

if [ "$AIRFLOW__CORE__EXECUTOR" = "CeleryExecutor" ]; then
export AIRFLOW__CORE__SQL_ALCHEMY_CONN="mysql://$AIRFLOW_DB_USER:$AIRFLOW_DB_PASSWORD@mysql-server:3306/airflow"
export AIRFLOW__CELERY__RESULT_BACKEND="db+mysql://$AIRFLOW_DB_USER:$AIRFLOW_DB_PASSWORD@mysql-server:3306/airflow"
wait_for_mysql
export AIRFLOW__CELERY__BROKER_URL="redis://redis:6379/1"
wait_for_redis
fi

if [ "$AIRFLOW__CORE__EXECUTOR" = "KubernetesExecutor" ]; then
export AIRFLOW__CORE__SQL_ALCHEMY_CONN="mysql://$AIRFLOW_DB_USER:$AIRFLOW_DB_PASSWORD@mysql-server:3306/airflow"
wait_for_mysql
fi

case "$1" in
webserver)
echo "---- Entrypoint: IN WEBSERVER ----"
wait_for_mysql
airflow upgradedb
echo "---- Entrypoint: airflow db is intialized ---- "

echo "---- Entrypoint: Creating Admin account for Admin User ---- "
# this will be the new syntax in next airflow upgrade:
# airflow users --create --username $AIRFLOW_USERNAME  --email $AIRFLOW_EMAIL --password $AIRFLOW_PASSWORD #--role Admin
# this is the current way of doing it:
airflow create_user -r Admin -u $AIRFLOW_USERNAME -e $AIRFLOW_EMAIL -p $AIRFLOW_PASSWORD -f Admin -l User


if [ "$AIRFLOW__CORE__EXECUTOR" = "KubernetesExecutor" ]; then
  # With the "Kubernetes" executor it should all run in one container.
  airflow scheduler &
fi

if [ "$AIRFLOW__CORE__EXECUTOR" = "LocalExecutor" ]; then
  # With the "Local" executor it should all run in one container.
  airflow scheduler &
fi

sleep 10
exec airflow webserver
;;

scheduler)
echo "---- Entrypoint: IN SCHEDULER ----"
wait_for_mysql
wait_for_redis
sleep 10
exec airflow "$@"
;;
worker)
echo "---- Entrypoint: IN WORKER ----"
wait_for_mysql
wait_for_redis
sleep 10
exec airflow worker
exec airflow "$@"
;;
flower)
echo "---- Entrypoint: IN FLOWER ----"
sleep 10
exec airflow "$@"
;;
*)
exec "$@"
;;
esac

[ketikat]

@tblazina

[tblazina]

Thanks @ketikat!

[bhavaniravi]

I have the same issue, running the entrypoint script on worker pods, any fixes for this?