Docker-airflow: Docker in Docker

[sj860908]

Have you tried running Docker in Docker with this set up?
I'm trying to run DockerOperator inside of Docker worker node.

Does anyone have any experience with this? any hints would be appreciated.

[puckel]

Not tested, but you might have a look to mount /var/run/docker.sock

[wongwill86]

Instead of DinD (Docker in Docker) I've gotten DooD (Docker outside of Docker) running. It hasn't been tested extensively yet but it seems to be working so far. The worst thing about DooD is the security implications of running docker on the host machine.

It's a combination of /var/run/docker.sock, giving airflow temporary sudo permissions.

https://github.com/puckel/docker-airflow/compare/master...wongwill86:dood?expand=1
it's quite hacky and it's an extra feature that not everyone may want to use so i'm not sure belongs in the official repo (but i'll be happy to add it if people think it's a good to have).

Hope that helps.

[efbbrown]

Hey @wongwill86 I'm trying to implement your solution on a container composed with the LocalExecutor file. I feel I'm pretty close but there's something wrong with the way I'm invoking the DockerOperator tasks in my dags.

I can execute docker run ... inside my container:

airflow@e08b37b8ea1d:~$ docker run docker/whalesay cowsay hello
 _______
< hello >
 -------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/

However when I run my dag I get an error. Here is my DockerOperator:

t3 = DockerOperator(
    # api_version='1.19',
    docker_url='tcp://localhost:2375', #Set your docker URL
    command='cowsay hello',
    image='docker/whalesay',
    # network_mode='bridge',
    task_id='docker_op_tester',
    dag=dag
)

And my error:

airflow@e08b37b8ea1d:~$ airflow test docker_sample docker_op_tester 2018-02-14
...
[2018-02-14 18:46:31,466] {{docker_operator.py:156}} INFO - Starting docker container from image docker/whalesay
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
...
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=2375): Max retries exceeded with url: /v1.24/images/json?filter=docker%2Fwhalesay%3Alatest&only_ids=0&all=0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7faf7f93f400>: Failed to establish a new connection: [Errno 111] Connection refused',))

Did you experience this issue at all? Do you have any suggestions for me? Is there something wrong with the docker_url I'm setting?

Many thanks.

[efbbrown]

Oops never mind, I found the solution to the issue I outlined above.

For anyone that is interested, the default for the docker_url argument in the DockerOperator is 'unix://var/run/docker.sock', which is what you want. So don't specify a docker_url and you're good to go.

[prideloki]

@efbbrown hi, i added /var/run/docker.sock:/var/run/docker.sock in my docker-compose-CeleryExecutor.yml.

But i got the error 'Connection aborted.', PermissionError(13, 'Permission denied'). have you experienced this?

[dylanbstorey]

@efbbrown @prideloki - I'm running into the same problem. If you figure it out tell me (if i do i'll post the solution.) Thanks

[brycedrennan]

I worked around this by running sudo chmod 777 /var/run/docker.sock on my host

[harris-chiu]

Similar to mounting a volume, this permission issue is caused with the USER airflow.
The way of fixing it is to include user airflow in group named 'docker' with the same group id that owns /var/run/docker.sock in the host machine.

1. Get the group id in /etc/group for docker group in the host machine.
2. Pass it as a environment variable for the containers in docker-compose
3. In the entry_point.sh, add the group named as 'docker' with the specified id in (1), and also add user airflow to that group.

[feluelle]

I ended up adding this to my Dockerfile.

RUN apt-get update -yqq && apt-get install -yqq --no-install-recommends sudo
RUN echo airflow:airflow | chpasswd && adduser airflow sudo
RUN echo "if [ -e /var/run/docker.sock ]; then echo airflow | sudo -S chmod 777 /var/run/docker.sock; fi" >> ${AIRFLOW_HOME}/.bashrc

I tried adding:

privileged: true
group_add:
  - staff
cap_add:
  - ALL

to my docker-compose file but neither of them worked. 😕

[tetsupanda]

I ended up adding this to my Dockerfile.

RUN apt-get update -yqq && apt-get install -yqq --no-install-recommends sudo
RUN echo airflow:airflow | chpasswd && adduser airflow sudo
RUN echo "if [ -e /var/run/docker.sock ]; then echo airflow | sudo -S chmod 777 /var/run/docker.sock; fi" >> ${AIRFLOW_HOME}/.bashrc

I tried adding:

privileged: true
group_add:
  - staff
cap_add:
  - ALL

to my docker-compose file but neither of them worked. 😕

@feluelle Did you ever figure this problem out?

[tetsupanda]

I was able to get it to finally work by adding this to my dockerfile:

RUN apt-get update && \
      apt-get -y install sudo

RUN echo airflow:airflow | chpasswd && adduser airflow sudo

And this to the entrypoint script

if [ -e /var/run/docker.sock ]; then 
  echo airflow | sudo -S chmod 777 /var/run/docker.sock; 
fi

There is likely a better way to do this. I just needed to POC some things in a local dev environment.

[ashkan-leo]

I get the following exception with @tetsupanda solution:

webserver_1  | Traceback (most recent call last):
webserver_1  |   File "/usr/local/bin/airflow", line 37, in <module>
webserver_1  |     args.func(args)
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
webserver_1  |     return f(*args, **kwargs)
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 900, in webserver
webserver_1  |     app = cached_app_rbac(None) if settings.RBAC else cached_app(None)
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/www/app.py", line 233, in cached_app
webserver_1  |     app = create_app(config, testing)
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/www/app.py", line 103, in create_app
webserver_1  |     models.Chart, Session, name="Charts", category="Data Profiling"))
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/flask_admin/contrib/sqla/view.py", line 330, in __init__
webserver_1  |     menu_icon_value=menu_icon_value)
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/flask_admin/model/base.py", line 818, in __init__
webserver_1  |     self._refresh_cache()
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/flask_admin/model/base.py", line 913, in _refresh_cache
webserver_1  |     self._search_supported = self.init_search()
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/flask_admin/contrib/sqla/view.py", line 581, in init_search
webserver_1  |     if tools.is_hybrid_property(self.model, name):
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/flask_admin/contrib/sqla/tools.py", line 209, in is_hybrid_property
webserver_1  |     return last_name in get_hybrid_properties(last_model)
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/flask_admin/contrib/sqla/tools.py", line 190, in get_hybrid_properties
webserver_1  |     for key, prop in inspect(model).all_orm_descriptors.items()
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/inspection.py", line 72, in inspect
webserver_1  |     "available for object of type %s" % type_
webserver_1  | sqlalchemy.exc.NoInspectionAvailable: No inspection system is available for object of type <class 'method'>

I had to add the following to the Dockerfile as well:

RUN pip3 uninstall -y SQLAlchemy \
    && pip3 install SQLAlchemy==1.3.15
RUN pip install docker

[feluelle]

@ashkan-leo tetsupanda's solution does not have anything to do with sqlalchemy?! I think the issue you had was related to an issue with airflow itself.

[kenter1]

I get the following exception with @tetsupanda solution:

webserver_1  | Traceback (most recent call last):
webserver_1  |   File "/usr/local/bin/airflow", line 37, in <module>
webserver_1  |     args.func(args)
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
webserver_1  |     return f(*args, **kwargs)
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 900, in webserver
webserver_1  |     app = cached_app_rbac(None) if settings.RBAC else cached_app(None)
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/www/app.py", line 233, in cached_app
webserver_1  |     app = create_app(config, testing)
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/www/app.py", line 103, in create_app
webserver_1  |     models.Chart, Session, name="Charts", category="Data Profiling"))
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/flask_admin/contrib/sqla/view.py", line 330, in __init__
webserver_1  |     menu_icon_value=menu_icon_value)
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/flask_admin/model/base.py", line 818, in __init__
webserver_1  |     self._refresh_cache()
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/flask_admin/model/base.py", line 913, in _refresh_cache
webserver_1  |     self._search_supported = self.init_search()
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/flask_admin/contrib/sqla/view.py", line 581, in init_search
webserver_1  |     if tools.is_hybrid_property(self.model, name):
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/flask_admin/contrib/sqla/tools.py", line 209, in is_hybrid_property
webserver_1  |     return last_name in get_hybrid_properties(last_model)
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/flask_admin/contrib/sqla/tools.py", line 190, in get_hybrid_properties
webserver_1  |     for key, prop in inspect(model).all_orm_descriptors.items()
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/inspection.py", line 72, in inspect
webserver_1  |     "available for object of type %s" % type_
webserver_1  | sqlalchemy.exc.NoInspectionAvailable: No inspection system is available for object of type <class 'method'>

I had to add the following to the Dockerfile as well:

RUN pip3 uninstall -y SQLAlchemy \
    && pip3 install SQLAlchemy==1.3.15
RUN pip install docker

I got the same problem when I tried to run the localexecutor and that fixed the problem.

I still dont know how to make dockeroperator work on CeleryExecutor

[feluelle]

@kenter1 I recommend to use the official airflow docker image.

I don't experience any issues with SQLAlchemy and localexecutor using the official airflow docker image.