Docker-airflow: Installing custom python packages succeeds, but dag cannot find them: "No module named 'x'"

[cslovell]

I've added the requirements.txt file with a python package x, and run the docker-compose celery command. Everything works, the package is installed, I go inside the docker container and I can verify the package is installed. However, the dags in the UI show an error:
Broken DAG: [/usr/local/airflow/dags/tuto.py] No module named 'x'

running which python from the command line inside the container and running sys.executable from inside a python operator both return the same path. What could be happening?

"""
Code that goes along with the Airflow located at:
http://airflow.readthedocs.org/en/latest/tutorial.html
"""
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta
import sys
#import pandas as pd
import math
import pandasdmx
from pandasdmx import Request
from ast import literal_eval

def get_oecd_data_dict(**kwargs):
    oecd = Request('OECD')
    data_response = oecd.data(resource_id='MUNW', key='all/all')
    data = data_response.write(s for s in data_response.data.series)
    l = []
    for index, row in data.iterrows():
        for i, item in row.iteritems():
            if not math.isnan(item):
                temp_dict = {
                    'year':index.year,
                    'country': i[0],
                    'indicator': i[1],
                    'value': item
                }
                l.append(temp_dict)
    return l


def get_python(**kwargs):
    res = sys.executable
    print(res)
    return res


default_args = {
    "owner": "cslovell",
    "depends_on_past": False,
    "start_date": datetime(2015, 6, 1),
    "email": ["[email protected]"],
    "email_on_failure": False,
    "email_on_retry": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=5),
    # 'queue': 'bash_queue',
    # 'pool': 'backfill',
    # 'priority_weight': 10,
    # 'end_date': datetime(2016, 1, 1),
}

dag = DAG("tutorial", default_args=default_args, schedule_interval=timedelta(1))

p1 = PythonOperator(
    task_id="get_data",
    provide_context=True,
    python_callable=get_oecd_data_dict,
    dag=dag,
    )

t1 = BashOperator(task_id="print_date", bash_command="date", dag=dag)

t2 = BashOperator(task_id="sleep", bash_command="sleep 5", retries=3, dag=dag)

templated_command = """
    {% for i in range(5) %}
        echo "{{ ds }}"
        echo "{{ macros.ds_add(ds, 7)}}"
        echo "{{ params.my_param }}"
    {% endfor %}
"""

t3 = BashOperator(
    task_id="templated",
    bash_command=templated_command,
    params={"my_param": "Parameter I passed in"},
    dag=dag,
)

t2.set_upstream(t1)
t3.set_upstream(t1)
p1.set_upstream(t1)

Inside the docker-compose celery:

volumes:
            - ./dags:/usr/local/airflow/dags
            - ./requirements.txt:/requirements.txt

From the compose logs:

webserver_1  |   Downloading https://files.pythonhosted.org/packages/e7/5d/7e29616b0376d0a15eb3013c909c4e9021b6c6e244ef1b48817b4d0c4d46/pandaSDMX-0.9-py2.py3-none-any.whl (45kB)
webserver_1  | Requirement already satisfied: pandas in /usr/local/lib/python3.6/site-packages (from -r /requirements.txt (line 2)) (0.24.2)
webserver_1  | Requirement already satisfied: requests in /usr/local/lib/python3.6/site-packages
(from pandasdmx->-r /requirements.txt (line 1)) (2.22.0)
webserver_1  | Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from pandasdmx->-r /requirements.txt (line 1)) (41.0.1)
webserver_1  | Collecting jsonpath-rw (from pandasdmx->-r /requirements.txt (line 1))
webserver_1  |   Downloading https://files.pythonhosted.org/packages/71/7c/45001b1f19af8c4478489fbae4fc657b21c4c669d7a5a036a86882581d85/jsonpath-rw-1.4.0.tar.gz
webserver_1  | Requirement already satisfied: lxml in /usr/local/lib/python3.6/site-packages (from pandasdmx->-r /requirements.txt (line 1)) (4.3.3)
webserver_1  | Requirement already satisfied: python-dateutil>=2.5.0 in /usr/local/lib/python3.6/site-packages (from pandas->-r /requirements.txt (line 2)) (2.8.0)
webserver_1  | Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/site-packages (from pandas->-r /requirements.txt (line 2)) (2019.1)
webserver_1  | Requirement already satisfied: numpy>=1.12.0 in /usr/local/lib/python3.6/site-packages (from pandas->-r /requirements.txt (line 2)) (1.16.4)
webserver_1  | Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests->pandasdmx->-r /requirements.txt (line 1)) (2019.3.9)
webserver_1  | Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests->pandasdmx->-r /requirements.txt (line 1)) (1.25.3)
webserver_1  | Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/site-packages (from requests->pandasdmx->-r /requirements.txt (line 1)) (3.0.4)
webserver_1  | Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/site-packages (from requests->pandasdmx->-r /requirements.txt (line 1)) (2.8)
webserver_1  | Collecting ply (from jsonpath-rw->pandasdmx->-r /requirements.txt (line 1))
webserver_1  |   Downloading https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl (49kB)
webserver_1  | Collecting decorator (from jsonpath-rw->pandasdmx->-r /requirements.txt (line 1))
webserver_1  |   Downloading https://files.pythonhosted.org/packages/5f/88/0075e461560a1e750a0dcbf77f1d9de775028c37a19a346a6c565a257399/decorator-4.4.0-py2.py3-none-any.whl
webserver_1  | Requirement already satisfied: six in /usr/local/lib/python3.6/site-packages (from jsonpath-rw->pandasdmx->-r /requirements.txt (line 1)) (1.12.0)
webserver_1  | Building wheels for collected packages: jsonpath-rw
webserver_1  |   Building wheel for jsonpath-rw (setup.py): started
webserver_1  |   Building wheel for jsonpath-rw (setup.py): finished with status 'done'
webserver_1  |   Stored in directory: /usr/local/airflow/.cache/pip/wheels/5c/00/9a/82822db383c2d96dcebf839786665a185f92d37e5026f9806f
webserver_1  | Successfully built jsonpath-rw
webserver_1  | Installing collected packages: ply, decorator, jsonpath-rw, pandasdmx
webserver_1  |   WARNING: The script jsonpath.py is installed in '/usr/local/airflow/.local/bin'
which is not on PATH.
webserver_1  |   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
webserver_1  | Successfully installed decorator-4.4.0 jsonpath-rw-1.4.0 pandasdmx-0.9 ply-3.11

[cslovell]

Figured it out: I had only mapped the requirements.txt volume to the webserver image, when I mapped it to any container that also had a mapping for the dag folder, then it worked.

[hernanborre]

Figured it out: I had only mapped the requirements.txt volume to the webserver image, when I mapped it to any container that also had a mapping for the dag folder, then it worked.

Congratulations! But I didn't get your solution! What did you do?
Thanks!