Airflow: "trying to overwrite a task will raise an exception"

[tooptoop4]

v1.10.9 was fine

i upgraded to v 1.10.12 and started getting below error after running get_state or clear on the cli:

/home/ec2-user/venv/lib/python3.7/site-packages/airflow/models/dag.py:1350: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id COMPLETE is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
category=PendingDeprecationWarning)

i am using the same taskid (named COMPLETE) on 2 different dags, in fact i receive this warning when running the get_state cli on a different (ie 3rd) dag (a that does not contain task_id COMPLETE)

[kaxil]

Can you post your DAG for us to reproduce please

[tooptoop4]

can't share but they are dynamic dag generators (a single .py creates 5 different dagids, then another .py creates 3 different dagids) , makes use of globals()[dag_id].

[kaxil]

I mean can you create a dummy DAG that you can share

[tooptoop4]

similar to https://bigdata-etl.com/apache-airflow-create-dynamic-dag/

[Alibm73]

@kaxil @tooptoop4 have the same issue.

 airflow version
1.10.12

when i type airflow webserver -p 7000 i get this error

 airflow webserver -p 7000
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-10-12 12:00:23,636] {__init__.py:50} INFO - Using executor CeleryExecutor
[2020-10-12 12:00:23,637] {dagbag.py:417} INFO - Filling up the DagBag from /root/airflow/dags
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:7000
Timeout: 120
Logfiles: - -
=================================================================
[2020-10-12 12:00:26 +0330] [3064] [INFO] Starting gunicorn 20.0.4
[2020-10-12 12:00:26 +0330] [3064] [INFO] Listening at: http://0.0.0.0:7000 (3064)
[2020-10-12 12:00:26 +0330] [3064] [INFO] Using worker: sync
[2020-10-12 12:00:26 +0330] [3100] [INFO] Booting worker with pid: 3100
[2020-10-12 12:00:26 +0330] [3101] [INFO] Booting worker with pid: 3101
[2020-10-12 12:00:26 +0330] [3103] [INFO] Booting worker with pid: 3103
[2020-10-12 12:00:26 +0330] [3107] [INFO] Booting worker with pid: 3107
[2020-10-12 12:00:27,792] {__init__.py:50} INFO - Using executor CeleryExecutor
[2020-10-12 12:00:27,793] {dagbag.py:417} INFO - Filling up the DagBag from /root/airflow/dags
[2020-10-12 12:00:27,807] {__init__.py:50} INFO - Using executor CeleryExecutor
[2020-10-12 12:00:27,809] {dagbag.py:417} INFO - Filling up the DagBag from /root/airflow/dags
[2020-10-12 12:00:27,818] {__init__.py:50} INFO - Using executor CeleryExecutor
[2020-10-12 12:00:27,820] {dagbag.py:417} INFO - Filling up the DagBag from /root/airflow/dags
[2020-10-12 12:00:27,819] {__init__.py:50} INFO - Using executor CeleryExecutor
[2020-10-12 12:00:27,820] {dagbag.py:417} INFO - Filling up the DagBag from /root/airflow/dags
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
/usr/local/lib/python3.6/site-packages/airflow/models/dag.py:1342: PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception.
  category=PendingDeprecationWarning)
^C[2020-10-12 12:00:52 +0330] [3064] [INFO] Handling signal: int
[2020-10-12 12:00:52,299] {cli.py:1127} INFO - Received signal: 2. Closing gunicorn.
[2020-10-12 12:00:52 +0330] [3101] [INFO] Worker exiting (pid: 3101)
[2020-10-12 12:00:52 +0330] [3100] [INFO] Worker exiting (pid: 3100)
[2020-10-12 12:00:52 +0330] [3103] [INFO] Worker exiting (pid: 3103)
[2020-10-12 12:00:52 +0330] [3107] [INFO] Worker exiting (pid: 3107)
[2020-10-12 12:00:53 +0330] [3064] [INFO] Shutting down: Master

[tooptoop4]

local patch on my 1.12 to revert https://github.com/apache/airflow/pull/8728 fixes this

[Alibm73]

@kaxil @tooptoop4
So my solution was setting load_examples = False in the airflow.cfg. Now the webserver started without any error.

I don't know this helps or not but figured posting it doesn't hurt #8728

[tooptoop4]

I already had load_examples = False but same issue. Reverting https://github.com/apache/airflow/pull/8728 fixes for me

[epx-dan]

I have the same issue after airflow webserver:
PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception. category=PendingDeprecationWarning)
I turned load_exampes from True to False in airflow.cfg, then this warning disappeared.

[kaxil]

I have the same issue after airflow webserver:
PendingDeprecationWarning: The requested task could not be added to the DAG because a task with task_id create_tag_template_field_result is already in the DAG. Starting in Airflow 2.0, trying to overwrite a task will raise an exception. category=PendingDeprecationWarning)
I turned load_exampes from True to False in airflow.cfg, then this warning disappeared.

This is fixed in 1.10.13