Airflow: Task instance details page blows up when no dagrun
Apache Airflow version:
1.10.9 although I think it would apply to 1.10.10, 1.10.8, 1.10.7, and 1.10.6
Kubernetes version (if you are using kubernetes) (use kubectl version):
Environment:
- Cloud provider or hardware configuration: aws
- OS (e.g. from /etc/os-release): Ubuntu 18.04.3 LTS
- Kernel (e.g.
uname -a):Linux 567ff872fc2d 3.13.0-170-generic #220-Ubuntu SMP Thu May 9 12:40:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux - Install tools:
- Others:
What happened:
If you have a DAG that has not been turned on before and you click on that DAG in the admin page and then in the DAG graph click on one of the tasks and then click on Task Instance Details you get the "Ooops" explosion error page. The error is:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/dist-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.7/dist-packages/flask_admin/base.py", line 69, in inner
return self._run_view(f, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/flask_admin/base.py", line 368, in _run_view
return fn(self, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/flask_login/utils.py", line 258, in decorated_view
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/airflow/www/utils.py", line 290, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/airflow/www/views.py", line 1041, in task
dep_context=dep_context)]
File "/usr/local/lib/python3.7/dist-packages/airflow/www/views.py", line 1039, in <listcomp>
failed_dep_reasons = [(dep.dep_name, dep.reason) for dep in
File "/usr/local/lib/python3.7/dist-packages/airflow/models/taskinstance.py", line 668, in get_failed_dep_statuses
dep_context):
File "/usr/local/lib/python3.7/dist-packages/airflow/ti_deps/deps/base_ti_dep.py", line 106, in get_dep_statuses
for dep_status in self._get_dep_statuses(ti, session, dep_context):
File "/usr/local/lib/python3.7/dist-packages/airflow/ti_deps/deps/dagrun_id_dep.py", line 51, in _get_dep_statuses
if not dagrun.run_id or not match(BackfillJob.ID_PREFIX + '.*', dagrun.run_id):
AttributeError: 'NoneType' object has no attribute 'run_id'
What you expected to happen:
No error page. It should have loaded the standard task instance details page.
I believe the reason that this is happening is because there is no check to see if a dagrun exists in DagrunIdDep:
https://github.com/apache/airflow/blob/master/airflow/ti_deps/deps/dagrun_id_dep.py#L50
This is a bit of an edge case because there isn't really any reason someone should be checking out task instance details for a DAG that has never been run. That said we should probably avoid the error page.
How to reproduce it:
If you have a DAG that has not been turned on before and you click on that DAG in the admin page and then in the DAG graph click on one of the tasks and then click on Task Instance Details you get the "Ooops" explosion error page.
Anything else we need to know:
nadflinn
All 21 comments
Thanks for opening your first issue here! Be sure to follow the issue template!
![boring-cyborg[bot] picture](https://avatars3.githubusercontent.com/in/50386?v=4&s=40)
boring-cyborg[bot]
on 15 Apr 2020
Hello,
I just installed Airflow for the first time, following the Quick Start guide.
I ran the bash example operator (still following the guide):
# run your first task instance
airflow run example_bash_operator runme_0 2015-01-01
And then, when I went to see how the run went, I got the error page with the mushroom cloud and the error.
Now, this is on Apache Airflow v1.10.11 (latest stable), which I see was released on July 10, 2020. And this pull request was merged on April 15th. And this issue is closed. But I'm running into the same broken behavior.
Here is my error page:
Ooops.
____/ ( ( ) ) \___
/( ( ( ) _ )) ) )\
(( ( )( ) ) ( ) )
((/ ( _( ) ( _) ) ( () ) )
( ( ( (_) (( ( ) .((_ ) . )_
( ( ) ( ( ) ) ) . ) ( )
( ( ( ( ) ( _ ( _) ). ) . ) ) ( )
( ( ( ) ( ) ( )) ) _)( ) ) )
( ( ( \ ) ( (_ ( ) ( ) ) ) ) )) ( )
( ( ( ( (_ ( ) ( _ ) ) ( ) ) )
( ( ( ( ( ) (_ ) ) ) _) ) _( ( )
(( ( )( ( _ ) _) _(_ ( (_ )
(_((__(_(__(( ( ( | ) ) ) )_))__))_)___)
((__) \\||lll|l||/// \_))
( /(/ ( ) ) )\ )
( ( ( ( | | ) ) )\ )
( /(| / ( )) ) ) )) )
( ( ((((_(|)_))))) )
( ||\(|(|)|/|| )
( |(||(||)|||| )
( //|/l|||)|\\ \ )
(/ / // /|//||||\\ \ \ \ _)
-------------------------------------------------------------------------------
Node: ip-172-20-2-227.us-west-2.compute.internal
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib64/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/airflow/.local/lib/python3.7/site-packages/flask_admin/base.py", line 69, in inner
return self._run_view(f, *args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/flask_admin/base.py", line 368, in _run_view
return fn(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/flask_login/utils.py", line 258, in decorated_view
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/utils.py", line 286, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/views.py", line 1055, in task
dep_context=dep_context)]
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/views.py", line 1053, in <listcomp>
failed_dep_reasons = [(dep.dep_name, dep.reason) for dep in
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 683, in get_failed_dep_statuses
dep_context):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line 106, in get_dep_statuses
for dep_status in self._get_dep_statuses(ti, session, dep_context):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/ti_deps/deps/dagrun_id_dep.py", line 51, in _get_dep_statuses
if not dagrun.run_id or not match(BackfillJob.ID_PREFIX + '.*', dagrun.run_id):
AttributeError: 'NoneType' object has no attribute 'run_id'
atsalolikhin-spokeo
on 11 Aug 2020
I don't have the rights to re-open this issue. Is it enough for me to comment on it here?
atsalolikhin-spokeo
on 11 Aug 2020
I ran into this in AWS, running Amazon Linux 2.
uname -a shows:
Linux ip-172-20-2-227.us-west-2.compute.internal 4.14.165-131.185.amzn2.x86_64 #1 SMP Wed Jan 15 14:19:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
atsalolikhin-spokeo
on 11 Aug 2020
This fix may not have been officially released. Can you check to see if the problem is in the master version?
mik-laj
on 11 Aug 2020
Thank you for re-opening this issue. I've installed the master version:
[airflow@ip-172-20-2-227 airflow]$ airflow version
2.0.0.dev0
[airflow@ip-172-20-2-227 airflow]$
I can't login through the Web UI, though. It's asking me for username and password.
When I started the webserver, it said:
[2020-08-11 18:25:27,417] {manager.py:719} WARNING - No user yet created, use flask fab command to do it.
I tried, but it didn't work. Help?
[airflow@ip-172-20-2-227 airflow]$ flask fab create-user
Role [Public]:
Username: admin
User first name: admin
User last name: admin
Email: [email protected]
Password:
Repeat for confirmation:
Usage: flask fab create-user [OPTIONS]
Error: Could not locate a Flask application. You did not provide the "FLASK_APP" environment variable, and a "wsgi.py" or "app.py" module was not found in the current directory.
[airflow@ip-172-20-2-227 airflow]$
Here is docs about your problem: https://github.com/apache/airflow/blob/master/UPDATING.md#drop-legacy-ui-in-favor-of-fab-rbac-ui
mik-laj
on 11 Aug 2020
Thank you! I figured it out, actually.
Error: Could not locate a Flask application. You did not provide the "FLASK_APP" environment variable, and a "wsgi.py" or "app.py" module was not found in the current directory.
[airflow@ip-172-20-2-227 airflow]$ find . -name wsgi.py
[airflow@ip-172-20-2-227 airflow]$ find . -name app.py
./airflow/www/app.py
[airflow@ip-172-20-2-227 airflow]$ cd airflow/www
[airflow@ip-172-20-2-227 www]$ flask fab create-user
Role [Public]: Admin
Username: admin
User first name: admin
User last name: admin
Email: [email protected]
Password:
Repeat for confirmation:
[2020-08-11 18:34:09,566] {dagbag.py:409} INFO - Filling up the DagBag from /home/airflow/airflow/dags
Please make sure to build the frontend in static/ directory and restart the server
[2020-08-11 18:34:09,735] {manager.py:719} WARNING - No user yet created, use flask fab command to do it.
User admin created.
[airflow@ip-172-20-2-227 www]$
(I did read the documentation section you linked to, thank you; just after I already created an admin user and logged in.)
Don't laugh, but I can't figure out how to unpause the bash example operator. Instead of a sliding toggle switch, there is a checkbox.
I installed airflow by running "pip3 install --user ." in the top directory of the top of master.
atsalolikhin-spokeo
on 11 Aug 2020
Here is a screenshot, showing there is no toggle switch like there was in 1.10.11
atsalolikhin-spokeo
on 11 Aug 2020
I see this on my console:
Please make sure to build the frontend in static/ directory and restart the server
You should compile JS files now, but this is a problematic process.
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#installing-yarn-and-its-packages
Another solution: can you try to unpause the DAG with CLI and then go directly to the URL?
mik-laj
on 11 Aug 2020
Okay. I used sqlite3 to change the setting in airflow.db via CLI as you suggested.
Now the UI is showing all 22 DAGs are unpaused.
update dag
set is_paused = '0';
I've installed JS using yarn as per the linked instructions. Now Airflow is showing 18 DAGs (before it was 22). The bash example operator DAG is gone.
atsalolikhin-spokeo
on 11 Aug 2020
Is the issue discussed in this ticket still present in this version?
mik-laj
on 11 Aug 2020
I'm sorry, I can't tell. I don't seem to have a fully functional Airflow installation.
The console (CLI) keeps telling me:
Please make sure to build the frontend in static/ directory and restart the server
but I don't know how to do that.
And the example bash operator disappeared from the UI after I did the "yarn prod run".
atsalolikhin-spokeo
on 11 Aug 2020
I triggered the "tutorial" DAG instead (from the UI), and it looks like it ran.
I can look at the task details. No mushroom cloud. So I would say this issue is fixed in master.
atsalolikhin-spokeo
on 11 Aug 2020
I'm using 1.10.9 and seeing the same error for task details on SLA misses. Can someone comment if this has been fixed or not
mdube9
on 22 Sep 2020
I have the same issue with my airflow, I am using version 1.10.12.
I cannot see the logs till the operation completes and even if the operation completes , i get this error. Why do we have to spend our time on tools with little or no support ? I will appreciate it, if there is a quick fix available as it appears a lot of people are facing the same problem.
jahanxb
on 20 Oct 2020
This should be fixed in the next release. See: https://github.com/apache/airflow/issues/11328
@turbaszek Can you confirm it?
mik-laj
on 20 Oct 2020
@mik-laj yes, #11328 should fix it
turbaszek
on 20 Oct 2020
Closed by https://github.com/apache/airflow/pull/11343
This will be released in 1.10.13
kaxil
on 13 Nov 2020
Related issues
mik-laj
路
3Comments
mik-laj
路
4Comments
d-lee
路
4Comments
ryanahamilton
路
3Comments
zacwellmer
路
4Comments