Airflow: task marked as failed when sys.exit(0)

[SergioShz]

Apache Airflow version:
apache-airflow (1.10.10)

Environment:
Docker version 19.03.12

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

• Kernel (e.g. uname -a):

4.19.76-linuxkit

What happened:

I'm testing this small DAG (see below) consisting of a simple task using PythonOperator. I'd like to exit the call with a exit code 0 when everything went fine, however airflow seems to be marking the task as failed when this happen.
The log file:

[2020-10-02 09:44:13,081] {taskinstance.py:669} INFO - Dependencies all met for <TaskInstance: TEST_0.test_task 2020-10-02T09:43:00+00:00 [queued]>
[2020-10-02 09:44:13,735] {taskinstance.py:669} INFO - Dependencies all met for <TaskInstance: TEST_0.test_task 2020-10-02T09:43:00+00:00 [queued]>
[2020-10-02 09:44:13,736] {taskinstance.py:879} INFO - 
--------------------------------------------------------------------------------
[2020-10-02 09:44:13,737] {taskinstance.py:880} INFO - Starting attempt 1 of 1
[2020-10-02 09:44:13,737] {taskinstance.py:881} INFO - 
--------------------------------------------------------------------------------
[2020-10-02 09:44:14,288] {taskinstance.py:900} INFO - Executing <Task(PythonOperator): test_task> on 2020-10-02T09:43:00+00:00
[2020-10-02 09:44:14,291] {standard_task_runner.py:53} INFO - Started process 10800 to run task
[2020-10-02 09:44:16,389] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: TEST_0.test_task 2020-10-02T09:43:00+00:00 [running]> 3a160363119b
[2020-10-02 09:44:17,433] {logging_mixin.py:112} INFO - doing stuff inside task
[2020-10-02 09:44:22,195] {logging_mixin.py:112} INFO - [2020-10-02 09:44:22,195] {local_task_job.py:103} INFO - Task exited with return code 0

What you expected to happen:
I'm expecting the task marked as succed, or at least understand why airflow marks this task as failed despite of the exit code 0.

How to reproduce it:
Here is the code

import sys
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.utils.dates import days_ago

default_args = {
    'depends_on_past': False,
    'start_date': days_ago(1),
    'retries': 0,
}

DAG_NAME = 'TEST_0'
SCH_INTERVAL = '0/1 * * * *'

def task():
    print('doing stuff inside task')
    sys.exit(0)

with DAG(
    DAG_NAME,
    default_args=default_args,
    schedule_interval=SCH_INTERVAL,
    ) as dag:

    task_instance = PythonOperator( task_id='test_task', python_callable=task)

And the status in the UI:

thanks!

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[kaxil]

You don't need to explicitly call sys.exit(0) -- as Airflow will do it under the hood if the commands succeded.

You can do the other way around, i.e. do a sys.exit(1) when one of the criteria you are checking it against fails

[SergioShz]

Hi, thanks for the quick answer!

Yes, that's what I'm currently doing as fix
but what I'm trying to point out is that if the command succeeds is the same as having a return code of 0 and that it's exactly what sys.exit(0) does explicitly, and in this case I don't see why the task should be marked as failed.

[mik-laj]

@SergioShz Airflow performs a few more operations after the execution of the operator's execute method, e.g. marks a task in the database as completed or executes or executes callbacks. If this code is not executed the task will always be marked as abnormal completion = failed.

[JeffryMAC]

If you change

def task():
    print('doing stuff inside task')
    sys.exit(0)

to:

def task():
    print('doing stuff inside task')
    return

wouldn't that solve the issue?

[SergioShz]

Hi, sorry for the late reply. I finally just avoided to use sys.exit(0). The answer of @mik-laj is what I needed just to understand a little bit more on why avoid to use this exit code.

I'm closing this, thanks all for your help!