Distributed: Frequent CommClosed issues w/ inproc comms on Prefect

Created on 13 Oct 2020 · 8Comments · Source: dask/distributed

In Prefect we're seeing issues with distributed >= 2.23.0 when using inproc comms. Everything works fine in 2.22.0, but in 2.23.0 we see comm closed errors and workers marked as dead.

I have a reproducer with this, but unfortunately haven't been able to reproduce without prefect yet (will keep trying):

Reproducer (requires prefect installed):

from prefect import Flow, task
from prefect.engine.executors import DaskExecutor
from time import sleep


@task
def get_data():
    sleep(0.5)
    return list(range(180))


@task
def inc(user):
    sleep(0.5)
    return user + 1


@task
def accum(data):
    return sum(data)


executor = DaskExecutor(cluster_kwargs={"nanny": False, "memory_limit": 0, "processes": False, "n_workers": 20, "threads_per_worker": 1, "silence_logs": 0})

with Flow("test_flow") as flow:
    accum(inc.map(get_data()))


if __name__ == "__main__":
    flow.run(executor=executor)

Logs on dask & distributed 2.30.0:

[2020-10-12 22:12:30] INFO - prefect.FlowRunner | Beginning Flow run for 'test_flow'
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO -   Scheduler at: inproc://192.168.1.36/40918/1
distributed.scheduler - INFO -   dashboard at:            localhost:8787
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/3
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65097
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-84m1pdwc
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/4
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65098
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-tghwjywi
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/5
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65099
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-aa13mh4n
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/6
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65100
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-tgp7cb_b
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/7
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65101
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-4ncd59jb
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/8
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65102
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-50ukew9q
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/9
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65103
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-0unm12uq
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/10
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65104
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-xvxakipk
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/11
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65105
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-88ryax75
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/12
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65106
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-ojz616ac
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/13
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65107
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-l97cqvxc
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/14
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65108
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-e22w6nsn
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/15
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65109
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-lgan9zf1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/16
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65110
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-t3r_s91p
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/17
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65111
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-vk8f4abf
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/18
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65112
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-vip55j4t
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/19
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65113
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-antub0bv
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/20
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65114
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-t4x30du5
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/21
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65115
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-h9kq54de
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/22
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65116
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-u5ijheja
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/3', name: 12, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/3
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/4', name: 7, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/4
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/5', name: 11, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/5
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/6', name: 6, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/6
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/7', name: 10, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/7
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/8', name: 5, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/8
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/9', name: 9, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/9
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/10', name: 4, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/10
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/11', name: 0, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/11
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/12', name: 3, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/12
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/13', name: 2, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/13
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/14', name: 1, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/14
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/15', name: 19, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/15
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/16', name: 18, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/16
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/17', name: 8, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/17
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/18', name: 17, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/18
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/19', name: 16, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/19
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/20', name: 15, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/20
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/21', name: 14, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/21
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/22', name: 13, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/22
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Receive client connection: Client-05baa1fe-0cd8-11eb-9fd6-acde48001122
distributed.core - INFO - Starting established connection
[2020-10-12 22:12:31] INFO - prefect.TaskRunner | Task 'get_data': Starting task run...
[2020-10-12 22:12:31] INFO - prefect.TaskRunner | Task 'get_data': finished task run for task with final state: 'Success'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc': finished task run for task with final state: 'Mapped'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[0]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[3]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[1]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[11]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[12]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[2]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[4]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[5]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[6]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[0]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[7]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[20]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[8]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[3]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[1]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[12]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[11]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[9]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[2]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[4]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[10]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[5]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[38]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[6]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[50]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[21]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[49]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[39]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[7]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[20]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[8]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[22]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[9]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[23]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[44]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[10]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[45]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[40]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[46]': Starting task run...
distributed.comm.inproc - WARNING - Closing dangling queue in <InProc  local=inproc://192.168.1.36/40918/64 remote=inproc://192.168.1.36/40918/1>
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[38]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[47]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[48]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[50]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[49]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[21]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[39]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[98]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[22]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[23]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[44]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[29]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[99]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[41]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[30]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[40]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[45]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[46]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[47]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[42]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[48]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[43]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[24]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[98]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[27]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[25]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[26]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[100]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[28]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[29]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[99]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[41]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[30]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[78]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[42]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[89]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[43]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[24]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[79]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[101]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[90]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[27]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[25]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[100]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[26]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[28]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[102]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[103]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[84]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[78]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[87]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[89]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[85]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[80]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[101]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[88]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[58]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[86]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[79]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[90]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[69]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[81]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[102]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[59]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[84]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[103]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[70]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[87]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[85]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[82]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[67]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[80]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[64]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[83]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[86]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[58]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[88]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[81]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[69]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[65]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[59]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[60]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[70]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[66]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[118]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[61]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[82]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[64]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[83]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[68]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[129]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[67]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[130]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[65]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[119]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[62]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[63]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[127]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[60]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[118]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[124]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[66]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[61]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[125]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[68]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[129]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[120]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[126]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[121]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[158]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[130]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[119]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[128]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[62]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[63]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[109]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[127]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[124]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[110]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[125]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[159]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[122]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[123]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[126]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[107]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[104]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[120]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[121]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[158]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[105]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[128]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[109]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[106]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[110]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[161]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[160]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[138]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[159]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[108]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[122]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[123]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[139]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[104]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[150]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[149]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[107]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[105]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[106]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[162]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[138]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[161]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[147]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[108]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[160]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[164]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[163]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[165]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[139]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[149]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[150]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[146]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[148]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[141]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[178]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[140]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[179]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[162]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[169]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[170]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[147]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[142]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[164]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[163]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[165]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[145]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[167]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[144]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[146]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[143]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[148]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[141]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[178]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[140]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[179]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[169]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[170]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[168]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[57]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[142]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[166]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[77]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[117]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[145]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[167]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[144]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[131]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[143]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[37]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[75]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[157]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[53]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[97]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[168]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[94]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[57]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[166]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[77]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[117]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[36]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[92]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[131]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[37]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[72]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[111]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[51]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[91]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[75]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[157]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[97]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[53]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[94]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[113]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[153]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[36]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[92]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[74]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[174]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[34]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[134]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[72]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[111]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[51]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[54]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[91]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[114]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[136]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[113]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[153]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[132]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[172]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[52]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[74]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[31]': Starting task run...
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/10
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[174]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[34]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[134]': finished task run for task with final state: 'Success'
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/11
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/12
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/13
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/14
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/15
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/16
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/17
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/18
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/19
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/11', name: 0, memory: 11, processing: 0>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/11
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/10', name: 4, memory: 11, processing: 1>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/10
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/12', name: 3, memory: 11, processing: 2>
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[54]': finished task run for task with final state: 'Success'
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/12
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/13', name: 2, memory: 11, processing: 3>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/13
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/14', name: 1, memory: 11, processing: 4>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/14
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/15', name: 19, memory: 10, processing: 5>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/15
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/16', name: 18, memory: 10, processing: 6>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/16
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/17', name: 8, memory: 11, processing: 7>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/17
distributed.batched - INFO - Batched Comm Closed: 
distributed.batched - INFO - Batched Comm Closed: 
distributed.batched - INFO - Batched Comm Closed: 
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/18', name: 17, memory: 10, processing: 9>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/18
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/19', name: 16, memory: 10, processing: 10>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/19
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/20
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[152]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[114]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[136]': finished task run for task with final state: 'Success'
distributed.batched - INFO - Batched Comm Closed: 
distributed.batched - INFO - Batched Comm Closed: 
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/21
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/22
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[132]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[172]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[52]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[31]': finished task run for task with final state: 'Success'
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/20', name: 15, memory: 10, processing: 13>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/20
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/21', name: 14, memory: 11, processing: 17>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/21
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/22', name: 13, memory: 11, processing: 21>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/22
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[152]': finished task run for task with final state: 'Success'
distributed.worker - INFO - Comm closed
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[95]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[71]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[55]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[96]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[137]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[32]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[33]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[95]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[71]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[55]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[96]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[33]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[137]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[32]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[23]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[7]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[9]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'get_data': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[3]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[10]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[0]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[23]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[9]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[51]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[7]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'get_data': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[8]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[0]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[10]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[3]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[6]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[5]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[1]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[53]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[22]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[51]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[45]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[8]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[4]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[6]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[5]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[12]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[11]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[1]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[22]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[53]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[30]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[2]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[29]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[45]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[40]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[4]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[20]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[12]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[35]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[11]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[41]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[30]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[2]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[29]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[47]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[21]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[40]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[46]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[20]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[31]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[35]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[50]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[37]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[41]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[27]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[47]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[46]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[21]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[54]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[38]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[26]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[50]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[31]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[37]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[27]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[44]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[70]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[28]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[42]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[54]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[38]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[26]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[48]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[49]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[67]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[44]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[28]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[70]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[42]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[90]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[34]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[63]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[25]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[48]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[49]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[36]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[67]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[24]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[69]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[90]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[34]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[63]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[25]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[103]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[52]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[130]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[39]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[36]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[102]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[24]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[69]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[43]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[62]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[103]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[52]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[130]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[39]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[80]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[124]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[66]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[77]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[102]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[97]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[43]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[62]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[64]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[65]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[80]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[66]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[124]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[77]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[57]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[128]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[83]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[97]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[94]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[100]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[64]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[88]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[65]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[84]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[57]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[92]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[128]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[83]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[94]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[100]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[88]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[76]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[109]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[84]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[87]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[79]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[82]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[72]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[92]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[74]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[76]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[109]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[56]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[79]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[87]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[82]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[136]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[72]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[89]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[58]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[86]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[74]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[98]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[68]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[56]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[112]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[136]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[106]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[89]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[58]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[86]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[98]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[68]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[81]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[117]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[101]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[99]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[85]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[112]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[129]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[106]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[118]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[81]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[61]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[101]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[117]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[85]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[99]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[114]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[113]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[129]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[93]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[118]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[91]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[120]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[135]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[61]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[132]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[113]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[114]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[105]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[121]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[93]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[91]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[73]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[120]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[75]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[135]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[116]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[115]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[132]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[107]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[105]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[121]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[125]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[127]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[73]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[123]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[75]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[116]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[115]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[59]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[107]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[138]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[160]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[125]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[127]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[123]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[126]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[134]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[131]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[111]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[59]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[60]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[138]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[165]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[160]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[147]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[126]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[148]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[111]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[134]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[131]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[60]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[104]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[119]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[152]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[78]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[165]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[153]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[147]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[154]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[148]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[163]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[104]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[152]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[119]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[78]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[122]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[153]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[141]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[110]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[133]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[154]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[161]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[163]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[144]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[164]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[122]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[133]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[157]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[141]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[110]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[150]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[161]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[145]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[108]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[164]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[159]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[144]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[142]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[162]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[157]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[169]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[150]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[158]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[145]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[108]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[159]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[140]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[149]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[156]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[142]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[162]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[139]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[151]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[169]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[174]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[158]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[143]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[140]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[149]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[156]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[146]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[168]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[139]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[170]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[151]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[155]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[171]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[174]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[177]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[143]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[172]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[146]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[167]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[168]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[170]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[176]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[166]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[171]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[155]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[177]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[178]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[172]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[179]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[173]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[175]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[167]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[176]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[166]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[178]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[179]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[173]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[175]': finished task run for task with final state: 'Success'
distributed.worker - WARNING -  Compute Failed
Function:  _maybe_run
args:      ('prefect-ea630327465142ab9810ea3f6966882e', <function run_task at 0x7fdaad74c310>)
kwargs:    {'task': <Task: accum>, 'state': None, 'upstream_states': {<Edge(key=data, mapped=False, flattened=False): inc to accum>: <Mapped: "Ready to proceed with mapping.">}, 'context': {'config': <Box: {'debug': False, 'home_dir': '/Users/jcristharif/.prefect', 'backend': 'cloud', 'server': {'host': 'http://localhost', 'port': 4200, 'host_port': 4200, 'endpoint': 'http://localhost:4200', 'database': {'host': 'localhost', 'port': 5432, 'host_port': 5432, 'name': 'prefect_server', 'username': 'prefect', 'password': 'test-password', 'connection_url': 'postgresql://prefect:test-password@localhost:5432/prefect_server', 'volume_path': '/Users/jcristharif/.prefect/pg_data'}, 'graphql': {'host': '0.0.0.0', 'port': 4201, 'host_port': 4201, 'debug': False, 'path': '/graphql/'}, 'hasura': {'host': 'localhost', 'port': 3000, 'host_port': 3000, 'admin_secret': '', 'claims_namespace': 'hasura-claims', 'graphql_url': 'http://localhost:3000/v1alpha1/graphql', 'ws_url': 'ws://localhost:3000/v1alpha1/graphql',
Exception: AttributeError("'NoneType' object has no attribute 'result'")

distributed.scheduler - INFO - Remove client variable-prefect-ea630327465142ab9810ea3f6966882e
distributed.scheduler - INFO - Remove client Client-05baa1fe-0cd8-11eb-9fd6-acde48001122
distributed.scheduler - INFO - Remove client Client-05baa1fe-0cd8-11eb-9fd6-acde48001122
distributed.scheduler - INFO - Close client connection: Client-05baa1fe-0cd8-11eb-9fd6-acde48001122

Source

jcrist

Most helpful comment

Ok, I've figured this out, it's a bug in the connect timeout logic in distributed. PR incoming.

jcrist on 14 Oct 2020

👍2 👀1

All 8 comments

Thanks @jcrist ! I can reproduce. Will this example scale up/down the number of workers or will the worker number stay at 20 ?

quasiben on 13 Oct 2020

With that configuration it should stay at 20.

jcrist on 13 Oct 2020

I see things near the end when the job is still running but workers are removed:

[2020-10-13 19:00:35] INFO - prefect.TaskRunner | Task 'inc[45]': finished task run for task with final state: 'Success'
distributed.batched - INFO - Batched Comm Closed:
distributed.batched - INFO - Batched Comm Closed:
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.scheduler - INFO - Remove worker <Worker 'inproc://10.33.227.162/20411/44', name: 27, memory: 5, processing: 11>
distributed.core - INFO - Removing comms to inproc://10.33.227.162/20411/44
distributed.scheduler - INFO - Remove worker <Worker 'inproc://10.33.227.162/20411/46', name: 43, memory: 5, processing: 11>
distributed.core - INFO - Removing comms to inproc://10.33.227.162/20411/46
distributed.scheduler - INFO - Remove worker <Worker 'inproc://10.33.227.162/20411/51', name: 8, memory: 5, processing: 12>
distributed.core - INFO - Removing comms to inproc://10.33.227.162/20411/51
distributed.scheduler - INFO - Remove worker <Worker 'inproc://10.33.227.162/20411/48', name: 48, memory: 5, processing: 14>
distributed.core - INFO - Removing comms to inproc://10.33.227.162/20411/48
distributed.scheduler - INFO - Remove worker <Worker 'inproc://10.33.227.162/20411/47', name: 21, memory: 1, processing: 15>
distributed.core - INFO - Removing comms to inproc://10.33.227.162/20411/47
distributed.scheduler - INFO - Remove worker <Worker 'inproc://10.33.227.162/20411/49', name: 42, memory: 1, processing: 17>
distributed.core - INFO - Removing comms to inproc://10.33.227.162/20411/49
distributed.scheduler - INFO - Remove worker <Worker 'inproc://10.33.227.162/20411/50', name: 25, memory: 1, processing: 19>
distributed.core - INFO - Removing comms to inproc://10.33.227.162/20411/50
distributed.scheduler - INFO - Remove worker <Worker 'inproc://10.33.227.162/20411/45', name: 6, memory: 1, processing: 21>
distributed.core - INFO - Removing comms to inproc://10.33.227.162/20411/45
distributed.scheduler - INFO - Remove worker <Worker 'inproc://10.33.227.162/20411/52', name: 40, memory: 1, processing: 25>
distributed.core - INFO - Removing comms to inproc://10.33.227.162/20411/52
distributed.batched - INFO - Batched Comm Closed:
[2020-10-13 19:00:35] INFO - prefect.DaskExecutor | Stopping executor, waiting for 145 active tasks to complete
[2020-10-13 19:00:35] INFO - prefect.TaskRunner | Task 'inc[76]': Starting task run...
[2020-10-13 19:00:35] INFO - prefect.TaskRunner | Task 'get_data': Starting task run...
[2020-10-13 19:00:35] INFO - prefect.TaskRunner | Task 'inc[0]': Starting task run...
[2020-10-13 19:00:35] INFO - prefect.TaskRunner | Task 'inc[14]': Starting task run...
[2020-10-13 19:00:35] INFO - prefect.TaskRunner | Task 'inc[3]': Starting task run...
[2020-10-13 19:00:35] INFO - prefect.TaskRunner | Task 'inc[76]': finished task run for task with final state: 'Success'
[2020-10-13 19:00:35] INFO - prefect.TaskRunner | Task 'get_data': finished task run for task with final state: 'Success'
[2020-10-13 19:00:35] INFO - prefect.TaskRunner | Task 'inc[0]': finished task run for task with final state: 'Success'
[2020-10-13 19:00:35] INFO - prefect.TaskRunner | Task 'inc[14]': finished task run for task with final state: 'Success'
[2020-10-13 19:00:35] INFO - prefect.TaskRunner | Task 'inc[3]': finished task run for task with final state: 'Success'
distributed.scheduler - INFO - Remove client variable-prefect-7a279b1f44d74b8cbdb4ef5747954fb4
distributed.scheduler - INFO - Remove client Client-58844f8e-0d86-11eb-8fbb-d8c49764f61b
distributed.scheduler - INFO - Remove client Client-58844f8e-0d86-11eb-8fbb-d8c49764f61b
distributed.scheduler - INFO - Close client connection: Client-58844f8e-0d86-11eb-8fbb-d8c49764f61b
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7fa495013790>>, <Task finished name='Task-4004' coro=<SpecCluster._correct_state_internal() done, defined
 at /datasets/bzaitlen/GitRepos/distributed/distributed/deploy/spec.py:320> exception=OSError("Timed out trying to connect to 'inproc://10.33.227.162/20411/1' after 10 s: Timed out trying to connect to 'inproc://10.33.227.162/20411/1' after 10 s: connect() didn't finish i
n time")>)
Traceback (most recent call last):
  File "/datasets/bzaitlen/GitRepos/distributed/distributed/comm/core.py", line 322, in connect
    _raise(error)
  File "/datasets/bzaitlen/GitRepos/distributed/distributed/comm/core.py", line 275, in _raise
    raise IOError(msg)
OSError: Timed out trying to connect to 'inproc://10.33.227.162/20411/1' after 10 s: connect() didn't finish in time

quasiben on 13 Oct 2020

Yeah, but no scale/adapt calls are being made, my assumption was this was due to comms failing and workers being marked as dead. Haven't dug in deeper than that.

jcrist on 13 Oct 2020

Here's a new reproducer that makes the dask-related things more visible (not buried in prefect's DaskExecutor). Still no reproducible example sans prefect.

from prefect import Flow, task
from prefect.engine.executors import Executor
from time import sleep

from contextlib import contextmanager

from prefect.engine.executors.base import Executor


class CustomExecutor(Executor):
    @contextmanager
    def start(self):
        from distributed import Client

        try:
            with Client(nanny=False, memory_limit=0, processes=False, n_workers=20, threads_per_worker=1, silence_logs=0) as client:
                self.client = client
                yield
        finally:
            self.client = None

    def submit(self, fn, *args, **kwargs):
        kwargs.pop("extra_context", None)
        return self.client.submit(fn, *args, **kwargs)

    def wait(self, futures):
        return self.client.gather(futures)


@task
def get_data():
    sleep(0.5)
    return list(range(180))


@task
def inc(user):
    sleep(0.5)
    return user + 1


@task
def accum(data):
    return sum(data)


with Flow("test_flow") as flow:
    accum(inc.map(get_data()))


if __name__ == "__main__":
    flow.run(executor=CustomExecutor())

Edit: nevermind, that doesn't seem to fail for me in the same way. Interesting, perhaps there is something tricky on the prefect side causing this

jcrist on 14 Oct 2020

Ok, I've figured this out, it's a bug in the connect timeout logic in distributed. PR incoming.

jcrist on 14 Oct 2020

👍2 👀1

See #4167.

jcrist on 14 Oct 2020

Fixed by #4176, closing.

jcrist on 4 Nov 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Provide clearer message for multiprocess error

djhoese · 3Comments

Bokeh RC Updates Breaks Dashboard

quasiben · 5Comments

Trigger callback on dask.distributed.Variable.set

sofroniewn · 5Comments

Clarification of the task graph: are futures allowed in tuples that aren't tasks?

madsbk · 4Comments

Timeseries Resource Usage Tracking

quasiben · 7Comments