Webodm: Deleting tasks during upload causes them to get stuck in a 'Deleting' state

[ChipwizBen]

How did you install WebODM? (Docker, natively, ...)?

Native

What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/)

Any

What is the problem?

When a task is uploading and you hit "Delete", it locks in this state.

What should be the expected behavior?

Task should be deleted.

How can we reproduce this? (What steps did you do to trigger the problem? What parameters are you using for processing? If possible please include a copy of your dataset uploaded on Google Drive or Dropbox. Be detailed)

Start task, wait for "Uploading to processing node". Delete task.

[ChipwizBen]

This is still an issue, but I've also found that tasks get stuck in a deleting state when deleted part-processed, not just during upload.

[pierotofy]

Any logs? Error messages?

The webodm-gunicorn and webodm-celery service logs would be the ones that could have some clues.

I think you can get the logs via:

journalctl -u webodm-celery.service

[ChipwizBen]

Nothing useful. I made a fake job [fd] to test the cancel/delete but it worked fine. I then tried cancelling a real job (that was stuck/frozen) at 13:45, but nothing showed for it in the logs when deleted. It's not clear when it froze or why, but it definitely didn't run out of resources. I've included some logs from earlier in the night - perhaps they might be useful. WebODM and NodeODM are on different physical servers, but I'm not using CluserODM.

Console:

INDEXING: 52,000,000 points processed; 52,000,000 points written; 227.94 seconds passed

INDEXING: 55,000,000 points processed; 55,000,000 points written; 236.618 seconds passed

INDEXING: 57,000,000 points processed; 57,000,000 points written; 241.38 seconds passed

INDEXING: 58,000,000 points processed; 58,000,000 points written; 243.698 seconds passed

Logs:

Jun 10 03:14:44 survey celery[1774]: [2019-06-10 03:14:44,614: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 03:14:44 survey celery[1774]: Traceback (most recent call last):
Jun 10 03:14:44 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 03:14:44 survey celery[1774]:     human_status(exitcode)),
Jun 10 03:14:44 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 03:47:35 survey celery[1774]: [2019-06-10 03:47:35,267: ERROR/ForkPoolWorker-3555] Task worker.tasks.process_task[2bf435e7-64e7-40b8-985f-e952f49e4ba5] raised unexpected: UnboundLocalError("local variable 'have_lock' referenced before assignment",)
Jun 10 03:47:35 survey celery[1774]: Traceback (most recent call last):
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/worker/tasks.py", line 65, in process_task
Jun 10 03:47:35 survey celery[1774]:     have_lock = lock.acquire(blocking=False)
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/lock.py", line 111, in acquire
Jun 10 03:47:35 survey celery[1774]:     if self.do_acquire(token):
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/lock.py", line 258, in do_acquire
Jun 10 03:47:35 survey celery[1774]:     client=self.redis))
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 2944, in __call__
Jun 10 03:47:35 survey celery[1774]:     return client.evalsha(self.sha, len(keys), *args)
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 2079, in evalsha
Jun 10 03:47:35 survey celery[1774]:     return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 668, in execute_command
Jun 10 03:47:35 survey celery[1774]:     return self.parse_response(connection, command_name, **options)
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 680, in parse_response
Jun 10 03:47:35 survey celery[1774]:     response = connection.read_response()
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 624, in read_response
Jun 10 03:47:35 survey celery[1774]:     response = self._parser.read_response()
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 284, in read_response
Jun 10 03:47:35 survey celery[1774]:     response = self._buffer.readline()
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 216, in readline
Jun 10 03:47:35 survey celery[1774]:     self._read_from_socket()
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 174, in _read_from_socket
Jun 10 03:47:35 survey celery[1774]:     data = recv(self._sock, socket_read_size)
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/_compat.py", line 79, in recv
Jun 10 03:47:35 survey celery[1774]:     return sock.recv(*args, **kwargs)
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/common.py", line 125, in _shutdown_cleanup
Jun 10 03:47:35 survey celery[1774]:     sys.exit(-(256 - signum))
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 280, in exit
Jun 10 03:47:35 survey celery[1774]:     return _exit()
Jun 10 03:47:35 survey celery[1774]: SystemExit
Jun 10 03:47:35 survey celery[1774]: During handling of the above exception, another exception occurred:
Jun 10 03:47:35 survey celery[1774]: Traceback (most recent call last):
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/celery/app/trace.py", line 374, in trace_task
Jun 10 03:47:35 survey celery[1774]:     R = retval = fun(*args, **kwargs)
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/celery/app/trace.py", line 629, in __protected_call__
Jun 10 03:47:35 survey celery[1774]:     return self.run(*args, **kwargs)
Jun 10 03:47:35 survey celery[1774]:   File "/webodm/worker/tasks.py", line 85, in process_task
Jun 10 03:47:35 survey celery[1774]:     if have_lock:
Jun 10 03:47:35 survey celery[1774]: UnboundLocalError: local variable 'have_lock' referenced before assignment
Jun 10 04:08:05 survey celery[1774]: [2019-06-10 04:08:05,518: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 04:08:05 survey celery[1774]: Traceback (most recent call last):
Jun 10 04:08:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 04:08:05 survey celery[1774]:     human_status(exitcode)),
Jun 10 04:08:05 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 04:23:55 survey celery[1774]: [2019-06-10 04:23:55,639: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 04:23:55 survey celery[1774]: Traceback (most recent call last):
Jun 10 04:23:55 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 04:23:55 survey celery[1774]:     human_status(exitcode)),
Jun 10 04:23:55 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 05:04:38 survey celery[1774]: [2019-06-10 05:04:38,437: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 05:04:38 survey celery[1774]: Traceback (most recent call last):
Jun 10 05:04:38 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 05:04:38 survey celery[1774]:     human_status(exitcode)),
Jun 10 05:04:38 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 06:23:54 survey celery[1774]: [2019-06-10 06:23:54,631: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 06:23:54 survey celery[1774]: Traceback (most recent call last):
Jun 10 06:23:54 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 06:23:54 survey celery[1774]:     human_status(exitcode)),
Jun 10 06:23:54 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 06:34:49 survey celery[1774]: [2019-06-10 06:34:49,761: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 06:34:49 survey celery[1774]: Traceback (most recent call last):
Jun 10 06:34:49 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 06:34:49 survey celery[1774]:     human_status(exitcode)),
Jun 10 06:34:49 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 07:39:34 survey celery[1774]: [2019-06-10 07:39:34,138: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 07:39:34 survey celery[1774]: Traceback (most recent call last):
Jun 10 07:39:34 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 07:39:34 survey celery[1774]:     human_status(exitcode)),
Jun 10 07:39:34 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 07:46:04 survey celery[1774]: [2019-06-10 07:46:04,226: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 07:46:04 survey celery[1774]: Traceback (most recent call last):
Jun 10 07:46:04 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 07:46:04 survey celery[1774]:     human_status(exitcode)),
Jun 10 07:46:04 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 07:55:24 survey celery[1774]: [2019-06-10 07:55:24,380: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 07:55:24 survey celery[1774]: Traceback (most recent call last):
Jun 10 07:55:24 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 07:55:24 survey celery[1774]:     human_status(exitcode)),
Jun 10 07:55:24 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 07:58:49 survey celery[1774]: [2019-06-10 07:58:49,457: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 07:58:49 survey celery[1774]: Traceback (most recent call last):
Jun 10 07:58:49 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 07:58:49 survey celery[1774]:     human_status(exitcode)),
Jun 10 07:58:49 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 08:19:59 survey celery[1774]: [2019-06-10 08:19:59,856: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 08:19:59 survey celery[1774]: Traceback (most recent call last):
Jun 10 08:19:59 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 08:19:59 survey celery[1774]:     human_status(exitcode)),
Jun 10 08:19:59 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 08:47:05 survey celery[1774]: [2019-06-10 08:47:05,334: ERROR/ForkPoolWorker-5329] Task worker.tasks.process_task[2d2c0642-cfb6-4311-a348-3ea722cac12f] raised unexpected: UnboundLocalError("local variable 'have_lock' referenced before assignment",)
Jun 10 08:47:05 survey celery[1774]: Traceback (most recent call last):
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/worker/tasks.py", line 65, in process_task
Jun 10 08:47:05 survey celery[1774]:     have_lock = lock.acquire(blocking=False)
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/lock.py", line 111, in acquire
Jun 10 08:47:05 survey celery[1774]:     if self.do_acquire(token):
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/lock.py", line 258, in do_acquire
Jun 10 08:47:05 survey celery[1774]:     client=self.redis))
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 2944, in __call__
Jun 10 08:47:05 survey celery[1774]:     return client.evalsha(self.sha, len(keys), *args)
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 2079, in evalsha
Jun 10 08:47:05 survey celery[1774]:     return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 668, in execute_command
Jun 10 08:47:05 survey celery[1774]:     return self.parse_response(connection, command_name, **options)
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 680, in parse_response
Jun 10 08:47:05 survey celery[1774]:     response = connection.read_response()
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 624, in read_response
Jun 10 08:47:05 survey celery[1774]:     response = self._parser.read_response()
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 284, in read_response
Jun 10 08:47:05 survey celery[1774]:     response = self._buffer.readline()
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 214, in readline
Jun 10 08:47:05 survey celery[1774]:     while not data.endswith(SYM_CRLF):
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/common.py", line 125, in _shutdown_cleanup
Jun 10 08:47:05 survey celery[1774]:     sys.exit(-(256 - signum))
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 280, in exit
Jun 10 08:47:05 survey celery[1774]:     return _exit()
Jun 10 08:47:05 survey celery[1774]: SystemExit
Jun 10 08:47:05 survey celery[1774]: During handling of the above exception, another exception occurred:
Jun 10 08:47:05 survey celery[1774]: Traceback (most recent call last):
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/celery/app/trace.py", line 374, in trace_task
Jun 10 08:47:05 survey celery[1774]:     R = retval = fun(*args, **kwargs)
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/celery/app/trace.py", line 629, in __protected_call__
Jun 10 08:47:05 survey celery[1774]:     return self.run(*args, **kwargs)
Jun 10 08:47:05 survey celery[1774]:   File "/webodm/worker/tasks.py", line 85, in process_task
Jun 10 08:47:05 survey celery[1774]:     if have_lock:
Jun 10 08:47:05 survey celery[1774]: UnboundLocalError: local variable 'have_lock' referenced before assignment
Jun 10 09:35:38 survey celery[1774]: INFO Executing grass script from /webodm/app/media/tmp/tmpncmak6qx_grass_engine: /usr/bin/grass76 -c /webodm/app/media/project/362/task/b2d49366-b7ec-4f96-8f99-8c047c1399dd/assets/odm_dem/dsm.tif location --exec sh script.sh
Jun 10 09:35:38 survey celery[1774]: [2019-06-10 09:35:38,668: INFO/ForkPoolWorker-5612] Executing grass script from /webodm/app/media/tmp/tmpncmak6qx_grass_engine: /usr/bin/grass76 -c /webodm/app/media/project/362/task/b2d49366-b7ec-4f96-8f99-8c047c1399dd/assets/odm_dem
Jun 10 09:40:23 survey celery[1774]: INFO Executing grass script from /webodm/app/media/tmp/tmpxo303c94_grass_engine: /usr/bin/grass76 -c /webodm/app/media/project/362/task/b2d49366-b7ec-4f96-8f99-8c047c1399dd/assets/odm_dem/dsm.tif location --exec sh script.sh
Jun 10 09:40:23 survey celery[1774]: [2019-06-10 09:40:23,496: INFO/ForkPoolWorker-5648] Executing grass script from /webodm/app/media/tmp/tmpxo303c94_grass_engine: /usr/bin/grass76 -c /webodm/app/media/project/362/task/b2d49366-b7ec-4f96-8f99-8c047c1399dd/assets/odm_dem
Jun 10 09:40:33 survey celery[1774]: INFO Executing grass script from /webodm/app/media/tmp/tmphxrg8ze9_grass_engine: /usr/bin/grass76 -c /webodm/app/media/project/362/task/b2d49366-b7ec-4f96-8f99-8c047c1399dd/assets/odm_dem/dsm.tif location --exec sh script.sh
Jun 10 09:40:33 survey celery[1774]: [2019-06-10 09:40:33,563: INFO/ForkPoolWorker-5651] Executing grass script from /webodm/app/media/tmp/tmphxrg8ze9_grass_engine: /usr/bin/grass76 -c /webodm/app/media/project/362/task/b2d49366-b7ec-4f96-8f99-8c047c1399dd/assets/odm_dem
Jun 10 10:04:10 survey celery[1774]: INFO Executing grass script from /webodm/app/media/tmp/tmp7nhg452t_grass_engine: /usr/bin/grass76 -c /webodm/app/media/project/243/task/327817c2-0e39-438c-9469-18b2736fddb4/assets/odm_dem/dsm.tif location --exec sh script.sh
Jun 10 10:04:10 survey celery[1774]: [2019-06-10 10:04:10,772: INFO/ForkPoolWorker-5778] Executing grass script from /webodm/app/media/tmp/tmp7nhg452t_grass_engine: /usr/bin/grass76 -c /webodm/app/media/project/243/task/327817c2-0e39-438c-9469-18b2736fddb4/assets/odm_dem
Jun 10 10:07:14 survey celery[1774]: INFO Executing grass script from /webodm/app/media/tmp/tmppjpli0o8_grass_engine: /usr/bin/grass76 -c /webodm/app/media/project/243/task/327817c2-0e39-438c-9469-18b2736fddb4/assets/odm_dem/dtm.tif location --exec sh script.sh
Jun 10 10:07:14 survey celery[1774]: [2019-06-10 10:07:14,474: INFO/ForkPoolWorker-5802] Executing grass script from /webodm/app/media/tmp/tmppjpli0o8_grass_engine: /usr/bin/grass76 -c /webodm/app/media/project/243/task/327817c2-0e39-438c-9469-18b2736fddb4/assets/odm_dem
Jun 10 10:27:48 survey celery[1774]: [2019-06-10 10:27:48,315: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 10:27:48 survey celery[1774]: Traceback (most recent call last):
Jun 10 10:27:48 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 10:27:48 survey celery[1774]:     human_status(exitcode)),
Jun 10 10:27:48 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 10:37:08 survey celery[1774]: [2019-06-10 10:37:08,448: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 10:37:08 survey celery[1774]: Traceback (most recent call last):
Jun 10 10:37:08 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 10:37:08 survey celery[1774]:     human_status(exitcode)),
Jun 10 10:37:08 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 10:51:34 survey celery[1774]: [2019-06-10 10:51:34,182: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 10:51:34 survey celery[1774]: Traceback (most recent call last):
Jun 10 10:51:34 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 10:51:34 survey celery[1774]:     human_status(exitcode)),
Jun 10 10:51:34 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 11:07:44 survey celery[1774]: [2019-06-10 11:07:44,442: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 11:07:44 survey celery[1774]: Traceback (most recent call last):
Jun 10 11:07:44 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 11:07:44 survey celery[1774]:     human_status(exitcode)),
Jun 10 11:07:44 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 11:08:19 survey celery[1774]: [2019-06-10 11:08:19,382: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 11:08:19 survey celery[1774]: Traceback (most recent call last):
Jun 10 11:08:19 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 11:08:19 survey celery[1774]:     human_status(exitcode)),
Jun 10 11:08:19 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 11:14:21 survey celery[1774]: [2019-06-10 11:14:21,939: ERROR/ForkPoolWorker-6227] Task worker.tasks.process_task[d7b95577-9a6b-4897-8cfb-4045470927c0] raised unexpected: UnboundLocalError("local variable 'have_lock' referenced before assignment",)
Jun 10 11:14:21 survey celery[1774]: Traceback (most recent call last):
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/worker/tasks.py", line 65, in process_task
Jun 10 11:14:21 survey celery[1774]:     have_lock = lock.acquire(blocking=False)
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/lock.py", line 111, in acquire
Jun 10 11:14:21 survey celery[1774]:     if self.do_acquire(token):
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/lock.py", line 258, in do_acquire
Jun 10 11:14:21 survey celery[1774]:     client=self.redis))
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 2944, in __call__
Jun 10 11:14:21 survey celery[1774]:     return client.evalsha(self.sha, len(keys), *args)
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 2079, in evalsha
Jun 10 11:14:21 survey celery[1774]:     return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 668, in execute_command
Jun 10 11:14:21 survey celery[1774]:     return self.parse_response(connection, command_name, **options)
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 680, in parse_response
Jun 10 11:14:21 survey celery[1774]:     response = connection.read_response()
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 624, in read_response
Jun 10 11:14:21 survey celery[1774]:     response = self._parser.read_response()
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 284, in read_response
Jun 10 11:14:21 survey celery[1774]:     response = self._buffer.readline()
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 216, in readline
Jun 10 11:14:21 survey celery[1774]:     self._read_from_socket()
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 174, in _read_from_socket
Jun 10 11:14:21 survey celery[1774]:     data = recv(self._sock, socket_read_size)
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/_compat.py", line 79, in recv
Jun 10 11:14:21 survey celery[1774]:     return sock.recv(*args, **kwargs)
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/common.py", line 125, in _shutdown_cleanup
Jun 10 11:14:21 survey celery[1774]:     sys.exit(-(256 - signum))
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 280, in exit
Jun 10 11:14:21 survey celery[1774]:     return _exit()
Jun 10 11:14:21 survey celery[1774]: SystemExit
Jun 10 11:14:21 survey celery[1774]: During handling of the above exception, another exception occurred:
Jun 10 11:14:21 survey celery[1774]: Traceback (most recent call last):
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/celery/app/trace.py", line 374, in trace_task
Jun 10 11:14:21 survey celery[1774]:     R = retval = fun(*args, **kwargs)
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/celery/app/trace.py", line 629, in __protected_call__
Jun 10 11:14:21 survey celery[1774]:     return self.run(*args, **kwargs)
Jun 10 11:14:21 survey celery[1774]:   File "/webodm/worker/tasks.py", line 85, in process_task
Jun 10 11:14:21 survey celery[1774]:     if have_lock:
Jun 10 11:14:21 survey celery[1774]: UnboundLocalError: local variable 'have_lock' referenced before assignment
Jun 10 11:18:01 survey celery[1774]: [2019-06-10 11:18:01,968: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 11:18:01 survey celery[1774]: Traceback (most recent call last):
Jun 10 11:18:01 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 11:18:01 survey celery[1774]:     human_status(exitcode)),
Jun 10 11:18:01 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 11:25:17 survey celery[1774]: [2019-06-10 11:25:17,053: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 11:25:17 survey celery[1774]: Traceback (most recent call last):
Jun 10 11:25:17 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 11:25:17 survey celery[1774]:     human_status(exitcode)),
Jun 10 11:25:17 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 11:47:12 survey celery[1774]: [2019-06-10 11:47:12,218: ERROR/ForkPoolWorker-6424] Task worker.tasks.process_task[59e39c60-bb4d-41ab-ae9b-031404aaebfb] raised unexpected: UnboundLocalError("local variable 'have_lock' referenced before assignment",)
Jun 10 11:47:12 survey celery[1774]: Traceback (most recent call last):
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/worker/tasks.py", line 65, in process_task
Jun 10 11:47:12 survey celery[1774]:     have_lock = lock.acquire(blocking=False)
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/lock.py", line 111, in acquire
Jun 10 11:47:12 survey celery[1774]:     if self.do_acquire(token):
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/lock.py", line 258, in do_acquire
Jun 10 11:47:12 survey celery[1774]:     client=self.redis))
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 2944, in __call__
Jun 10 11:47:12 survey celery[1774]:     return client.evalsha(self.sha, len(keys), *args)
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 2079, in evalsha
Jun 10 11:47:12 survey celery[1774]:     return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 668, in execute_command
Jun 10 11:47:12 survey celery[1774]:     return self.parse_response(connection, command_name, **options)
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/client.py", line 680, in parse_response
Jun 10 11:47:12 survey celery[1774]:     response = connection.read_response()
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 624, in read_response
Jun 10 11:47:12 survey celery[1774]:     response = self._parser.read_response()
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 284, in read_response
Jun 10 11:47:12 survey celery[1774]:     response = self._buffer.readline()
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 216, in readline
Jun 10 11:47:12 survey celery[1774]:     self._read_from_socket()
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/redis/connection.py", line 176, in _read_from_socket
Jun 10 11:47:12 survey celery[1774]:     if isinstance(data, bytes) and len(data) == 0:
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/common.py", line 125, in _shutdown_cleanup
Jun 10 11:47:12 survey celery[1774]:     sys.exit(-(256 - signum))
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 280, in exit
Jun 10 11:47:12 survey celery[1774]:     return _exit()
Jun 10 11:47:12 survey celery[1774]: SystemExit
Jun 10 11:47:12 survey celery[1774]: During handling of the above exception, another exception occurred:
Jun 10 11:47:12 survey celery[1774]: Traceback (most recent call last):
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/celery/app/trace.py", line 374, in trace_task
Jun 10 11:47:12 survey celery[1774]:     R = retval = fun(*args, **kwargs)
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/celery/app/trace.py", line 629, in __protected_call__
Jun 10 11:47:12 survey celery[1774]:     return self.run(*args, **kwargs)
Jun 10 11:47:12 survey celery[1774]:   File "/webodm/worker/tasks.py", line 85, in process_task
Jun 10 11:47:12 survey celery[1774]:     if have_lock:
Jun 10 11:47:12 survey celery[1774]: UnboundLocalError: local variable 'have_lock' referenced before assignment
Jun 10 12:45:47 survey celery[1774]: [2019-06-10 12:45:47,656: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 12:45:47 survey celery[1774]: Traceback (most recent call last):
Jun 10 12:45:47 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 12:45:47 survey celery[1774]:     human_status(exitcode)),
Jun 10 12:45:47 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 13:06:42 survey celery[1774]: [2019-06-10 13:06:42,831: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 13:06:42 survey celery[1774]: Traceback (most recent call last):
Jun 10 13:06:42 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 13:06:42 survey celery[1774]:     human_status(exitcode)),
Jun 10 13:06:42 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 13:19:57 survey celery[1774]: [2019-06-10 13:19:57,929: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 13:19:57 survey celery[1774]: Traceback (most recent call last):
Jun 10 13:19:57 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 13:19:57 survey celery[1774]:     human_status(exitcode)),
Jun 10 13:19:57 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 13:21:07 survey celery[1774]: [2019-06-10 13:21:07,938: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
Jun 10 13:21:07 survey celery[1774]: Traceback (most recent call last):
Jun 10 13:21:07 survey celery[1774]:   File "/webodm/python3-venv/lib/python3.5/site-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
Jun 10 13:21:07 survey celery[1774]:     human_status(exitcode)),
Jun 10 13:21:07 survey celery[1774]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
Jun 10 13:21:57 survey celery[1774]: INFO Removing Task [NDVI 190605] (671665d6-577a-4aa0-b463-676ee71b21c8)
Jun 10 13:21:57 survey celery[1774]: [2019-06-10 13:21:57,481: INFO/ForkPoolWorker-6976] Removing Task [NDVI 190605] (671665d6-577a-4aa0-b463-676ee71b21c8)
Jun 10 13:21:57 survey celery[1774]: INFO Cleaning up OAM datastore for task 671665d6-577a-4aa0-b463-676ee71b21c8
Jun 10 13:21:57 survey celery[1774]: [2019-06-10 13:21:57,785: INFO/ForkPoolWorker-6976] Cleaning up OAM datastore for task 671665d6-577a-4aa0-b463-676ee71b21c8
Jun 10 13:23:03 survey celery[1774]: INFO Automatically assigned processing node proc02.nwk1.com to Task [NDVI 190605] (1ae7d4f2-477c-4d8f-ab55-ab1daf36722c)
Jun 10 13:23:03 survey celery[1774]: [2019-06-10 13:23:03,865: INFO/ForkPoolWorker-6986] Automatically assigned processing node proc02.nwk1.com to Task [NDVI 190605] (1ae7d4f2-477c-4d8f-ab55-ab1daf36722c)
Jun 10 13:23:03 survey celery[1774]: INFO Processing... Task [NDVI 190605] (1ae7d4f2-477c-4d8f-ab55-ab1daf36722c)
Jun 10 13:23:03 survey celery[1774]: [2019-06-10 13:23:03,876: INFO/ForkPoolWorker-6986] Processing... Task [NDVI 190605] (1ae7d4f2-477c-4d8f-ab55-ab1daf36722c)
Jun 10 13:23:04 survey celery[1774]: INFO Automatically assigned processing node proc01.nwk1.com to Task [NDVI 190605 (Cloned)] (15158489-fd8e-49db-9b43-397c884c2d72)
Jun 10 13:23:04 survey celery[1774]: [2019-06-10 13:23:04,994: INFO/ForkPoolWorker-6982] Automatically assigned processing node proc01.nwk1.com to Task [NDVI 190605 (Cloned)] (15158489-fd8e-49db-9b43-397c884c2d72)
Jun 10 13:23:05 survey celery[1774]: INFO Processing... Task [NDVI 190605 (Cloned)] (15158489-fd8e-49db-9b43-397c884c2d72)
Jun 10 13:23:05 survey celery[1774]: [2019-06-10 13:23:05,006: INFO/ForkPoolWorker-6982] Processing... Task [NDVI 190605 (Cloned)] (15158489-fd8e-49db-9b43-397c884c2d72)
Jun 10 13:38:30 survey celery[1774]: INFO Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/21A8501A_6738_11E9_AE1B_7C5735DC49B0.JPG to 2048x1365
Jun 10 13:38:30 survey celery[1774]: [2019-06-10 13:38:30,844: INFO/ForkPoolWorker-7097] Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/21A8501A_6738_11E9_AE1B_7C5735DC49B0.JPG to 2048x1365
Jun 10 13:38:31 survey celery[1774]: INFO Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/A3F80E38_6739_11E9_BDF0_EE5935DC49B0.JPG to 2048x1365
Jun 10 13:38:31 survey celery[1774]: [2019-06-10 13:38:31,521: INFO/ForkPoolWorker-7097] Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/A3F80E38_6739_11E9_BDF0_EE5935DC49B0.JPG to 2048x1365
Jun 10 13:38:32 survey celery[1774]: INFO Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/30B8A1B2_6739_11E9_880A_3E5935DC49B0.JPG to 2048x1365
Jun 10 13:38:32 survey celery[1774]: [2019-06-10 13:38:32,176: INFO/ForkPoolWorker-7097] Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/30B8A1B2_6739_11E9_880A_3E5935DC49B0.JPG to 2048x1365
Jun 10 13:38:32 survey celery[1774]: INFO Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/FDD67E60_6736_11E9_8816_B35535DC49B0.JPG to 2048x1365
Jun 10 13:38:32 survey celery[1774]: [2019-06-10 13:38:32,846: INFO/ForkPoolWorker-7097] Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/FDD67E60_6736_11E9_8816_B35535DC49B0.JPG to 2048x1365
Jun 10 13:38:33 survey celery[1774]: INFO Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/62598A66_6738_11E9_B4FD_E05735DC49B0.JPG to 2048x1365
Jun 10 13:38:33 survey celery[1774]: [2019-06-10 13:38:33,585: INFO/ForkPoolWorker-7097] Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/62598A66_6738_11E9_B4FD_E05735DC49B0.JPG to 2048x1365
Jun 10 13:38:34 survey celery[1774]: INFO Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/E407954C_673A_11E9_B9D8_C45B35DC49B0.jpg to 2048x1365
Jun 10 13:38:34 survey celery[1774]: [2019-06-10 13:38:34,376: INFO/ForkPoolWorker-7097] Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/E407954C_673A_11E9_B9D8_C45B35DC49B0.jpg to 2048x1365
Jun 10 13:38:35 survey celery[1774]: INFO Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/EDD5407A_6734_11E9_847B_855235DC49B0.JPG to 2048x1365
Jun 10 13:38:35 survey celery[1774]: [2019-06-10 13:38:35,104: INFO/ForkPoolWorker-7097] Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/EDD5407A_6734_11E9_847B_855235DC49B0.JPG to 2048x1365
Jun 10 13:38:35 survey celery[1774]: INFO Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/9A6BFAB2_6731_11E9_8755_5E4E35DC49B0.JPG to 2048x1365
Jun 10 13:38:35 survey celery[1774]: [2019-06-10 13:38:35,827: INFO/ForkPoolWorker-7097] Resized /webodm/app/media/project/405/task/c893b6b9-623d-41fb-a880-71afe009e1d4/9A6BFAB2_6731_11E9_8755_5E4E35DC49B0.JPG to 2048x1365
Jun 10 13:38:35 survey celery[1774]: INFO Processing... Task [fd] (c893b6b9-623d-41fb-a880-71afe009e1d4)
Jun 10 13:38:35 survey celery[1774]: [2019-06-10 13:38:35,871: INFO/ForkPoolWorker-7097] Processing... Task [fd] (c893b6b9-623d-41fb-a880-71afe009e1d4)
Jun 10 13:38:39 survey celery[1774]: INFO Canceling Task [fd] (c893b6b9-623d-41fb-a880-71afe009e1d4)
Jun 10 13:38:39 survey celery[1774]: [2019-06-10 13:38:39,681: INFO/ForkPoolWorker-7097] Canceling Task [fd] (c893b6b9-623d-41fb-a880-71afe009e1d4)
Jun 10 13:39:00 survey celery[1774]: INFO Restarting Task [fd] (c893b6b9-623d-41fb-a880-71afe009e1d4)
Jun 10 13:39:00 survey celery[1774]: [2019-06-10 13:39:00,563: INFO/ForkPoolWorker-7097] Restarting Task [fd] (c893b6b9-623d-41fb-a880-71afe009e1d4)
Jun 10 13:39:48 survey celery[1774]: INFO Processing status: 30 for Task [fd] (c893b6b9-623d-41fb-a880-71afe009e1d4)
Jun 10 13:39:48 survey celery[1774]: [2019-06-10 13:39:48,551: INFO/ForkPoolWorker-7105] Processing status: 30 for Task [fd] (c893b6b9-623d-41fb-a880-71afe009e1d4)
Jun 10 13:39:56 survey celery[1774]: INFO Restarting Task [fd] (c893b6b9-623d-41fb-a880-71afe009e1d4)
Jun 10 13:39:56 survey celery[1774]: [2019-06-10 13:39:56,940: INFO/ForkPoolWorker-7106] Restarting Task [fd] (c893b6b9-623d-41fb-a880-71afe009e1d4)
Jun 10 13:40:19 survey celery[1774]: INFO Removing Task [fd] (c893b6b9-623d-41fb-a880-71afe009e1d4)
Jun 10 13:40:19 survey celery[1774]: [2019-06-10 13:40:19,310: INFO/ForkPoolWorker-7104] Removing Task [fd] (c893b6b9-623d-41fb-a880-71afe009e1d4)
Jun 10 13:40:19 survey celery[1774]: INFO Cleaning up OAM datastore for task c893b6b9-623d-41fb-a880-71afe009e1d4
Jun 10 13:40:19 survey celery[1774]: [2019-06-10 13:40:19,734: INFO/ForkPoolWorker-7104] Cleaning up OAM datastore for task c893b6b9-623d-41fb-a880-71afe009e1d4
lines 420-477/477 (END)

What's odd is that the original task seems to freeze without logs. I started it last night and there has been nothing new logged to it (at least including its ID) since it was started:

root@survey:~# journalctl -u webodm-celery.service | grep 88b35943-6adb-4af7-a1a9-ae68fb35f063
Jun 09 21:15:04 survey celery[1774]: INFO Automatically assigned processing node proc08.nwk1.com to Task [ECCO] (88b35943-6adb-4af7-a1a9-ae68fb35f063)
Jun 09 21:15:04 survey celery[1774]: [2019-06-09 21:15:04,307: INFO/ForkPoolWorker-1278] Automatically assigned processing node proc08.nwk1.com to Task [ECCO] (88b35943-6adb-4af7-a1a9-ae68fb35f063)
Jun 09 21:15:04 survey celery[1774]: INFO Processing... Task [ECCO] (88b35943-6adb-4af7-a1a9-ae68fb35f063)
Jun 09 21:15:04 survey celery[1774]: [2019-06-09 21:15:04,318: INFO/ForkPoolWorker-1278] Processing... Task [ECCO] (88b35943-6adb-4af7-a1a9-ae68fb35f063)
root@survey:~#

[fritzagr]

i am having this issue when canceling a task during upload to processing node. i am going to try and reproduce the problem on a clean install so the log is not cluttered with other running tasks

[fritzagr]

ok. i have found a pattern. the cancel only hangs when uploading to a processing node other than localhost. in my case it hangs when uploading to the lightning net or another server on the same network.

[fritzagr]

below is the only information in the webodm-celery log on the webODM server.

Jun 25 16:27:08 StineDM02 celery[15891]: INFO Processing... Task [Task of 2019-06-25T21:18:58.233Z] (c3f632de-c979-4b97-84e0-6a174fd7f772)
Jun 25 16:27:08 StineDM02 celery[15891]: [2019-06-25 16:27:08,191: INFO/ForkPoolWorker-19711] worker.tasks.process_task[63df9395-3e45-4658-bf89-4cd797f8048c]: Processing... Task [Task of 2019-06-25T21:18:58.233Z] (c3f632de-c979-4b97-84e0-6a174fd7f772)

[pierotofy]

Thanks @fritzagr. I will try to reproduce this and see if I can find a fix.

[pierotofy]

Can confirm, this is a bug.

[pierotofy]

This is fixed in the cameracalib branch. Will be available as an update as soon as it's merged into master.