Webodm: Periodical 30s+ tile serving freezes

[LinFor]

What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/)

WebODM running via docker on Ubuntu 2004 host.
Client is Chrome 84 on Windows 10

What is the problem?

Periodical 30s+ tile serving freezes while browsing ortophoto, dsm, dtm (possible any tiles) through "View map" button (/map/project/1/task/xxx/).
On each freeze log shows some rasterio exceptions:

/usr/local/lib/python3.8/dist-packages/rasterio/io.py:133: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  return writer(mempath, 'w+', driver=driver, width=width,
[2020-08-05 15:32:01 +0000] [75] [INFO] Autorestarting worker after current request.
[2020-08-05 15:32:01 +0000] [75] [INFO] Worker exiting (pid: 75)
[2020-08-05 15:32:01 +0000] [76] [INFO] Booting worker with pid: 76
INFO Initializing GRASS engine using /usr/bin/grass78
/usr/local/lib/python3.8/dist-packages/rasterio/io.py:133: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  return writer(mempath, 'w+', driver=driver, width=width,
[2020-08-05 15:32:31 +0000] [76] [INFO] Autorestarting worker after current request.
[2020-08-05 15:32:31 +0000] [76] [INFO] Worker exiting (pid: 76)
[2020-08-05 15:32:31 +0000] [77] [INFO] Booting worker with pid: 77
INFO Initializing GRASS engine using /usr/bin/grass78
/usr/local/lib/python3.8/dist-packages/rasterio/io.py:133: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  return writer(mempath, 'w+', driver=driver, width=width,
[2020-08-05 15:33:03 +0000] [77] [INFO] Autorestarting worker after current request.
[2020-08-05 15:33:03 +0000] [77] [INFO] Worker exiting (pid: 77)
[2020-08-05 15:33:03 +0000] [78] [INFO] Booting worker with pid: 78
INFO Initializing GRASS engine using /usr/bin/grass78
/usr/local/lib/python3.8/dist-packages/rasterio/io.py:133: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  return writer(mempath, 'w+', driver=driver, width=width,
[2020-08-05 15:33:31 +0000] [78] [INFO] Autorestarting worker after current request.
[2020-08-05 15:33:31 +0000] [78] [INFO] Worker exiting (pid: 78)
[2020-08-05 15:33:32 +0000] [79] [INFO] Booting worker with pid: 79
INFO Initializing GRASS engine using /usr/bin/grass78
/usr/local/lib/python3.8/dist-packages/rasterio/io.py:133: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  return writer(mempath, 'w+', driver=driver, width=width,
WARNING Not Found: /api/projects/1/tasks/48a75b94-1dd2-40d9-8ede-3aebbd5eafcd/orthophoto/tiles/2/2/1.png
[2020-08-05 15:34:08 +0000] [79] [INFO] Autorestarting worker after current request.
[2020-08-05 15:34:08 +0000] [79] [INFO] Worker exiting (pid: 79)
[2020-08-05 15:34:09 +0000] [80] [INFO] Booting worker with pid: 80
INFO Initializing GRASS engine using /usr/bin/grass78
/usr/local/lib/python3.8/dist-packages/rasterio/io.py:133: NotGeoreferencedWarning: Dataset has no geotransform set. The identity matrix may be returned.
  return writer(mempath, 'w+', driver=driver, width=width,

What should be the expected behavior?

No freezes, continous tile loading.

[pierotofy]

Could you share the all.zip archive (from Download Assets --> All Assets (.zip) ) for the task in question by uploading it somewhere (Dropbox, Google Drive, etc.)?

Did this happen with older versions or is it a newer problem?

[LinFor]

Could you share the all.zip archive (from Download Assets --> All Assets (.zip) ) for the task in question by uploading it somewhere (Dropbox, Google Drive, etc.)?
Yes: https://drive.google.com/file/d/1RNjaTDUHkHF2lobtSoiY3jz1IzIL_zE1/view?usp=sharing
But I see this freezes on all my datasets.

Did this happen with older versions or is it a newer problem?
I have been using WebODM for about three months, freezes are observed all the time.

[pierotofy]

I'll look into this in the next few days and update this issue with my findings.

[pierotofy]

I can't seem to find issues; it is a medium size orthophoto, so the dynamic tiler has to do some work. I can see that if the program is running on a low powered computer or one with a slow disk this could cause the tiler to hang due to too many requests not being completed quickly enough.

What are the specs of the machine (hard disk in particular)?

[LinFor]

I use consumer-grade machine (i3-3225 cpu, 16gb ram) with two spindle disks in striped (raid0) configuration. Possible it's a root cause due to small iops. But browsing odm-generated tiles through tileserver-gl works well without freezes.
Maybe possible use something like mbtiles format as storage on disk to take advantage of prefetching and read-ahead caching on low speed disks?

[pierotofy]

Mm, tileserver-gl works differently, so not exactly a fair comparison. Would be interesting to compare performance of https://github.com/DHI-GRAS/terracotta though.

There's definitely opportunities to speed up the dynamic tiler though. I have noticed some slow downs myself.

[pierotofy]

This might be solved with #978

Please update and see if the error persists? :pray: