Cvat: Timeout: Incomplete Dataset Export Download

[cyrilzakka]

My actions before raising this issue

• [x] Read/searched the docs
• [x] Searched past issues

So I've been trying to export one of my datasets for the past several days but in vain. Dumping annotations seems to be working fine, but exporting the dataset always results in a timeout halfway through the download. I've attempted this on two different networks and browsers with no success.

Expected Behaviour

Trying to export a dataset (4.1 GB) should download successfully, without any timeouts or errors.

Current Behaviour

Download timeouts after 2GB have downloaded and the resulting file is unusable.

Possible Solution

I've tried circumventing the timeout in multiple ways, which have all resulted in failure.
1 - I tried using wget in Colab to get around what I thought was slow internet, but the downloaded file is empty and of Length = 0.

!wget --no-check-certificate  --load-cookies cookies.txt \
'https://cvat.org/api/v1/tasks/###/dataset?format=TFRecord%201.0&action=download'

2- Update: This only worked once and now times out at 2GB as well.

I tried using Selenium on Colab and it was successful:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')

options.add_experimental_option("prefs", {
  "download.default_directory": r"/content/",
  "download.prompt_for_download": False,
  "download.directory_upgrade": True,
  "safebrowsing.enabled": True
})

wd = webdriver.Chrome('chromedriver',options=options)
wd.get("https://cvat.org/auth/login")

username = wd.find_element_by_id("username")
password = wd.find_element_by_id("password")

username.send_keys("username")
password.send_keys("password")
element = wd.find_element_by_css_selector('.login-form-button')
wd.execute_script("arguments[0].click();", element)

from selenium.webdriver import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

wait = WebDriverWait(wd, 10)
actions = ActionChains(wd)

open = wd.find_element_by_css_selector("#root > section > main > div > div:nth-child(3) > div > div:nth-child(1) > div:nth-child(4) > div:nth-child(1) > div > a")
wd.execute_script("arguments[0].click();", open)

action_button = wd.find_element_by_css_selector("#root > section > main > div > div > div.ant-row-flex.ant-row-flex-space-between.ant-row-flex-middle.cvat-task-top-bar > div:nth-child(2) > button")
actions.move_to_element(action_button).perform()

export_data = wd.find_element_by_css_selector("body > div:nth-child(9) > div > div > ul > li:nth-child(3) > div")
actions.move_to_element(export_data).perform()

data = wd.find_element_by_xpath("//*[@id='export_task_dataset$Menu']/li[1]")
wd.execute_script("arguments[0].click();", data)

Steps to Reproduce (for bugs)

1. Create Task
2. Annotate a few things
3. Export dataset
4. Download timeouts before successful download.

Context

I can't export my data for use in research.

Your Environment

CVAT.org

[azhavoro]

Hi, can you download the dataset with web UI manually?
Also please specify the task id.

[cyrilzakka]

I can't download with the web UI manually which is why I resorted to my workarounds. The manual download results in truncation after 2GB have been downloaded and results in a file named 'dataset' that's empty. The task ID is 17431

[azhavoro]

Thanks for the report!

[kevincolyar]

I can't download with the web UI manually which is why I resorted to my workarounds. The manual download results in truncation after 2GB have been downloaded and results in a file named 'dataset' that's empty. The task ID is 17431

I have the same issue when trying to export a dataset. Reports downloading a 4GB file via Chrome, but the resulting 'dataset' file is empty.

[rilut]

@kevincolyar yes I still have the same problem in 2021. did you manage to find the solutoin?

[rilut]

@azhavoro is there any temporary workaround to download dataset?

[nullket]

I have the exact same issue, but with 1 GB. It downloads until 1 GB and then fails leaving a incomplete 1 GB file. I thought this might be related to https://github.com/openvinotoolkit/cvat/issues/956 but I could not find any info in the logs.

Maybe there is a simpler workaround but you can mount the docker volume and copy created zip by hand (before cache cleaner deletes it again). The steps would by

1. Try to download the file (so the zip is created).
2. Check the cvat docker logs to figure out the naming of the zip file (something like The task 'abc' is exported as 'YOLO 1.1' at '/home/django/data/tasks/80/export_cache/dataset_cvat-for-images-11.ZIP' and available for downloading for the next 10:00:00.)'
3. On the docker host run a simple cp (adjust the filename according to step 2 and might use a smaller image instead of ubuntu like alpine (I had it already on the disk for another job)) docker run --rm --volumes-from cvat -v /tmp:/tmphost ubuntu cp /home/django/data/tasks/80/export_cache/dataset_cvat-for-images-11.ZIP /tmphost/dataset.zip
4. Find the data in the tmp folder of you host

[cyrilzakka]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 39 4200M   39 1667M    0     0  20.0M      0  0:03:29  0:01:23  0:02:06 20.5M
curl: (18) transfer closed with 2655909917 bytes remaining to read

Bug is still there. Even with blazing internet speed I'm only able to download 39-42%

[JobCollins]

I have also experienced this issue. I'm trying to export a 5.6GB zipped dataset and the download timeout is at 1GB always. Is there a workaround for this?

[azhavoro]

@JobCollins Hi, try to adjust --limit-request-body value https://github.com/openvinotoolkit/cvat/blob/develop/supervisord.conf#L65. In the near future we plan to remove any data size limitations by default.

[JobCollins]

@azhavoro thanks for this but my annotation job was done online (cvat.org). I resolved to use online after the local installation kept failing. Or could I sign in to my online profile via local? That would be great.

[JobCollins]

Thanks a lot, @azhavoro . Using your advice,

1.I downloaded the annotations online (which has no issues downloading) then
2.uploaded the annotations to the offline installation and job
3.changed the local supervisord.conf code line 65

And I got my full dataset exported. Quite a workaround!😅