Colabtools: Colab permanently hangs (even between sessions) when executing a very large file operation

[cduguet]

Colab is permanently frozen after I tried to unzip a 10gb 7z file.
When I tried to unzip it, the progress was so slow that I wanted to cancel it. I tried:

• Restarting Runtime
• Changing Runtime Type (GPU, NO-GPU)
• Creating an entirely new Notebook
  to no avail.

Steps to reproduce:

• Download all files from https://drive.google.com/drive/folders/0B7EVK8r0v71peklHb0pGdDl6R28 to the notebook.
• Unify all the 14 parts by using !7zr x img_celeba.7z.001 -tsplit
• Unzip them using !p7zip -d img_celeba.7z
• Delete the working cell
• Enjoy breaking Google!

[colaboratory-team]

Sorry for the trouble. I suspect the inflated files are exhausting the available disk capacity, which can cause the stuck behavior you observe. (Of course, improving the quality of error messages and recovering from this situation is an ongoing focus.)

An immediate work-around is to use a local backend, or to continue with a smaller subset of the overall data set.

[cduguet]

... or to continue with a smaller subset of the overall data set.

Does this mean that there is currently a way to recover or solution is on development? If so I'd love to know when it's available, so I am able to start using colab again.

Thank you!

[colaboratory-team]

Backends are ephemeral and discarded after a period of inactivity, so you should be fine in a few hours.

An immediate work-around is to change the hardware accelerator runtime type in notebook settings (Runtime menu -> Change runtime type). This will change to a fresh, distinct backend that should permit you to connect immediately.

(As with disk fullness, we are working on making it simpler to reset stuck VMs without waiting.)

[CumaliTurkmenoglu]

Hi, I have a similar problem. Run time died! pup-ups and the kernel restarts(all variables lost) while reading train.csv (~5 GB) from Gdrive. No problem with test.csv which is relatively too small.

train_downloaded=drive.CreateFile({'id':'1rJjSqP-c2uptNbtmRuf2m6cMT3uvV3Pp'})
train_downloaded.GetContentFile('train.csv')
test_downloaded=drive.CreateFile({'id':'1Bwg1CEfE3bPEiNnpo_M6j6rms7Im2Mav'})
test_downloaded.GetContentFile('test.csv')

df_test=pd.read_csv('test.csv')
df_train=pd.read_csv('train.csv')

[colaboratory-team]

To reset your backend, select the command 'Reset all runtimes...' from the Runtime menu.

[MuhammadUmarFarooq786]

Active community attracts me to use TensorFlow.
Thumbs up!

[Masum06]

Anyone found any solution? Colab restarts runtime when I try to open a 4GB text file.

[colaboratory-team]

Anyone found any solution? Colab restarts runtime when I try to open a 4GB text file.

Please open a distinct issue describing how you are uploading the file and we'll follow up there. This is likely a distinct issue from the disk fullness issue discussed here.

[PatrickRuan]

Any one had any solution for this speed issue?

!kaggle competitions download -c dogs-vs-cats -p ./drive/ColabOK/data
os.chdir(DATA_PATH) #change dir
!mkdir train
!unzip -q train.zip -d train/ #unzip data in train/

I encountered an issue, “speed is very slow” during unzip.
I unziped train.zip data from kaggle dogs-vs-cats. (543MB)
there are 25000 images at train.zip, size is around 15KB ~ 40KB each.
the unzip speed is around 10 image/min. :’(

any idea? thanks in advanced…

[colaboratory-team]

@PatrickRuan your issue (I/O speed) is unrelated to this issue (disk exhaustion). Please open new issues for new problems.
Your issue is likely that you're unzipping a large number of small files into a Google Drive-mounted directory. Don't do that - unzip to the VM's local disk instead.
https://research.google.com/colaboratory/faq.html#drive-timeout describes related issues that will crop up intermittently if you succeed in unzipping 25k files into Google Drive and then try to use that directory.