Fastai: Update example for fastai v1 Google Colab GPU users to set num_workers=0

Created on 22 Oct 2018  路  4Comments  路  Source: fastai/fastai

Memory issues with Google Colab if don't set num_workers=0

To Reproduce
Make a Colab project with FastAI V1.

Expected behavior
The model should fit without error.

Instead it gives an error:
RuntimeError: DataLoader worker (pid xxx) is killed by signal: Bus error.

Additional context
This is caused by PyTorch v1 preview but messes up FastAI users using Colab with GPU following the cats-dogs example. https://github.com/fastai/fastai/blob/master/examples/dogs_cats.ipynb

Resolved by adding _num_workers=0_ in
data = ImageDataBunch.from_folder(path, ds_tfms=get_transforms(), tfms=imagenet_norm, size=224)

Suggest adding num_workers=0 and/or a note for Colab users.

Most helpful comment

@zzzhacker

data = ImageDataBunch.from_folder(
    path, 
    ds_tfms=get_transforms(), 
    tfms=imagenet_norm, 
    size=224,
    num_workers=0,
)

You may need to run it once first without num_workers = 0 and then set num_workers=0.

All 4 comments

We can't put a warning in each notebook because of this bug that should be resolved on the pytorch side at some point before v1 is officially out. Adding num_workers=0 would slow down the notebook for every user, which isn't what we want either.
So I don't see an ideal solution and think the best place to warn users would be a pinned thread on the forum.

@discdiver in which code line should I add this num_workers = 0 , I am having the same problem

@zzzhacker

data = ImageDataBunch.from_folder(
    path, 
    ds_tfms=get_transforms(), 
    tfms=imagenet_norm, 
    size=224,
    num_workers=0,
)

You may need to run it once first without num_workers = 0 and then set num_workers=0.

I'm closing this issue. Fastai_v1 doesn't support Colab as long as Colab doesn't support pytorch. It's not helpful to beginners to have to worry about those things so please continue exchanging tricks to make it work on one of the advanced forums.
Thanks!

Was this page helpful?
0 / 5 - 0 ratings