Colabtools: Slow to load images from my google drive

Created on 3 Apr 2018 · 8Comments · Source: googlecolab/colabtools

I've recently tried using Colab for CNNs. I'm trying to use the files that I've saved in my google drive. But the loading time of the files are very high and at the end the notebook crashes.

Any help on this?

Thanks

Source

dhavalmj007

👍4

All 8 comments

How are you connecting to Google Drive?

pointyointment on 10 Apr 2018

I'm using this first to authenticate
!apt-get install -y -qq software-properties-common python-software-properties module-init-tools !add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null !apt-get update -qq 2>&1 > /dev/null !apt-get -y install -qq google-drive-ocamlfuse fuse from google.colab import auth auth.authenticate_user() from oauth2client.client import GoogleCredentials creds = GoogleCredentials.get_application_default() import getpass !google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL vcode = getpass.getpass() !echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}

then this to mount drive
!mkdir -p drive !google-drive-ocamlfuse drive

dhavalmj007 on 11 Apr 2018

I'd recommend trying the bundled Drive client like so:

from google.colab import drive
drive.mount('/content/gdrive')

If you encounter issues working with Drive files, please open a new issue describing what you tried and the precise error message you observed.

colaboratory-team on 20 Feb 2019

😕2 👎2

Hi There, I am trying to load my database via Google drive. I have 8,000 images but when I try to load my database in order to use like a entry data to CNNs. It is too slow, given an example, the same code in Laptop this function runs 1 minute aprox. but in colab more than one hour.
Do you know what is the problem about that?

The drive cliente is:

from google.colab import drive
drive.mount('/content/drive')

and the code is

def create_training_data():
    count = 0

    for category in CATEGORIES:
        print(category)
        path = os.path.join(DATADIR, category)
        class_num = CATEGORIES.index(category)
        flag = 0
        for img in os.listdir(path):
            try:

                img_array = cv2.imread(os.path.join(path,img))

                """
                #training_data.append(img_array)
                if flag == 0:
                  flag = 1
                  training_data = np.array(img_array)
                #label_data.append(class_num)
                else:
                  training_data = np.concatenate([[training_data],[img_array]])
                 """
                count +=1
                if count == 100:
                  count=0
                  print("cargue 100 imagenes")
            except Exception as e:
                pass



    xTrain, xTest, yTrain, yTest = train_test_split(training_data, label_data, test_size = 0.25, random_state = 0)
    xTrain = np.array(xTrain)
    xTest = np.array(xTest)
    yTrain = np.array(yTrain)
    yTest = np.array(yTest)
    return (xTrain, yTrain), (xTest, yTest)

LeonardoLed on 21 Jun 2019

👀16

Hi,
I'm having the save issue.
To load the data, Im doing

from google.colab import drive
drive.mount("/content/drive/")

and then iterating over the images in folder using glob and loading them using OpenCV. This is quite fast in my local system which has 7 years old i7 and 16gb ram. But on colab, the data loading is ridiculously slow.

Any suggestions around it?