Colabtools: Colab doesn't see files in Google Drive

[phly95]

For example, if I open a file in Google drive, and I have a txt file in the same folder, it doesn't see the other files in that folder.

[ShumzZ]

Do you mean that you can open a file in Google drive with Colab, while cannot see other files in the same folder? What exactly do you mean by "see", do you mean the other files cannot be accessed or you cannot perform other actions on those files?
If you mount your google drive correctly, you should be able to see them by:
!ls whatEverNameYouUsedToMountGoogleDrive/"My Drive"/

Update: to mount Google Drive
from google.colab import drive
drive.mount('/content/whatEverNameYouUsedToMountGoogleDrive')

[soheekim911]

I'm facing the same issue again and again. I have mounted to gdrive using the package what ShumzZ told. But in the sub directory, only one of the many files in the directory shows up when I use print(os.listdir('./gdrive/path/to/directory/')). What is more strange is that this problem happens sometime and sometimes all the files in the directory shows up. What's wrong with it? I think this isn't a synchronization issue because It's been a while since I uploaded the file. What is happening?

[ShumzZ]

I'm facing the same issue again and again. I have mounted to gdrive using the package what ShumzZ told. But in the sub directory, only one of the many files in the directory shows up when I use print(os.listdir('./gdrive/path/to/directory/')). What is more strange is that this problem happens sometime and sometimes all the files in the directory shows up. What's wrong with it? I think this isn't a synchronization issue because It's been a while since I uploaded the file. What is happening?

Hi @soheekim911
I don't know if this helps, but for me, although I mounted the Google Drive with drive.mount('/content/gdrive'), all of my files were surprisingly under
'/content/gdrive/"My Drive"/' , as I was expecting '/content/gdrive/'.

In order to list all the files, you could run !ls /content/gdrive/'My Drive'/, or if you want the file names returned as a list, run print(os.listdir(r'/content/gdrive/My Drive/'). ( Note the r flag, since os.listdir('/content/gdrive/"My Drive"/') won't work)

I'm not sure if this 'My Drive' thing is just how my own google drive was initiated or is it always like that. Anyway, hope this helps with your problem.

[Gotman88]

I am having the exact same issue.
I had mounted my drive yesterday and had no problem running my Python script.
Wanted to run it again today only to find out that the directory, which is full of PDFs, is empty according to Colab, therefore I cannot access any of those files.

Did you find any fix in the end?

Thanks

[soheekim911]

Hi,
Each time I try to mount my drive, I got different results. Sometimes files
appeared, but sometimes not.
As a result, I couldn't find a solution.
Instead, I'm using GCP(google cloud platform) for more stable computing.
I'm sorry that I can't give you the answer you want.

On Fri, Nov 8, 2019 at 12:49 AM Gotman88 [e-mail] wrote:

I am having the exact same issue.
I had mounted my drive yesterday and had no problem running my Python
script.
Wanted to run it again today only to find out that the directory, which is
full of PDFs, is empty according to Colab, therefore I cannot access any of
those files.

Did you find any fix in the end?

Thanks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/googlecolab/colabtools/issues/339?email_source=notifications&email_token=AH7NE4VY5PHZ2I3P2C3MBS3QSQ2HFA5CNFSM4GDDP25KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDM3GKQ#issuecomment-551138090,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AH7NE4TLMC6UHAMWG2VNFULQSQ2HFANCNFSM4GDDP25A
.

[Gotman88]

No worries, I just found a workaround in mounting the drive to another path.
So I remount it to /content/drive2 instead of /content/drive which I used in the first place.
Weird bug, weird workaround, but so far it is working for me..!

[soheekim911]

Oh, it's weird! But I should try it next time I use colab.
Thanks!

On Fri, Nov 8, 2019 at 2:06 AM Gotman88 [e-mail] wrote:

No worries, I just found a workaround in mounting the drive to another
path.
So I remount it to /content/drive2 instead of /content/drive which I used
in the first place.
Weird bug, weird workaround, but so far it is working for me..!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/googlecolab/colabtools/issues/339?email_source=notifications&email_token=AH7NE4R76T4NKX66Z5FGGDDQSRDKVA5CNFSM4GDDP25KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDNDTVQ#issuecomment-551172566,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AH7NE4U5EWW2WWJTFN2RFQDQSRDKVANCNFSM4GDDP25A
.

[cinjon]

This is happening for me too.

[jcwright77]

Same issue. Can see the mounted drive and files in the Files panel on the left but cannot open either by clicking on it or loading in a cell. It's like the notebook does not actually have permissions. Tried factory reset, remounting, etc. Eventually copied to the content to a new notebook which worked for a while then the problem returned.

[Rian-II]

I am experiencing the same issue. The difference for me though is that the file explorer on the left is showing the same files as returned by print(os.listdir(r'/content/gdrive/My Drive/'). But there are definitely files missing. No problem yesterday, but suddenly a problem today.
What is really strange is that the folder only contains CSV files and they were all uploaded on exactly the same date. But only some of them are appearing.
The workaround to remount to /content/drive2 as suggested by @Gotman88 is not working for me.

[Rian-II]

Terminating the active session and reloading the tab eventually solved this. This was only temporarily because as soon as I imported the first file, the issue returned and I can now only see a small subset of the files in that folder.

I tried making a copy of the files in the same folder and they then suddenly appeared. After renaming them to their original names, they were still displaying. So the best workaround for me so far is to copy your files, delete the old versions and rename the new files to the previous name. If this workaround continues to work, I'll automate this with:

# Copy folder to new
!cp -av "/content/drive/my_folder_path/my_folder" "/content/drive/my_folder_path/my_folder_new"
# Remove original folder, subfolder and files
!rm -rf "/content/drive/my_folder_path/my_folder"
# Rename new folder to old name
!mv "/content/drive/my_folder_path/my_folder_new" "/content/drive/my_folder_path/my_folder"

[wanfuse123]

I have found the reason why one cant mount ones own google drive for these things is because of a race condition with google . First it was suggested that changing the mount location from /content/gdrive to /content/something else but this didnt fix it. What I ended up doing was copying manually the files that are copied to google drive, then installing the google drive desktop application I would then in windows 10 go to the folder which is now located on google drive and disable file permissions inheritance and then manually putting full control rights on the folder to the users group and to authenticated users group. This seems to have fixed this for me. Other times I have noticed with these colabs (not this one in particular but some of the components used like the trained models are missing from the repository (as if they had been removed) Only solution for this is to look around for other sources of these files. This includes scurrying through google search engine and also looking at the git checkout level to find branches besides master and also looking for projects that cloned the project on github to see if they still include the files. Hope this helps!

[juniorjasin]

I still have this issue, if I opened colab, mount drive and find some files they are there, but the next time I do this, some of them disappear and I cannot copied to the current colab instance.

[ealgoo]

Same problem. Here.
Mounted gDrive and then accessed some folders and files without problem.
However, after some time, I don't see files in certain folder. (I also see the missing folder/files in the left file exploration panel).

Remounting under different name helped and served me a as a work around. However, I would like to understand how to solve this issue fundamentally.

[wanfuse123]

This works!

https://youtu.be/PoETnY1ju20

On Thu, Sep 10, 2020, 9:11 PM Ealgoo [e-mail] wrote:

Same problem. Here.
Mounted gDrive and then accessed some folders and files without problem.
However, after some time, I don't see files in certain folder. (I also see
the missing folder/files in the left file exploration panel).

Remounting under different name helped and served me a as a work around.
However, I would like to understand how to solve this issue fundamentally.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/googlecolab/colabtools/issues/339#issuecomment-690813622,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADDBZWIR3JTUJTBHYDN22CTSFF2N5ANCNFSM4GDDP25A
.

[d-diaz]

I have also seen this happen. After mounting my drive using drive.mount('/content/drive', force_remount=True) I can use the file explorer view on the left of the colab window and browse into a few folders and see the files and folders I want to use. After executing a few cells to pip install additional packages, I click refresh on the file browser view on the left of the colab window and some of the files and folders that were there before are now gone, and the colab VM cannot see them anymore using ls or os.path.exists, for example. These files and folders have existed on my Google Drive for several weeks (if not months), and I can still see all of them when I go through the web browser to Google Drive. @wanfuse123, the problem is masking GB of data in hundreds-to-thousands of files, so I am not eager for a workaround that involves downloading and copying lots of data my local machine and resyncing to drive.

[juniorjasin]

For me, drive.mount('/content/drive', force_remount=True) don't work, I found it in other github threat (that I cannot find) that if the directory you want to access has big files, in my case the directory has ~15gb in files I couldn't copy the files, but if I move the file I want to copy to another directory without anything else I can copy it.

That work for me.

[wanfuse123]

You may find that just creating the folder in google sync drive and setting
permissions, followed by empty folder sync (remembering to put permission
on folder and have it apply to all files and folders beneith) might be
enough to use your g drive normally from the colab

Regards,

Steven

On Fri, Sep 11, 2020, 1:54 PM David Diaz [e-mail] wrote:

I have also seen this happen. After mounting my drive using drive.mount('/content/drive',
force_remount=True) I can use the file explorer view on the left of the
colab window and browse into a few folders and see the files and folders I
want to use. After executing a few cells to pip install additional
packages, I click refresh on the file browser view on the left of the colab
window and some of the files and folders that were there before are now
gone, and the colab VM cannot see them anymore using ls or os.path.exists,
for example. These files and folders have existed on my Google Drive for
several weeks (if not months), and I can still see all of them when I go
through the web browser to Google Drive. @wanfuse123
https://github.com/wanfuse123, the problem is masking GB of data in
hundreds-to-thousands of files, so I am not eager for a workaround that
involves downloading and copying lots of data my local machine and
resyncing to drive.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/googlecolab/colabtools/issues/339#issuecomment-691232656,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADDBZWMNX52ACWW3ONUIDQTSFJP55ANCNFSM4GDDP25A
.

[victorpmacc]

The solution that worked for me was: click on top-right corner where you have the disk and RAM usage. Force the shut down on the session mounted. Reload the colab page. Mount again.

Hope it helps anyone out there!