Gdrive: Download of large folder stops with 403 rate limit exceeded

[jstray]

After about 1 minute of downloading, and a few dozen files, gdrive download stopped with the following error:

Failed to download file: googleapi: got HTTP response code 403 with body: {
 "error": {
  "errors": [
   {
    "domain": "usageLimits",
    "reason": "rateLimitExceeded",
    "message": "Rate Limit Exceeded"
   }
  ],
  "code": 403,
  "message": "Rate Limit Exceeded"
 }
}

It may have been going quickly through a number of small files when this happened.
What is the most reliable way to pull down a large folder with gdrive?

[wfeiereisen]

I have also had this same problem. From the google documentation, they have apparently set a daily limit. If directly using their API, they have suggestions about how to fix it, but I do not know how to fix it through gdrive. Suggestions? My problem would be solved if there were a gdrive option to zip and then download. Feature request?

[cnrting]

problem +1 when try to upload a directory containing a lot of files,500+ i suppose.

[wfeiereisen]

Yes, up to maybe 3000 small (400kb) files at a time. This is output from a Raspberry Pi camera that is watching a birds nest. Each file is pushed individually to Google Drive during the day. The mass download of each day's files happens through:

_system("/usr/local/bin/gdrive download query -f \"name contains '$query_name' and name contains 'jpg'\";");_

I'd let Google zip them and send one file if I could, but I don't know how.

[jstray]

I eventually "solved" this by moving to rclone. Using the -v option to print additional debug reveals that rclone implements exponential backoff when it encounters rate limits, and this enables it to complete successfully.

[martinbrook]

Affected here also with upload of hundreds of smallish 1-10MB files,

[martinbrook]

Looks like sync up/down has code to handle the rate limit.

Just needs adding to non-sync up/download?

https://github.com/prasmussen/gdrive/commit/bdd7877be9c503e7968c221f3396d239edb267d2

[7yl4r]

duplicate of #132 .

[k0d3r3d]

+1 uploading lots of files I got a ratelimit error , I am using powershell to trigger gdrive, so upon the error i told it to wait for a bit then retry the same file transfer, in my preliminary testing seems to work around the error.

[bogdan-calapod]

+1 - Same problem here

[euklid]

Hi there, I only implemented a fix for the download and download query sub commands, see PR #309.

I think you can use the same approach for the other sub commands.
TBH, I'm only using this tool to download a large set of files from a public Google drive and thus I had not looked into the sync/upload and other features. Feel free to rip off my code to use it for fixes in the other parts that need exponential back off

[JeffLabonte]

I have had the same problem here as well. The way I dealt with it is just to exec gdrive once again until it makes it. I was thinking about creating a bash script for now to handle the 403 error to reinitiate the download.

[yangl57]

Had same issue. Exponential back off wasn't a solution for my case. Moved to 'rclone' which works.