Rclone: Any plan to add support to Google Photos?

[lssong99]

If possible, please add a support to upload photo/video files to Google Photos directly!

Although it's possible to add a "Google Photos" folder in Google Drive, and all your Google Photos will be there (organized by date folder), However, photos uploaded into this folder does not seems to reflect into Google Photos.

Also, if we upload in "High Quality" than there will be unlimited storage size for photos and Video. I am not sure the "down-sizing" is done locally or remotely by Google Photos server, however...

I realize Google Photo is not a good place to organize photos but it's a good place to share photos with others. And with a stock of 300k+ photos I really don't want to have my PC running for God-knows-how-long for the upload.... It's the job of RPi!

[ncw]

Thanks for the suggestion.

There seems to be a comprehensive API here

https://developers.google.com/picasa-web/docs/2.0/developers_guide_protocol

[lssong99]

Yes, but I believe Picasa and it's API will be depreciated from May/1...

[ncw]

Foo! Is there a different API for google photos that you can find?

[buchno]

The Picasa API will continue to support uploading images:

Beginning May 1st, 2016, we’ll start rolling out changes to the Picasa Web Albums Data API and no longer support the following functionality:

• Flash support
• Community search
• Mutation operations other than uploads
• All support for tags, comments, and contacts

The API will still support other functions, including reading photos, reading albums, reading photos in albums, and uploading new photos. Although these operations will continue to be supported and the protocol will remain the same, the content included in the responses and the operation behavior may change.

https://developers.google.com/picasa-web/

Edit: Judging by this answer, it will however count towards the storage quota.

[lssong99]

Even if it count towards the storage quota, it is still possible to use "reclaim space" in the setting of Photos.google.com which will re-compress (or just convert?) it into non-quota related "high resolution" photos.

Thus before Google comes up with new Photos API (God knows when...), do you think it's the best solution we could get so far?

[kiddick]

+1

may be interesting wavemaking/GooglePhotosSync: Bi-directional sync between local folder and Google Photos/Picasa

[calisro]

I've been trying to accomplish this for a long time. I have a huge local store of photos that I want to store in google photos at 'high resolution' as to have them NOT count towards the quotas but instead only use it to be able to see these photos across google devices. If we upload to drive, we get hit with quotas. It would be absolutely awesome to be able to sync them with rclone at a 'under the quota' limit to G-Photos.

[lv913]

+1

Another useful functionality could be to arrange photos/videos in albums, i.e. using the folder name for that.
Nick, do you think that it would be possible to have Google Photos in roadmap soon or later?

Thanks!

[BenoitDuffez]

I was able to sync photos using this option from Google Drive:

This created a folder called Google Photos which I could sync using:

rclone --exclude '*.{dng,CR2}' --include '{IMG,PANO,VID}_*.{jpg,mp4}' sync 'MyRemoteName:/Google Photos/2016' /mnt/raid0/GooglePhotos/2016/

This examples removes the RAW images because they come from my DSLR, which is already backed up. I'm only interested in the pics uploaded from mobile.

[ncw]

@BenoitDuffez nice writeup thanks - a great way of downloading your pics from google photos.

It doesn't solve the upload part though which is what this issue is about.

[kiddick]

@ncw that's strange that Google still doesn't have official API for that..

[leocrawford]

I'm the author of picasawebsync and believe that the picasaweb API is the only one that currently works for reasons that baffle me. However the binding libraries don't support OAUTH2 which is the only one supported by the REST API, so you have to make a manual workaround.

A quick look though seems to imply that google now does offer a two way sync between drive and photos (https://support.google.com/photos/answer/6156103?co=GENIE.Platform%3DAndroid&hl=en-GB) though I can't readily test it as I'm a google apps user

[TimJWatts]

2 comments based on tests I just did:

a) You can upload to Google Drive and have them displayed in Google Photos if you turn the setting on as someone may have said above. With the exception that Panasonic RW2 raw files render in GDrive but do not show in GPhotos.

b) GDrive is quite a nice medium to display and share photos. But does not have all the polish of GPhotos. And does not have a free option for JPEGs (won't affect me - I want to archive my RW2s and processed TIFFs/DNGs).

But it would be nice to have rclone support if it can be figured out :)

[TioBorracho]

Sadly the reclaim space feature wouldn't work since the photos are uploaded via google drive according to this

[gbvaz]

Hi, I know this is not a new topic, but is there any solution to the problem @TioBorracho was talking about? I'm facing the same situation, as related on Google Photos support forum topic.

Any help is appreciated!

[Webreaper]

Just seen this thread. You may be interested in the little app I put together to solve this:
https://github.com/Webreaper/GooglePhotoSync/
It was primarily designed for running on a Mac desktop (it sits in the menu bar, like the Google Drive app, and just uploads/downloads photos to GPhotos. But I also introduced a 'headless' mode, so you can run it on a box without a GUI. I have it running on my Synology NAS, and used it for a few months to upload pictures to the cloud.
The app currently uploads at full res but once I remember the setting I plan to make it an option to upload at the 16MP setting (which is free and doesn't count against your quota) - the idea being that I'll use my unlimited Amazon Cloud storage for proper backup (via rclone) and then have a low-res copy in GPhotos to take advantage of their advanced image search (at least until Amazon bring their intelligent search, aka Family Vault, to the UK).

You could also run leocrawford's excellent sync tool, which would work too.

As has been discussed, the 'two-way' sync between drive and GPhotos is massively limited:

• Despite uploading all photos into named, organised folders (albums) into GPhotos, if you make them available through GDrive they just appear in month/year folders, which is really annoying.
• As has been mentioned, uploading to GDrive and then making the pictures available in GPhotos will not upload them as 'free' resolution. However, you could periodically click the 'Recover storage' option after doing this, and it'll resize them for you. I did this earlier this week (having moved all my full-res images to Amazon Drive) and it resized 1.2TB of images (around 200k x 5MB-9MB images) in about 36 hours.

Perhaps a neat solution would be to see if there's an API for 'Recover Storage' - you could then run rclone to upload to Gdrive, and then automatically trigger a 'recover storage' process once it's uploaded. :D

[lv913]

Hi Mark Otway,

is there any chance to run GooglePhotoSync on a standard Linux-ARM box?

Thanks,

Livio

2017-02-01 10:23 GMT+01:00 Mark Otway [e-mail]:

Just seen this thread. You may be interested in the little app I put
together to solve this:
https://github.com/Webreaper/GooglePhotoSync/
It was primarily designed for running on a Mac desktop (it sits in the
menu bar, like the Google Drive app, and just uploads/downloads photos to
GPhotos. But I also introduced a 'headless' mode, so you can run it on a
box without a GUI. I have it running on my Synology NAS, and used it for a
few months to upload pictures to the cloud.
The app currently uploads at full res but once I remember the setting I
plan to make it an option to upload at the 16MP setting (which is free and
doesn't count against your quota) - the idea being that I'll use my
unlimited Amazon Cloud storage for proper backup (via rclone) and then have
a low-res copy in GPhotos to take advantage of their advanced image search
(at least until Amazon bring their intelligent search, aka Family Vault, to
the UK).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-276609870, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARz94N-SvIiWwQRVT2ZDgeWQ17XeVl5wks5rYE76gaJpZM4Hh4qq
.

[Webreaper]

Don't see why not, it's just vanilla Java. I've run it on a Synology box
which is basically Linux. Unfortunately though, it won't work any more
because Google recently killed the API access for creating albums, and
haven't replaced it. You just get a 500 error now. It's caused a great deal
of consternation on the picasaweb API developer forum....

On Tue, 28 Feb 2017, 18:29 lv913, [e-mail] wrote:

Hi Mark Otway,

is there any chance to run GooglePhotoSync on a standard Linux-ARM box?

Thanks,

Livio

2017-02-01 10:23 GMT+01:00 Mark Otway [e-mail]:

Just seen this thread. You may be interested in the little app I put
together to solve this:
https://github.com/Webreaper/GooglePhotoSync/
It was primarily designed for running on a Mac desktop (it sits in the
menu bar, like the Google Drive app, and just uploads/downloads photos to
GPhotos. But I also introduced a 'headless' mode, so you can run it on a
box without a GUI. I have it running on my Synology NAS, and used it for
a
few months to upload pictures to the cloud.
The app currently uploads at full res but once I remember the setting I
plan to make it an option to upload at the 16MP setting (which is free
and
doesn't count against your quota) - the idea being that I'll use my
unlimited Amazon Cloud storage for proper backup (via rclone) and then
have
a low-res copy in GPhotos to take advantage of their advanced image
search
(at least until Amazon bring their intelligent search, aka Family Vault,
to
the UK).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-276609870, or
mute
the thread
<
https://github.com/notifications/unsubscribe-auth/ARz94N-SvIiWwQRVT2ZDgeWQ17XeVl5wks5rYE76gaJpZM4Hh4qq

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-283122905, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AErI7uyd9psN9E56t7MBVrQZRxfyvmI1ks5rhGdlgaJpZM4Hh4qq
.

[gerardo-junior]

Any prediction for this feature?

[dzg]

I would also love to be able to rclone to/from Google Photos.

Many items which are in my Google Photos are not showing up or syncing via the Google Photos folder in my Google Drive as they're supposed to.

The "Create a Google Photos folder" option in Drive is apparently unreliable; it does not show all items from Google Photos.

[Webreaper]

It's a bit pointless requesting this feature when Google have shut down the API that allows you to write to GPhotos. Also, I can tell you from my experience of writing a client back when the API was still available, it's in no way rich enough or functional enough to be able to implement with rclone.

If you're putting photos in Gdrive and they're not showing up in GPhotos, you should raise a support ticket with Google. I have a quarter of a million photos in Gdrive and they all show up just fine in Photos.

[dzg]

@Webreaper Actually I'm more concerned with GP > GD ... My phone auto uploads all items to GP, then I use GD to sync them to my desktop. I noticed today that many items in my GP are not showing up in my GD.

[Webreaper]

Ah, yes, we'll, given the lack of support for GPhotos, that doesn't surprise me at all. I had all sorts of problems with GPhotos not backing up photos, and so on. In the end I've moved to an entirely Gdrive based workflow.

1. Turn off Google photos backup
2. Install this: https://play.google.com/store/apps/details?id=com.ttxapps.drivesync
3. Configure to sync to a particular folder when any new pics are taken
4. Use rclone to sync from Gdrive back down to my pc.
   Works a treat, and a billion times more reliable (and faster too) than using Google's backup and autosync.

[lv913]

What about Google space usage? I suppose that uploading pictures through
Gdrive will not be free as it is using Gphotos native backup... am I wrong?

Il 24/Dic/2017 20:57, "Mark Otway" [e-mail] ha scritto:

Ah, yes, we'll, given the lack of support for GPhotos, that doesn't
surprise me at all. I had all sorts of problems with GPhotos not backing up
photos, and so on. In the end I've moved to an entirely Gdrive based
workflow.

1. Turn off Google photos backup
2. Install this: https://play.google.com/store/
   apps/details?id=com.ttxapps.drivesync
   https://play.google.com/store/apps/details?id=com.ttxapps.drivesync
3. Configure to sync to a particular folder when any new pics are taken
4. Use rclone to sync from Gdrive back down to my pc.
   Works a treat, and a billion times more reliable (and faster too) than
   using Google's backup and autosync.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-353800827, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ARz94Fl3yXWEAEj6uw17MXxnx6Jm7a4Yks5tDqyWgaJpZM4Hh4qq
.

[Webreaper]

If you upload photos to either GDrive or GPhotos and they're less than 16Mp, then they don't count against your storage tarifff. However, I'm using this as a primary offsite backup, so need photos stored in full ('original', as Google calls it) resolution, so yes, it's not free.

[dzg]

@Webreaper -- That sync app looks cool; but in the past I've always had issues with 3rd party syncers, mainly their not running until I launch them manually. Does this reliably upload in the background immediately upon file creation? THanks

[Webreaper]

I've been using it for about a year, and it's worked totally reliably. On my wife's phone I had to enable the constant notification for the instant upload to work, but that's not a big deal.

[danoc]

Many items which are in my Google Photos are not showing up or syncing via the Google Photos folder in my Google Drive as they're supposed to.

The "Create a Google Photos folder" option in Drive is apparently unreliable; it does not show all items from Google Photos.

@Webreaper – I'm having this same issue and it's driving me nuts. Have you made an progress on debugging? The checkbox seems to have a mind of its own.

If I can fix that functionality, I'll be able to use rclone to backup my Google Photos. Until then, I'm stuck with running Takeout every one in a while. 🙄

[Webreaper]

As I said earlier in the thread, Do it the other way around. Push the photos into drive, and use the "show drive photos in Google photos" option.

[dzg]

@Webreaper ... but if you "Show Google Drive photos & videos in your Photos library" won't it show ALL photos/videos from ALL folders in Drive? I really don't want that; I just want my CAMERA photos in Photos. Not all the junk from my millions of other folders.

[Webreaper]

Then you're out of luck....

[ncw]

I use rclone to backup google photos via drive. I find it works very well, but it is necessary to run rclone dedupe on it every now and again as it often seems to have duplicate file names in.

@Webreaper wrote:

It's a bit pointless requesting this feature when Google have shut down the API that allows you to write to GPhotos. Also, I can tell you from my experience of writing a client back when the API was still available, it's in no way rich enough or functional enough to be able to implement with rclone.

Do you think I should close this issue then now that the Google photos API is gone? When did that go - I seemed to have missed that?

[neocryptek]

The new Backup and Sync client from Google (which replaces the old Drive-only and Photos-only backup clients) interacts with Google Photos via a new Drive API in a very similar way to how "Computers" are backed up ("descendantOfRoot": false so it only shows under Computers in the Drive web UI and not under the normal Drive folder hierarchy) see #1773. This issue should probably be repurposed to implement that new Drive-based API (since it appears to only be some extra flags on existing Drive API calls to control send to Photos ("alwaysShowInPhotos": true) and original vs high-quality) instead of the old and dead picasa web albums API. Not sure if Google has documented the Drive-based Photos API anywhere yet though.

[danoc]

@neocryptek – Any more info on this part?

The new Backup and Sync client from Google (which replaces the old Drive-only and Photos-only backup clients) interacts with Google Photos via a new Drive API in a very similar way to how "Computers" are backed up ("descendantOfRoot": false so it only shows under Computers in the Drive web UI and not under the normal Drive folder hierarchy)

Not sure if I understand how Google Photos relates to descendantOfRoot.

[ncw]

@neocryptek interesting... If you come across any docs can you link them?

[ghost]

Good news on the horizon - TechCrunch

[neocryptek]

Direct link to API reference:
https://developers.google.com/photos/library/reference/

[amatiasq]

I'm using that API and I'm getting two URLs per picture:

• productUrl: Contains a private link to the working Google Photos application which loads the photo in full resolution (which is publicly accessible by the way)
• baseUrl: Contains a public link to a thumbnail ~500x300

I haven't found a way to get the full resolution picture URL from the API. If someone finds it please notify it.

EDIT: Nevermind, you have to read the file metadata and append the required resolution to the baseUrl:

const { baseUrl, mediaMetadata } = media;
const { width, height } = mediaMetadata;
return `${baseUrl}=w${width}-h${height}`;

[prateeksriv]

Hi @amatiasq ,
Are you saying that you already implemented this new remote?

[amatiasq]

@prateeksriv No, I was creating a standalone sync program from Google Photos API to Amazon S3 when I found this project. I don't know how to create a remote for it.

[lssong99]

After checking through the Google Photos API, I propose following ideas for rclone to fully support two way sync Google Photos:

A: from local upload to GP

1. It put all files in Year/Month directory structures
2. It can (optionally) upload full res or free size
3. (Optionally) Create Album based on local directory structure...

B: from GP download to local

1. If Album information is presented, then add photo downloaded into local folder named after Album
2. If Album is not available, then just maintain GP structure (Year/Month)

How do you guys think?

[Webreaper]

That's exactly how I did it in my sync utility (mentioned earlier in the thread). Would be like a very simple 1-directory-deep file sync.

[ebridges]

@lssong99 I really like the approach you've sketched out. My only suggestion would be to make the formatting of the directory & album names minimally customizable: e.g. yyyy/MM, yyyy-MM-dd, etc.

Also, AFAIK GP does not allow for nested folders, so the albums named Year/Month would not actually be a subfolder. Perhaps that could be glossed over from local -> remote / remote -> local, but should be kept in mind.

[leocrawford]

For the mapping, you need to do it in a ay that remains distinct otherwise to photos on one side might resolve to the same name on the other side. My python code to do this is available here if needed:

https://github.com/leocrawford/picasawebsync

[prateeksriv]

Looks like Google is coming up with a proper API interface:
https://developers.google.com/photos/library/guides/get-started

[philips]

The Go library is here: http://godoc.org/google.golang.org/api/photoslibrary/v1

You can find a couple of example apps: https://godoc.org/google.golang.org/api/photoslibrary/v1?importers

Probably the best plan is to use the id field as the filename. As mentioned in the previous content (https://github.com/ncw/rclone/issues/369#issuecomment-391292753) you essentially request baseURL with the height and width in the metadata field.

{
  "mediaItems": [
    {
      "id": "super_long_string",
      "productUrl": "https://photos.google.com/lr/photo/super_long_string",
      "baseUrl": "https://lh3.googleusercontent.com/lr/private_unauthenticated_long_string",
      "mimeType": "image/jpeg",
      "mediaMetadata": {
        "creationTime": "2018-07-13T23:17:56Z",
        "width": "3036",
        "height": "4048",
        "photo": {
          "cameraMake": "Google",
          "cameraModel": "Pixel",
          "focalLength": 4.67,
          "apertureFNumber": 2,
          "isoEquivalent": 494
        }
      }

[ebridges]

If sync'ing wouldn't you want to use the original filename, rather than create one based on an ID?

[philips]

I didn't see a way to get to the original filename. Do you @ebridges?

[ebridges]

Perhaps I'm misunderstanding! My understanding is that since rclone is a sync library the client would be reading a list of photos as gathered from local disk and then uploading them to GPhotos. As such, the client for the GPhotos storage system would have the original filenames. Please advise if I'm missing your point.

[philips]

My use case is copying from Google Photos to S3. Essentially like the http
source in rclone already.

Regardless I don't think gphotos preseves filenames on upload.

On Sat, Jul 14, 2018, 8:36 AM Edward Q. Bridges [e-mail]
wrote:

Perhaps I'm misunderstanding! My understanding is that since rclone is a
sync library the client would be reading a list of photos as gathered from
local disk and then uploading them to GPhotos. As such, the client for the
GPhotos storage system would have the original filenames. Please advise if
I'm missing your point.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-405030997, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AACDCIENNwyyfIq-Bo0sMKrAGXrJETjqks5uGg_7gaJpZM4Hh4qq
.

[Webreaper]

Gphotos does preserve filename. That's how I was able to write a Google
Photos sync tool I mentioned earlier in the thread.

On Sat, 14 Jul 2018, 17:12 Brandon Philips, [e-mail]
wrote:

My use case is copying from Google Photos to S3. Essentially like the http
source in rclone already.

Regardless I don't think gphotos preseves filenames on upload.

On Sat, Jul 14, 2018, 8:36 AM Edward Q. Bridges [e-mail]
wrote:

Perhaps I'm misunderstanding! My understanding is that since rclone is a
sync library the client would be reading a list of photos as gathered
from
local disk and then uploading them to GPhotos. As such, the client for
the
GPhotos storage system would have the original filenames. Please advise
if
I'm missing your point.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-405030997, or
mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AACDCIENNwyyfIq-Bo0sMKrAGXrJETjqks5uGg_7gaJpZM4Hh4qq

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-405033239, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AErI7kx9vchBXaxKkzr7xNu82IF9hK3kks5uGhhngaJpZM4Hh4qq
.

[Webreaper]

By the way, if you just want to copy from Gphotos to S3, and you don't care
about filenames or folders then just enable "Show Google photos in Drive"
and use Rclone to sync from Gdrive to S3.

On Sat, 14 Jul 2018, 17:12 Brandon Philips, [e-mail]
wrote:

My use case is copying from Google Photos to S3. Essentially like the http
source in rclone already.

Regardless I don't think gphotos preseves filenames on upload.

On Sat, Jul 14, 2018, 8:36 AM Edward Q. Bridges [e-mail]
wrote:

Perhaps I'm misunderstanding! My understanding is that since rclone is a
sync library the client would be reading a list of photos as gathered
from
local disk and then uploading them to GPhotos. As such, the client for
the
GPhotos storage system would have the original filenames. Please advise
if
I'm missing your point.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-405030997, or
mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AACDCIENNwyyfIq-Bo0sMKrAGXrJETjqks5uGg_7gaJpZM4Hh4qq

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-405033239, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AErI7kx9vchBXaxKkzr7xNu82IF9hK3kks5uGhhngaJpZM4Hh4qq
.

[philips]

I don't see filename in the API. Can you link please?
https://developers.google.com/photos/library/reference/rest/v1/mediaItems#MediaItem

On Sat, Jul 14, 2018, 9:30 AM Mark Otway [e-mail] wrote:

By the way, if you just want to copy from Gphotos to S3, and you don't care
about filenames or folders then just enable "Show Google photos in Drive"
and use Rclone to sync from Gdrive to S3.

On Sat, 14 Jul 2018, 17:12 Brandon Philips, [e-mail]
wrote:

My use case is copying from Google Photos to S3. Essentially like the
http
source in rclone already.

Regardless I don't think gphotos preseves filenames on upload.

On Sat, Jul 14, 2018, 8:36 AM Edward Q. Bridges <
[email protected]>
wrote:

Perhaps I'm misunderstanding! My understanding is that since rclone is
a
sync library the client would be reading a list of photos as gathered
from
local disk and then uploading them to GPhotos. As such, the client for
the
GPhotos storage system would have the original filenames. Please advise
if
I'm missing your point.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-405030997, or
mute
the thread
<

https://github.com/notifications/unsubscribe-auth/AACDCIENNwyyfIq-Bo0sMKrAGXrJETjqks5uGg_7gaJpZM4Hh4qq
>

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-405033239, or
mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AErI7kx9vchBXaxKkzr7xNu82IF9hK3kks5uGhhngaJpZM4Hh4qq

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-405034322, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AACDCHrFZuI46fmqrKyLDfIyyWRB8GxZks5uGhyPgaJpZM4Hh4qq
.

[Webreaper]

Ah, no idea about that. My tool used the old API. I was more meaning that
Gphotos does store the filename. Whether the new API exposes that is a
different question. Sorry!

On Sat, 14 Jul 2018, 18:49 Brandon Philips, [e-mail]
wrote:

I don't see filename in the API. Can you link please?

https://developers.google.com/photos/library/reference/rest/v1/mediaItems#MediaItem

On Sat, Jul 14, 2018, 9:30 AM Mark Otway [e-mail] wrote:

By the way, if you just want to copy from Gphotos to S3, and you don't
care
about filenames or folders then just enable "Show Google photos in Drive"
and use Rclone to sync from Gdrive to S3.

On Sat, 14 Jul 2018, 17:12 Brandon Philips, [e-mail]
wrote:

My use case is copying from Google Photos to S3. Essentially like the
http
source in rclone already.

Regardless I don't think gphotos preseves filenames on upload.

On Sat, Jul 14, 2018, 8:36 AM Edward Q. Bridges <
[email protected]>
wrote:

Perhaps I'm misunderstanding! My understanding is that since rclone
is
a
sync library the client would be reading a list of photos as gathered
from
local disk and then uploading them to GPhotos. As such, the client
for
the
GPhotos storage system would have the original filenames. Please
advise
if
I'm missing your point.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-405030997,
or
mute
the thread
<

https://github.com/notifications/unsubscribe-auth/AACDCIENNwyyfIq-Bo0sMKrAGXrJETjqks5uGg_7gaJpZM4Hh4qq

>

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-405033239, or
mute
the thread
<

https://github.com/notifications/unsubscribe-auth/AErI7kx9vchBXaxKkzr7xNu82IF9hK3kks5uGhhngaJpZM4Hh4qq
>

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-405034322, or
mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AACDCHrFZuI46fmqrKyLDfIyyWRB8GxZks5uGhyPgaJpZM4Hh4qq

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-405039095, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AErI7rBASjadRzGpXed9NWYd1JRyoC-Tks5uGi8TgaJpZM4Hh4qq
.

[aslafy-z]

I'm sure that Google stores the original filename of the medias. It shows when downloading the file and in the info panel.

I found a feature request asking for including the original filename within the media item object (https://issuetracker.google.com/issues/79656863) that includes a bad workaround. A header is clipped to the baseUrl containing the actual original file name, querying it asking for a 1x1 pixels image could do the trick. It looks like content-disposition:inline;filename="signal-2016-08-09-211213.jpg"

[stickenhoffen]

They fixed it! :)

https://developers.google.com/photos/library/support/release-notes#2018-07-31

[vb0]

Can somebody summarize where are we? Is the API usable at least in theory to meet the original request (upload to Google Photos)?

[amatiasq]

I haven't upload any photos but consumed the API and it worked good enough. I see endpoints to upload new content but I haven't used them.

I'm pretty sure the API is ready for this feature.

[ncw]

I'm pretty sure the API is ready for this feature.

Anyone want to work on it?

[aslafy-z]

I haven't found any way to upload pictures in "high quality" (unlimited
storage offer) through Google Photos upload API other than calling the
"Save some space" trigger which convert any already uploaded pictures to
"high quality" (can take days).
Any alternative ?

[di-dc]

@aslafy-z correct, I don't think there's any way to upload via the API as "high quality": https://developers.google.com/photos/library/guides/api-limits-quotas

"All media items uploaded to Google Photos using the API are stored in full resolution at original quality. They count toward the user’s storage.

Note: If your uploads exceed 25MB per user, your application should remind the user that these uploads will count towards storage in their Google Account."

[Webreaper]

I'd only want original quality. But for anyone who does, they could either
use imagemagick/graphics magic, or if we wanted to be really fancy, have
rclone call either of those if available to shrink pics before upload. 😁

On Tue, 28 Aug 2018, 01:41 di-dc, [e-mail] wrote:

@aslafy-z https://github.com/aslafy-z correct, I don't think there's
any way to upload via the API as "high quality":
https://developers.google.com/photos/library/guides/api-limits-quotas

"All media items uploaded to Google Photos using the API are stored in
full resolution at original quality. They count toward the user’s storage.

Note: If your uploads exceed 25MB per user, your application should remind
the user that these uploads will count towards storage in their Google
Account."

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-416413304, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AErI7uHO34gxD_NoyW_52dowMKiX3UYxks5uVJG9gaJpZM4Hh4qq
.

[PowerKiKi]

My use-case would be opposite, only upload "for free". Do you know if resizing (via imagemagick) before uploading would still be considered "free storage" by Google ? Are the spec of resolution, compression rate and file size documented anywhere in order to be considered "free storage" ?

I always assumed that only Google authored tools would be able to flag images as "free" when uploading them, and that uploading manually any files, even small one, would always count toward your quota...

[Webreaper]

No, it doesn't work like that. If you upload images that are small enough
("less than 16MP" in Google parlance) then they don't count against your
quota. They don't have to be converted by Google, or flagged explicitly.
The biggest challenge is that nobody - to my knowledge - has ever found a
definitive answer to the question of "what does Google count as 16MP". From
a bit of googling, that seems to mean "the longest side of the image
exceeds 2048 pixels".

On Tue, 28 Aug 2018 at 06:29 Adrien Crivelli [e-mail]
wrote:

My use-case would be opposite, only upload "for free". Do you know if
resizing (via imagemagick) before uploading would still be considered "free
storage" by Google ? Are the spec of resolution, compression rate and file
size documented anywhere in order to be considered "free storage" ?

I always assumed that only Google authored tools would be able to flag
images as "free" when uploading them, and that uploading manually any
files, even small one, would always count toward your quota...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369#issuecomment-416455850, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AErI7omZBvNA6HHK7-_kKqnCX2lgq1Vvks5uVNUSgaJpZM4Hh4qq
.

[stickenhoffen]

Is this something that can be added? I'm not using the free storage, I want to store my photos in the original format. My current use case is actually that I want to sync all of my photos from one Google account to another.

[ammgws]

Have you tried the library share feature? (Although you can only share with a max of one account)

[stickenhoffen]

Thanks @ammgws !

[skennedy]

Anyone want to work on it?

@ncw I'm new to Go and this codebase, but I'd be willing to give it a shot if no one has started already

Really love the app btw

[ncw]

@skennedy if you (or anyone else) would like to give it a go then I'd be delighted!

See the CONTRIBUTING guide and in particular the writing a new backend section :-)

It is likely this would need a custom set of integration tests as this isn't a "general purpose" remote, only being able to store photos.

[calisro]

a Go library written for this

https://github.com/nmrshll/google-photos-api-client-go

[dzg]

Any news on this?

[xeor]

https://www.androidpolice.com/2019/06/12/google-drive-photos-sync/ as a heads-up for those relying on that as a fix..

[Webreaper]

Well, that sucks big-time. :(

On Wed, 12 Jun 2019 at 15:12, Lars Solberg [e-mail] wrote:

https://www.androidpolice.com/2019/06/12/google-drive-photos-sync/ as a
heads-up for those relying on that as a fix..

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=ABFMR3Q2OYKHBJCT5X6A32LP2D74TA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXQRW6I#issuecomment-501291897,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABFMR3UV4QYQYBDBG2KFT2LP2D74TANCNFSM4B4HRKVA
.

[vb0]

Well actually things are already very bad so it's hard to suck more than it does now. Actually it might be better. Why it sucks? Photos->Drive was actually the only way I know to bulk-delete things in Photos (as in you enable Photos to be shown in Drive and then you have some folders in Drive just by year where you can very well delete stuff and it'll get deleted in Photos). Otherwise except for shift-click or some other nonsense I don't know how to delete stuff in Photos.

NOW for the other direction (Drive->Photos) it doesn't work at all for Gsuite users! And for the "normal" (usually free, 15GB) users where it works it is next to useless as they'll run out of space immediately if uploading with rclone to gdrive (with the idea that the pictures will end up in Photos as well).

How it can be better? "If you really want to move photos and videos between Drive and Photos, Google will have a new option in Photos called "Upload from Drive." That lets you manually choose images in Drive to import into Photos. That's better than nothing, I suppose." This could be very useful, assuming you don't need to manually do much and that you can just point it to a big folder in Drive (uploaded with rclone) and tell it: put it to Photos too!

[Webreaper]

I don't see filename in the API. Can you link please?

Jus going back to this conversation earlier in the thread - I just tested the API, and if you use the Media Item 'get' endpoint, it definitely returns the filename.

https://developers.google.com/photos/library/reference/rest/v1/mediaItems/get

Here's a snippet of output. So doing the sync should be straightforward. That said, they seem to only return the creation date, not the last-update date, which is an impediment to writing decent sync code.
{
"id": "some unique image ID",
"productUrl": "https://photos.google.com/lr/photo/some unique ID",
"baseUrl": "https://lh3.googleusercontent.com/lr/blah blah blah some huge long ID",
"mimeType": "image/jpeg",
"mediaMetadata": {
"creationTime": "2017-05-24T19:08:27Z",
"width": "3024",
"height": "4032",
"photo": {
"cameraMake": "samsung",
"cameraModel": "SM-G935F",
"focalLength": 4.2,
"apertureFNumber": 1.7,
"isoEquivalent": 100
}
},
"filename": "20170524_200827.jpg"
}

[rektide]

Just want to chime in & re-share more clearly what @xeor already obtusely shared: Google is deprecating the Google Photos <-> Google Drive sync capability (in mother frakking less than a month! frelllllll!!!).

This is what I've been using to access my Google Photos collection, & has been my primary use of rclone. All of a sudden native Google Photos support has become a lot more important to me. I went hunting through issues to find Google Photos support, after I heard of this new addition to the gcemetary & just want to clearly share the state of affairs for this ticket that I suddenly care much more about.

[rektide]

@Webreaper thanks for finding that filename. I'd challenge someone to try uploading two different files with the same filename & see what happens, to be sure we have our bases covered.

I feel like I may have reset my camera in the past & ended up with multiple different files having the same file name, & gphotos dealing with it ok. I don't know how/if rclone could/should handle that!

[Webreaper]

The way I handled it in my sync app using the old API was to assume that
there's a 1:1 relationship between folders and albums. However, you're
right, I think it's possible to have multiple files with the same filename
in a single folder/album. We'd probably need to have the same sort of
functionality as the GDrive dedupe in rclone (since Drive can also have
multiple files with the same name in a single folder).

On Wed, 12 Jun 2019 at 21:38, rektide [e-mail] wrote:

@Webreaper https://github.com/Webreaper thanks for finding that
filename. I'd challenge someone to try uploading two different files with
the same filename & see what happens, to be sure we have our bases covered.

I feel like I may have reset my camera in the past & ended up with
multiple different files having the same file name, & gphotos dealing with
it ok. I don't know how/if rclone could/should handle that!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=ABFMR3W3YD4HVCNBTVJICV3P2FM7LA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXRXINQ#issuecomment-501445686,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABFMR3W6D4OSMKQQ64UXIIDP2FM7LANCNFSM4B4HRKVA
.

[calisro]

Yep. I came to the same conclusion.

https://github.com/ncw/rclone/issues/2922

[calisro]

Thing is... It appears that although drive and photos won't sync, their docs sure do imply that the user can sync directly to both services. So it may all be okay in the end.

[ncw]

Switching off drive to photos syncing is going to affect how I backup my google photos, so I think it is time I put some effort into this!

It looks like google have now got a decent API

https://developers.google.com/photos/library/guides/overview

There doesn't appear to be a go module to help with that as google have withdrawn https://google.golang.org/api/photoslibrary/v1 that https://github.com/nmrshll/google-photos-api-client-go is based on.

However the API is simple enough that I propose building a google photos backend just using the REST API.

If anyone else is already working on this then please let me know and we can collaborate.

[scelfo]

Nick, it's great to see you prioritizing this work! Do you intend to make this a two way backend? I would like to use rclone to upload new local photos from my camera into google photos.

[DurvalMenezes]

Nick, it's great to see you prioritizing this work! Do you intend to make this a two way backend? I would like to use rclone to upload new local photos from my camera into google photos.

+1 ! It would be great to be able to use rclone instead of the clunky and unreliable web interface to download/upload photos!

[ncw]

Nick, it's great to see you prioritizing this work! Do you intend to make this a two way backend? I would like to use rclone to upload new local photos from my camera into google photos.

My first goal will be a read only backend at which point I'll release a beta. I would like it to be a full read-write backend for release if possible.

[ncw]

What the rclone filesystem should look like for google photos needs some discussion.

Google photos aren't stored in directories like a normal file system. The natural way to show them would be all 50,000 files in a single directory which isn't very useful.

There are albums, which photos can appear in multiple of, and there are ways of searching through the photos of these the date based ones look the most useful.

So I think the gphotos backend should have a synthetic root directory, something like this

- media
    - all
        - image
        - image
    - by-year
        - 2019
            - image
            - image
        - 2018
            - image
            - image
    - by-year-month
        - 2019-01
            - image
            - image
        - 2019-02
            - image
            - image
    - by-media-type
        - photo
        - video
    - other filters here
- album
    - name of album
        - picture
        - picture
    - name of album
        - picture
        - picture
- shared albums

Flags could also (or in addition) be provided for the filters.

What would people think if uploads were only possible to albums? That would simplify the syncing logic as these behave the most like directories.

[Webreaper]

For me, uploading only to albums makes the most sense. When I wrote the sync for the old Photos API, that's how I did it.

[ghost]

Interestingly this is what I used to want to but the nature of Google
Photos has changed my use case. I dump everything into Google now as the
primary as our phones are already doing this - I actually like just
dragging files into the browser and forgetting about it. This comes after
years of trying to do it "well" myself. However, I would like to have a
local backup arranged by year... ideally. This is what I have been trying
to find a solution to.

On Fri, 14 Jun 2019 at 11:12, Mark Otway [e-mail] wrote:

For me, uploading only to albums makes the most sense. When I wrote the
sync https://github.com/Webreaper/GooglePhotoSync for the old Photos
API, that's how I did it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=ABPENH4GAZG4TDJT6CJTDLTP2NVKHA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXWLWAQ#issuecomment-502053634,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABPENH2JQVFZ6OSJX3M5KATP2NVKHANCNFSM4B4HRKVA
.

[ncw]

Interestingly this is what I used to want to but the nature of Google Photos has changed my use case. I dump everything into Google now as the primary as our phones are already doing this - I actually like just dragging files into the browser and forgetting about it. This comes after years of trying to do it "well" myself.

Just dumping stuff in would argue for a /upload folder which doesn't show any files except those which have been uploaded recently. This isn't very sync friendly though but would be great for one off dumps of files. Maybe rclone could search for a duplicate before uploading.

I'm trying to avoid rclone having a folder with all the google media in a flat directory structure which might be millions of items.

However, I would like to have a local backup arranged by year... ideally. This is what I have been trying to find a solution to.

That is my preference for backing up google photos.

[Webreaper]

Presumably for those who favour backing up by year - and don't really care about albums - then they would organise their local structure into:

- root
    - 2019
        - picture1
        - picture2
    - 2018
        - picture3
        - picture4

And then sync using the album-based structure. They would end up with an album for each year. But that would be a reasonable approach?

[ghost]

Also, albums being a virtual concept is actually quite useful. Photos are
photos - albums have a one to many concept and work well this way.

On Fri, 14 Jun 2019 at 11:47, Mark Otway [e-mail] wrote:

Presumably for those who favour backing up by year - and don't really care
about albums - then they would organise their local structure into:

• root
  
  
  
  • 2019
    
    
    
    
    
    • picture1
    
    
    
    • picture2
    
    
    
  
  
  • 2018
    
    
    
    
    
    • picture3
    
    
    
    • picture4
    
    
    
  
  

And then sync using the album-based structure. They would end up with an
album for each year. But that would be a reasonable approach?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=ABPENH6SXC5ZFQGAX2UB35TP2NZNJA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXWN6HA#issuecomment-502062876,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABPENH5FDSOUNW4REXC3X2DP2NZNJANCNFSM4B4HRKVA
.

[ebridges]

I have found that having photos organized in albums by year can still leave one with too many photos in each album. The folder structure that I've found to work best is like this:

- root
    - 2019
        - 2019-01-01
              - picture1
              - picture2
              - picture3
        - 2019-01-02
              - picture1
              - picture2
              - picture3
etc...

[ebridges]

A primary issue that I faced in trying to keep photos in sync with Google Photos is that their API does not (currently) offer a way to delete images. https://issuetracker.google.com/issues/109759781

[ncw]

I have found that having photos organized in albums by year can still leave one with too many photos in each album. The folder structure that I've found to work best is like this:

- root
    - 2019
        - 2019-01-01

That is the exact scheme I use for my photo archive so I'll implement that for definite!

A primary issue that I faced in trying to keep photos in sync with Google Photos is that their API does not (currently) offer a way to delete images. https://issuetracker.google.com/issues/109759781

I failed to notice that :-(

This means that this backend can't possibly be a full featured backend. That is probably OK but it will make writing integration tests very difficult if you can't upload photos and then delete them.

I'm going to concentrate on two use cases

• backing up your gphotos
• uploading new photos into either an album or a general /uploads directory.
  
  
  
  • If we can't delete then it can't be the destination for rclone sync only rclone copy.
  
  

[ebridges]

I'd enjoy helping out on that -- I do have some experience working with the API and am always looking for ways to brush up on Go.

How do you normally plan features like this? Is it just via this thread?

[DurvalMenezes]

I have found that having photos organized in albums by year can still leave one with too many photos in each album. The folder structure that I've found to work best is like this:

- root
    - 2019
        - 2019-01-01

That is the exact scheme I use for my photo archive so I'll implement that for definite!

+1 to that! ;-)

• backing up your gphotos
• uploading new photos into either an album or a general /uploads directory.
  *If we can't delete then it can't be the destination for rclone sync only rclone copy.

That would work just great for my case: not having photos deleted in Google Drive after deleting them locally and then sync'ing is not such a big deal.

One observation: how is the "picture quality" thing going to be handled? You folks know GooglePhotos only allow unlimited photos (ie, without counting towards your total Google quota) in "High Quality", not in "Original" form, right? Additionally, wouldn't that change the size of the pictures on the remote and force a (re)copy every time? (I'm not familiar with the Google Photos API, perhaps these are non-issues -- I'm just bringing them up so they can be discussed).

[ncw]

One observation: how is the "picture quality" thing going to be handled? You folks know GooglePhotos only allow unlimited photos (ie, without counting towards your total Google quota) in "High Quality", not in "Original" form, right? Additionally, wouldn't that change the size of the pictures on the remote and force a (re)copy every time? (I'm not familiar with the Google Photos API, perhaps these are non-issues -- I'm just bringing them up so they can be discussed).

I was imagining "high quality" vs "original" would be an command line option but I read this in the API docs

All media items uploaded to Google Photos using the API are stored in full resolution at original quality. They count toward the user’s storage.

Note: If your uploads exceed 25MB per user, your application should remind the user that these uploads will count towards storage in their Google Account.

Not uploading duplicates is a problem I haven't figured out yet.

I'd enjoy helping out on that -- I do have some experience working with the API and am always looking for ways to brush up on Go.

Great!

How do you normally plan features like this? Is it just via this thread?

Normally one issue per feature. If it gets really complicated then we can make a design doc.

[Webreaper]

Not uploading duplicates is a problem I haven't figured out yet.

I worked around this with a combination of file-size, filename, created date and so on, to identify uniqueness within an album. Assuming your local storage is sane, it shouldn't be too difficult. The downside is that presumably if there's no 'delete' API, you also can't 'overwrite' a photo, which is a real pain.
The other problem I had with the old API (and it looks to persist in the new API - I thought these guys at Google were supposed to be good at coding/design?!?) is that there's no lastUpdated date or other metadata, so even if there is an overwrite API, it's really hard to figure out when to apply it without re-uploading everything from scratch every time. In the end, I did something super-hacky, writing the last-updated date to the EXIF/IPTC tags of the photo before uploading, and then querying that when I decided whether a photo had changed and needed uploading. It's pretty horrible though.

[DurvalMenezes]

All media items uploaded to Google Photos using the API are stored in full resolution at original quality. They count toward the user’s storage.

Oh crap! :-( This will make it unfeasible for my usage here, and I guess for almost anyone :-(

@ncw: would it be possible to use this new feature to copy files between different Google Photo accounts? If yes, I would at least be able to work around the limitation by using my unlimited EDU account...

[Webreaper]

This will make it unfeasible for my usage here, and I guess for almost anyone

Not sure about that. I pay £7 a month for my Google One account and that gives me 2TB for backing up all my stuff, including photos. So it'll be extremely useful. ;)

[DurvalMenezes]

Not sure about that. I pay £7 a month for my Google One account and that gives me 2TB for backing up all my stuff, including photos. So it'll be extremely useful. ;)

That's just because you are not an old Scrooge^H^H^H^H^H^H^H^H^H^H frugal person like me ;-)

Seriously, I have a separate, unlimited EDU account and I plan to make full use of it before paying Google anything :-)

[ncw]

If anyone wants to have a go, here is a prototype google photos backend. It is read only for the time being. I implemented a few different ways of reading the media.

https://beta.rclone.org/branch/v1.48.0-003-g8551241c-gphotos-beta/ (uploaded in 15-30 mins)

[ncw]

Fixed the Entry doesn't belong in directory bug:

https://beta.rclone.org/branch/v1.48.0-003-gf2c25527-gphotos-beta/ (uploaded in 15-30 mins)
(uploaded in 15-30 mins)

[Maxr1998]

Just tried it, and it works perfectly for my use case! Currently (finally) in the process of backing up all my pictures from Google Photos.Thanks a lot!

[ghost]

I have tried it too and it looks great. How does it currently manage
chnages to the local file system? If I change the date on a photo in GP
will this "move" to the new year folder locally? I thought I saw that
happening but cannot now reproduce.

On Tue, 18 Jun 2019 at 11:57, Max Rumpf [e-mail] wrote:

Just tried it, and it works perfectly for my use case! Currently (finally)
backing up all my pictures from Google Photos.Thanks a lot!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=ABPENH2ZEPPGLSPCJV4XZWTP3C5S3A5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX6ABTI#issuecomment-503054541,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABPENH6WJFPGZEI74SFEAPTP3C5S3ANCNFSM4B4HRKVA
.

[ncw]

Fixed downloading videos (rclone was downloading them as images).

So if anyone downloaded videos, please delete them and re-download!

https://beta.rclone.org/branch/v1.48.0-006-g8aea25d7-gphotos-beta/ (uploaded in 15-30 mins)

If I change the date on a photo in GP will this "move" to the new year folder locally? I thought I saw that
happening but cannot now reproduce.

Yes it should do if you can change the Created date on gogle photos (I don't know whether that is possible or not.)

[Maxr1998]

So, I noticed that the file sizes of the downloaded media hugely differs from the sizes listed on Photos itself. For example, the photos are generally less than half the size (but at the same resolution), and one 2160p HEVC video with a bitrate of ~42 mbit/s and a size of 380 MB changed to 2.7 MB of 1080p, ~180 kbit/s H.264 video. I backup everything in original quality. Did I miss an option to change the download quality? Does the API only serve compressed files? Looking forward to hear some information on this.

[DrSpaldo]

I am also following and looking forward to this. I would like to be able to use the Synology to backup Google Photos

[peterloeffler]

Oh man!
Nearly started to investigate in the API and go for myself but luckily found this first.
You saved me many many hours.
Thx a lot!

[ncw]

So, I noticed that the file sizes of the downloaded media hugely differs from the sizes listed on Photos itself. For example, the photos are generally less than half the size (but at the same resolution), and one 2160p HEVC video with a bitrate of ~42 mbit/s and a size of 380 MB changed to 2.7 MB of 1080p, ~180 kbit/s H.264 video. I backup everything in original quality. Did I miss an option to change the download quality? Does the API only serve compressed files? Looking forward to hear some information on this.

Here is what the docs say about downloading

https://developers.google.com/photos/library/guides/access-media-items#base-urls

Rclone is using =d to download images.

• This strips EXIF location (according to the docs and my tests)
• The download direct from google photos website has the EXIF location
• See: https://issuetracker.google.com/issues/112096115

Rclone is using =dv to download videos

• This downloads a really compressed version of the video compared to downloading it via the google photos web interface
• The docs don't state the video isn't the original
• See: https://issuetracker.google.com/issues/113672044

Unfortunately I think photos web UI must be using a different API, one which isn't publically available.

This is all rather annoying - if you can't download the originals via the API I'm wondering whether this backend will be useful at all. I guess it would make a backup of last resort.

[Maxr1998]

@ncw thanks for the information. Yeah, without having the same quality as Google Takeout, it doesn't really make a lot of sense, I hope Google extends/improves the API to fix this shortcoming - it's still our own data after all, so why shouldn't we be allowed to download it in it's original quality.

[ncw]

I've commented on the issues and I've also signed up to be a Google Photos partner.

So should I release this backend?

• I can put lots of warnings in the docs about it not downloading original quality pics
• But there will always be people that don't notice those warnings

So worth releasing anyway?

[Webreaper]

If there's warnings in the backend setup, then people can't really claim
they haven't seen them as configuring the back-end is pretty interactive.

I haven't been keeping up, but is this download-only for now? My interest
is really to upload to GPhotos, although given all the restrictions (can't
delete, etc) I'm going to stick with Amazon Prime Photos for my 'original'
photos backups.

On Fri, 21 Jun 2019 at 10:18, Nick Craig-Wood [e-mail]
wrote:

I've commented on the issues and I've also signed up to be a Google Photos
partner.

So should I release this backend?

• I can put lots of warnings in the docs about it not downloading
  original quality pics
• But there will always be people that don't notice those warnings

So worth releasing anyway?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=ABFMR3XYR3NEYCHEBQB6P2TP3SMEPA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYH5ZDQ#issuecomment-504355982,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABFMR3UYY3BMWUISEVTOC6LP3SMEPANCNFSM4B4HRKVA
.

[rhatguy]

I second those comments. Frankly I don't "trust" Google with my
originals. They have shut down to many services and are to free with
reducing quality for me to trust them with data that I intend to keep
long-term (lifetime). I do however love their interface and would like to
use them as one of the ways to view/browse my photo collection which is
primarily housed and protected in other ways.

On Fri, Jun 21, 2019, 5:20 AM Mark Otway [e-mail] wrote:

If there's warnings in the backend setup, then people can't really claim
they haven't seen them as configuring the back-end is pretty interactive.

I haven't been keeping up, but is this download-only for now? My interest
is really to upload to GPhotos, although given all the restrictions (can't
delete, etc) I'm going to stick with Amazon Prime Photos for my 'original'
photos backups.

On Fri, 21 Jun 2019 at 10:18, Nick Craig-Wood [e-mail]
wrote:

I've commented on the issues and I've also signed up to be a Google
Photos
partner.

So should I release this backend?

• I can put lots of warnings in the docs about it not downloading
  original quality pics
• But there will always be people that don't notice those warnings

So worth releasing anyway?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=ABFMR3XYR3NEYCHEBQB6P2TP3SMEPA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYH5ZDQ#issuecomment-504355982
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/ABFMR3UYY3BMWUISEVTOC6LP3SMEPANCNFSM4B4HRKVA

.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=AC3I765AZRANC23F7JGPPHDP3SMPLA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYH6AKY#issuecomment-504356907,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC3I762XYR3H65B6BI65UF3P3SMPLANCNFSM4B4HRKVA
.

[Webreaper]

Well, I'd probably not go that far. I trust Google with my originals - they've been my go-to for Cloud backup of photos for years. My problem was that when I hit > 1TB, it was going to cost me $80/month to go to the next plan. So I switched to Prime Photos, where I get unlimited original storage of photos, for free. I currently have 2TB of pics on there. Sadly, no rclone support. ;)

What I use Google Photos for is their search. I resize all my pictures and upload 800x600 thumbs to Google Photos, and their image recognition is so smart that it helps me find things that aren't tagged or named (e.g., search for "red car"). Currently, I can just rclone the thumbs to GDrive, and they feed into GPhotos automatically. In future, I'd need some sort of sync tool to put them in GPhotos directly. So annoying!

[rhatguy]

It's not so much that I worry about them loosing data as them making is
inaccessible or unusable. I've got no problem storing data on drive
because it has a industry standard way of accessing the data and is easy to
get the data out of.

I spent loads of time tagging pictures in Picasa and uploading to picasaweb
only to find that Google photos doesn't respect any of those tags. I did
find a 3rd party program that would write the tags Picasa created into the
metadata of the pictures though. Now I'm worried to use photos for tagging
because I haven't seen a good way to export that info into a standard
format a year or two from now when they decide to move onto the next
iteration of photo storage. There doesn't seem to be any way to update the
metadata on a photo if you for instance change the capture date or tag
someone. Given that how can we invest time in creating that info inside
photos.

On Fri, Jun 21, 2019, 7:36 AM Mark Otway [e-mail] wrote:

Well, I'd probably not go that far. I trust Google with my originals -
they've been my go-to for Cloud backup of photos for years. My problem was
that when I hit > 1TB, it was going to cost me $80/month to go to the next
plan. So I switched to Prime Photos, where I get unlimited original storage
of photos, for free. I currently have 2TB of pics on there. Sadly, no
rclone support. ;)

What I use Google Photos for is their search. I resize all my pictures and
upload 800x600 thumbs to Google Photos, and their image recognition is so
smart that it helps me find things that aren't tagged or named (e.g.,
search for "red car"). Currently, I can just rclone the thumbs to GDrive,
and they feed into GPhotos automatically. In future, I'd need some sort of
sync tool to put them in GPhotos directly. So annoying!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=AC3I76YJQ5Z2PK2RR3JYBSLP3S4KTA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYIHFGA#issuecomment-504394392,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC3I76Z2HSMHHAO5WITGDIDP3S4KTANCNFSM4B4HRKVA
.

[ebridges]

It's of doubtful usage, to me at least. Is there a way to mark a backend as "alpha" or something?

[calisro]

I think you should still publish it. For me though I need the ability to write to photos instead of read. I'm going to sync to drive myself and then upload to photos for the app interface.

[Webreaper]

I spent loads of time tagging pictures in Picasa and uploading to picasaweb only to find that Google photos doesn't respect any of those tags.

Google Photos stores the IPTC tags - if you upload a photo to it, and then re-download, the tags are still there. It makes me sad that GPhotos doesn't read those tags and index them for searching, but what would I know.

[rhatguy]

It stores them but do any of the changes that you make inside the photos
app make it into the metadata? That's why I'd want to sync TO photos
instead of FROM photos. To me it's a read only interface because any
changes there are not persisted as far as I know.

On Fri, Jun 21, 2019, 8:39 AM Mark Otway [e-mail] wrote:

I spent loads of time tagging pictures in Picasa and uploading to
picasaweb only to find that Google photos doesn't respect any of those tags.

Google Photos stores the IPTC tags - if you upload a photo to it, and then
re-download, the tags are still there. It makes me sad that GPhotos doesn't
read those tags and index them for searching, but what would I know.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=AC3I767S2T5JWBRA7VCHNSLP3TDY7A5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYILFEY#issuecomment-504410771,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC3I766VFMWCZQSCCQY7V3DP3TDY7ANCNFSM4B4HRKVA
.

[g0kuz]

I'm confused,

When july 10 hits, there's no option for google folder?

[ncw]

@calisro wrote:

I think you should still publish it. For me though I need the ability to write to photos instead of read. I'm going to sync to drive myself and then upload to photos for the app interface.

OK I'll publish it :-)

I'd like a bit of advice on the upload...

I'm planning that you can make albums in /albums with mkdir and upload to them.

If you upload to an album then the sync process will work pretty naturally, except for the fact that there can't be a nested directory structure within an album.

I did have a plan where you'd be able to copy files to /uploads. These will remain in uploads for the duration of the run of rclone but will disappear afterwards, so acting like an uploads directory for an FTP server. The files would then appear in the /media directories according to their creation date.

This will solve just general uploads but it will make repeated syncing impossible without making lots of duplicates. That has made me think this isn't such a good idea and I should scrap it.

General media uploads can always be done to an album, then the album can be removed and the media will remain.

Is upload to an album sufficient for everyone's needs?

Any thoughts?

[calisro]

I don't use the album feature really at all. I was always concerned with spending time making albums only to have them go away if Google made changes or if I wanted to move photos elsewhere. and given that its hard to extract stuff now (except using take-out) i doubt i'd use them.

For my needs, i'm looking to just copy (or preferably sync if possible) many thousands of photos to the photos app so they are presented in the GUIs. The directories i'm trying to sync though will have nested folders. I'm not really concerned about losing the folder names since i'm not using 'google photos' as the source of record but rather just a GUI. The key for me is that the date data is correct based on the photo metadata which I think google photos will implicitly scrape from the EXIF data after upload.

Additionally, it would be nice to upload these to lower resolution since it isn't the source of record and I can save space on my account. But even if they are full quality, that works as well.

I think the most important aspect first and foremost is the ability to 'sync' correctly without creating large duplicates.

[kardianos]

@ncw Thank you for making this backend and publishing it. GPhotos is a decent photos bucket, but I really want a backup of my photos on a different provider / local drive of some quality. I personally don't need the original quality, or even metadata. I just don't want to loose them.

Uploads to albums sound fine to me...

[Webreaper]

Per my comment earlier in the thread, albums are nice if they're automatically created based on the folders in which the uploading images are stored locally. So if I have:

 photo
    - Birthday
    - Christmas
    - New Puppy

the upload would upload the photos into those 3 albums. This is exactly how the folders within GDrive are represented today - each folder is mapped to an Album.

There's a couple of reasons it would be good to do this

1. For those of us who have photos organised into album folders, the same albums would be created in GPhotos.
2. For people who just want a single upload of all the photos, they could either have one huge album, or put all their images in the root 'photo' folder, and they'll all just be uploaded without an album.

It's worth mentioning that when you actually upload to GPhotos (rather than GPhotos being a 'view' on a GDrive folder) I believe there's a limit of something like 1,000 photos in an album. So uploading hundreds of thousands of photos to GPhoto without segregating them into Albums might be problematic.

[calisro]

@Webreaper I like that approach. I'd also like a flag though to disable album creation but I could live without a flag as well.

There is no limit to upload to gphotos (non-album) but there is an 'album' content limit yes. :(

[ncw]

Here is the latest with a directory listing bug fixed (it was skipping the first item) and the ability to make albums and upload to albums. Note you can't remove albums (it isn't in the api) and you can't upload to albums you didn't create with rclone (also not in the API).

Note that uploads count against your drive quota.

https://beta.rclone.org/branch/v1.48.0-010-g63014f46-gphotos-beta/ (uploaded in 15-30 mins)

@calisro wrote:

For my needs, i'm looking to just copy (or preferably sync if possible) many thousands of photos to the photos app so they are presented in the GUIs. The directories i'm trying to sync though will have nested folders. I'm not really concerned about losing the folder names since i'm not using 'google photos' as the source of record but rather just a GUI. The key for me is that the date data is correct based on the photo metadata which I think google photos will implicitly scrape from the EXIF data after upload.

Sync should work to albums now. However they won't support nested folders...

See below for an idea on how this might work.

Additionally, it would be nice to upload these to lower resolution since it isn't the source of record and I can save space on my account. But even if they are full quality, that works as well.

I don't think that the API allows anything but full res uploads. Ideally it would allow the low res uploads that don't count against your quota but I don't think it does.

I think the most important aspect first and foremost is the ability to 'sync' correctly without creating large duplicates.

Sync to albums should work now...

@Webreaper wrote:

Per my comment earlier in the thread, albums are nice if they're automatically created based on the folders in which the uploading images are stored locally. So if I have:

    - Birthday
    - Christmas
    - New Puppy

the upload would upload the photos into those 3 albums. This is exactly how the folders within GDrive are represented today - each folder is mapped to an Album.

If you had a 2 level deep layout like the above then rclone will work as you wish now.

That gives me an idea though - what if rclone represented nested directories

• root
  
  
  
  • sub
    
    
    
    
    
    • subsub
    
    
    
    • subsub2
    
    
    
  
  

As albums with names "root", "root/sub", "root/sub/subsub", "root/sub/subsub"? That would be one way of trying to flatten a hierarchical filesystem into google photos rather than munging the file names. (Those / would be a unicode equivalent probably).

That would have the advantage that listing the albums only would create the directory tree. I already cache the albums so this might work quite neatly.

@calisro wrote:

I like that approach. I'd also like a flag though to disable album creation but I could live without a flag as well.

Do you mean a --read-only flag? Or rclone could offer you a read only option on backend creation so when it gets the oauth token it requests a read only one.

[ebridges]

As albums with names "root", "root/sub", "root/sub/subsub", "root/sub/subsub"?

I was actually about to write a comment proposing this exact convention. At that point, you can support arbitrary nesting of folders.

Thanks for the fast response on all of this!! :boom:

[ncw]

Here is an approaching complete google photos backend :-)

The docs are here: googlephotos.md - I'd appreciate a read through to check them over and see if they make sense.

I implemented the heirachical album scheme and it makes the space under album/ behave pretty much like a normal file system.

https://beta.rclone.org/branch/v1.48.0-013-gbb4eeedf-gphotos-beta/ (uploaded in 15-30 mins)

[calisro]

Is there a reason that we can only write to albums? Albums have file limits and I rarely use them. Is it possible to also write direct without albums like nmrshll/gphotos-uploader-cli does?

Also I see severe rate limiting. Is that typical? I could barely list 50 photos on my first lsl.

2019/06/26 13:28:12 DEBUG : pacer: low level retry 1/10 (error Quota exceeded for quota metric 'photoslibrary.googleapis.com/read_requests' and limit 'ReadsPerMinutePerUser' of service 'photoslibrary.googleapis.com' for consumer 'project_number:480718780999'. (429 RESOURCE_EXHAUSTED))

Thankyou @ncw

[ncw]

Is there a reason that we can only write to albums? Albums have file limits and I rarely use them. Is it possible to also write direct without albums like nmrshll/gphotos-uploader-cli does?

Any files you write to albums will appear in the general files list, so you could upload to an album then delete it...

You can read the contents of albums and delete files from then both of which will make the rclone copy/sync experience relatively normal.

The problem with making a general upload is that rclone needs to be able to see if the file is already uploaded and there is no API for searching for a filename. Rclone can't even list the recently uploaded files, it can only list files by EXIF info which bears no relation to the datestamp on the file.

The best I could do is implement an /upload directory. This would upload any files copied in, and it would list out any files that have been uploaded in this run of rclone. This means that if rclone had to do --retries then duplicates wouldn't be uploaded.

However if you quitted rclone and restarted it then it would upload everything again. I note in my tests that if you upload an image twice, then google only stores it once, so this only costs you bandwidth not storage at google.

I could store the uploaded files on disk I suppose.

I could also list all the media and find the file name like that, however (as you noted) google photos is quite slow and that would take some time!

Which one of those options would work for you?

1. /upload directory which only shows files in uploaded in this session of rclone
2. /upload directory which shows all files uploaded in the last X days (cached on disk)
3. /upload directory which would be really slow on first access but would show all the files.

Option 1 is relatively easy and could be upgraded to option 2 if required. I guess 2. and 3. could be combined so a slow read to fill up a local db of files. Though there isn't an API method to keep that updated which is a bit painful...

There is another potential problem here with directories - I'd have to munge them into the file name I think, so if you copied dir/subdir/file.jpg to uploads then it should make a image with filename dir/subdir/file.jpg otherwise there will be lots of duplicates. Does that sound OK?

So in summary, rclone can provide a good sync experience to albums. A general purpose upload will be lacking in various different ways!

Also I see severe rate limiting. Is that typical? I could barely list 50 photos on my first lsl.

2019/06/26 13:28:12 DEBUG : pacer: low level retry 1/10 (error Quota exceeded for quota metric 'photoslibrary.googleapis.com/read_requests' and limit 'ReadsPerMinutePerUser' of service 'photoslibrary.googleapis.com' for consumer 'project_number:480718780999'. (429 RESOURCE_EXHAUSTED))

I'm pretty sure that is because I haven't had the oauth verified yet - I'm in the process of doing that now. I'll then get bigger limits and I can increase ReadsPerMinutePerUser.

[calisro]

However if you quitted rclone and restarted it then it would upload everything again. I note in my tests that if you upload an image twice, then google only stores it once, so this only costs you bandwidth not storage at google.

Are you sure that is the case? When I played with the nmrshll/gphotos-uploader-cli it uploaded duplicates so I hope this is the case with the way rclone is doing it.

So in summary, rclone can provide a good sync experience to albums. A general purpose upload will be lacking in various different ways!

Hm. I understand. I'm not opposed to doing albums if the albums provide a better sync experience. Any of the other options either make syncs painful or carry the weight of a local db. I really would rather not deal with local cache db. I need to play around with this a bit on a new account to see how the albums will work. My biggest issue with Albums is the 1000 file limit. Perhaps rclone could check if the limit will be exceeded and do something better than simply fail. Although i'm not sure the solution there either.

I'm pretty sure that is because I haven't had the oauth verified yet - I'm in the process of doing that now. I'll then get bigger limits and I can increase ReadsPerMinutePerUser.

Is this similar to regular drive where creating our own credentials will help with sync? or if we did that, we'd need to be 'verified' as well?

[ncw]

However if you quitted rclone and restarted it then it would upload everything again. I note in my tests that if you upload an image twice, then google only stores it once, so this only costs you bandwidth not storage at google.

Are you sure that is the case? When I played with the nmrshll/gphotos-uploader-cli it uploaded duplicates so I hope this is the case with the way rclone is doing it.

That is how it appears to work for me - if you upload the same media twice, you only get one media item.

I've implemented the /uploaddirectory now which seems to work reasonably well.

So in summary, rclone can provide a good sync experience to albums. A general purpose upload will be lacking in various different ways!

Hm. I understand. I'm not opposed to doing albums if the albums provide a better sync experience. Any of the other options either make syncs painful or carry the weight of a local db. I really would rather not deal with local cache db. I need to play around with this a bit on a new account to see how the albums will work. My biggest issue with Albums is the 1000 file limit. Perhaps rclone could check if the limit will be exceeded and do something better than simply fail. Although i'm not sure the solution there either.

According to the API docs

Each album can contain up to 20,000 media items. Requests to add more items that would exceed this limit will fail.

I'm pretty sure that is because I haven't had the oauth verified yet - I'm in the process of doing that now. I'll then get bigger limits and I can increase ReadsPerMinutePerUser.

Is this similar to regular drive where creating our own credentials will help with sync? or if we did that, we'd need to be 'verified' as well?

Using your own APIs will give you just what I've got at the moment, but you won't have to share, so it is probably a good idea right now.

Getting verified is a hassle, and looks like I'll have to get verified as a google photos partner as well to get any decent transaction limit!

[calisro]

I've implemented the /upload directory now which seems to work reasonably well.

How is that currently working? Like option 1 above?

[ncw]

I've implemented the /upload directory now which seems to work reasonably well.

How is that currently working? Like option 1 above?

Yes, that was easy.

[etique57]

Hello,

Just chiming to say that the rclone support of Google Photos, as far as I can understand it (didn't try yet), seems to perfectly fit my use-case:

• publishing (=/= backup) a curated list of full-res pictures & movies
• retrieving the "magic" Assistant pictures automatically created based on the pictures uploaded

So thanks @ncw for the beautiful work, and looking forward to put it to good use!

[lv913]

Can you provide an updated documentation link? The previous one seems not working anymore.

[ncw]

The docs are here: https://tip.rclone.org/googlephotos/

I've merged this to master now so it is in the latest beta. However I have a really limited number of transactions available until rclone is authenticated (which I'm doing at the moment) so you might want to make your own client_id.

[ncw]

Hello,

Just chiming to say that the rclone support of Google Photos, as far as I can understand it (didn't try yet), seems to perfectly fit my use-case:

• publishing (=/= backup) a curated list of full-res pictures & movies
• retrieving the "magic" Assistant pictures automatically created based on the pictures uploaded

So thanks @ncw for the beautiful work, and looking forward to put it to good use!

:-)

Note that the videos aren't full res when you download them though - check: https://tip.rclone.org/googlephotos/#limitations for the annoyingly long list of limitations which I hope google will fix.

[Peregrinox]

Hello,

I'd just made some tests with Beta 1.48.56,

rclone copy test gphotos:Album\test

If the "test" dir/Album has more than 1 image, the above command creates an album for each image, all with same name "test". At least, this is what I observe in the google photos android App

[ncw]

Hello,

I'd just made some tests with Beta 1.48.56,

rclone copy test gphotos:Album\test

If the "test" dir/Album has more than 1 image, the above command creates an album for each image, all with same name "test". At least, this is what I observe in the google photos android App

I've reproduced this - thanks for reporting :-) A work-around is to create the album first with rclone mkdir. I'll post a fix shortly.

[ncw]

I've merged a fix for the problem reported by @Peregrinox . That will be in the latest beta in 15-30 mins with version v1.48.0-063-gd7016866

[Peregrinox]

@ncw, many thanks for the fix.

After some more experimentation, now with beta 063, I founded that creation of Album with 'rclone mkdir gphotos:/album/test' before adding images to it, works well if there is some delay between the two comands.

In the bash script:

ALBUM= some dir with multiples images
rclone mkdir gphotos:album/"$ALBUM"
# sleep 6
rclone copy "$ALBUM" gphotos:album/"$ALBUM"

Without the intermediate "sleep" delay, the "copy" causes the same behaviour, as described in my previous post, of one album per image in the ALBUM folder. It seems that album creation is not inmediatly avaiable to the API just after the "mkdir".

Of course, typing the comands in a shell, causes the same effect as the "sleep X" delay if you dont't have superfasts fingers

[ncw]

Without the intermediate "sleep" delay, the "copy" causes the same behaviour, as described in my previous post, of one album per image in the ALBUM folder. It seems that album creation is not inmediatly avaiable to the API just after the "mkdir".

Hmm, you shouldn't need the rclone mkdir now at all. What happens if you try without it?

[Peregrinox]

It´s behaving nice now. I think that my tests with some old duplicated album names were the source of my troubles. I tried again with some fresh album names and images.

Sorry about the confusion.

There is no need to mkdir at all. My albumns are upped great with copy or move

[ncw]

There is no need to mkdir at all. My albumns are upped great with copy or move

Great :-)

[badzz]

Hi, Thanks a lot for the hard work
If I want to do a full backup of my gphotos and keep the albums structure (some photos are not in albums),should I do
rclone sync remote:media/all /mnt/backup/media
and
rclone sync remote:album /mnt/backup/album
And I will have twice my photos that are in albums , or there is a better way ?
Thanks
PS: There seems to be a bug with album names containing / character . the name stops before the /

[calisro]

Im a little confused. I understand that videos aren't being downloaded at full resolution but what about pictures? I took a look at the limitations and it only talks about EXIF data. To me it looks like the downloads of images are being scaled so I wanted to confirm. If so, that should be added to the limitations page.

[badzz]

@calisro
Hi
the bug clearly states that the sizes are different as well. My test showsthat indeed as well
rclone doc does not say it, but it is indeed the case (plus EXIF location missing)

https://issuetracker.google.com/issues/112096115

du -sh web.jpg api.jpg
2.4M web.jpg
616K api.jpg

On Thu, Jul 11, 2019 at 4:37 PM calisro [e-mail] wrote:

Im a little confused. I understand that videos aren't being downloaded at
full resolution but what about pictures? I took a look at the limitations
and it only talks about EXIF data. To me it looks like the downloads of
images are being scaled so I wanted to confirm. If so, that should be added
to the limitations page.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=ABR5UYNAU4LEDAT42B6X3PLP65ARNA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZW45AY#issuecomment-510512771,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABR5UYKF6MIJDQHIU3H2IMLP65ARNANCNFSM4B4HRKVA
.

[calisro]

(Wasn't sure if I should open a new ISSUE. I can if you'd like @ncw )

I was testing this today. I'm using this to upload photos rather than downloading them for this test.

I'm finding that I am seeing a lot of corrupted on transfers. I'm not sure why but it is consistent for many photos/vids. I thought there may be the case where the file(s) already existed in GPhotos uploaded by a different method but that doesn't appear to be the case as I've searched both in the web interface and the rclone cli to see if it exists.

This is one example:

root@xxxx:/data/bin# rclone copy test robgs-photos:album -vv --tpslimit=1 --include="2019/05/Emerson and Maddy Joke Talen1 show.jpg"
2019/07/11 13:48:17 DEBUG : rclone: Version "v1.48.0-066-g5433021e-beta" starting with parameters ["rclone" "copy" "test" "robgs-photos:album" "-vv" "--tpslimit=1" "--include=2019/05/Emerson and Maddy Joke Talen1 show.jpg"]
2019/07/11 13:48:17 DEBUG : Using config file from "/root/.rclone.conf"
2019/07/11 13:48:17 INFO  : Starting HTTP transaction limiter: max 1 transactions/s with burst 1
2019/07/11 13:48:17 DEBUG : Google Photos path "album": List: dir=""
2019/07/11 13:48:18 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:18 DEBUG : Google Photos path "album": List: dir="2019"
2019/07/11 13:48:18 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:18 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:18 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:18 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:18 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:18 DEBUG : 2019/07: Excluded
2019/07/11 13:48:18 DEBUG : 2019/06: Excluded
2019/07/11 13:48:18 DEBUG : 2019/01: Excluded
2019/07/11 13:48:18 DEBUG : 2019/03: Excluded
2019/07/11 13:48:18 DEBUG : 2019/04: Excluded
2019/07/11 13:48:18 DEBUG : Google Photos path "album": List: dir="2019/05"
2019/07/11 13:48:18 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:18 DEBUG : Google Photos path "album": Put: src=2019/05/Emerson and Maddy Joke Talen1 show.jpg
2019/07/11 13:48:18 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Update: src=2019/05/Emerson and Maddy Joke Talen1 show.jpg
2019/07/11 13:48:18 INFO  : Google Photos path "album": Waiting for checks to finish
2019/07/11 13:48:18 INFO  : Google Photos path "album": Waiting for transfers to finish
2019/07/11 13:48:21 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: >Update: err=<nil>
2019/07/11 13:48:21 DEBUG : Google Photos path "album": >Put: 
2019/07/11 13:48:21 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/11 13:48:21 DEBUG : Google Photos path "album": List: dir="2019/05"
2019/07/11 13:48:22 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:23 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/11 13:48:23 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/11 13:48:23 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/11 13:48:23 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/11 13:48:23 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/11 13:48:23 ERROR : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: corrupted on transfer: sizes differ 85040 vs 67499
2019/07/11 13:48:23 INFO  : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Removing failed copy
2019/07/11 13:48:24 ERROR : Attempt 1/3 failed with 2 errors and: corrupted on transfer: sizes differ 85040 vs 67499
2019/07/11 13:48:24 DEBUG : Google Photos path "album": List: dir=""
2019/07/11 13:48:24 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:24 DEBUG : Google Photos path "album": List: dir="2019"
2019/07/11 13:48:24 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:24 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:24 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:24 DEBUG : Google Photos path "album": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2019/07/11 13:48:24 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:24 DEBUG : 2019/07: Excluded
2019/07/11 13:48:24 DEBUG : 2019/06: Excluded
2019/07/11 13:48:24 DEBUG : 2019/01: Excluded
2019/07/11 13:48:24 DEBUG : 2019/03: Excluded
2019/07/11 13:48:24 DEBUG : 2019/04: Excluded
2019/07/11 13:48:24 DEBUG : Google Photos path "album": List: dir="2019/05"
2019/07/11 13:48:25 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/11 13:48:25 DEBUG : Google Photos path "album": Put: src=2019/05/Emerson and Maddy Joke Talen1 show.jpg
2019/07/11 13:48:25 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Update: src=2019/05/Emerson and Maddy Joke Talen1 show.jpg
2019/07/11 13:48:25 INFO  : Google Photos path "album": Waiting for checks to finish
2019/07/11 13:48:25 INFO  : Google Photos path "album": Waiting for transfers to finish

EDIT: Well this is interesting. If I upload it to the /upload path instead of albums, it works. Now its in the GUI too. and if I rerun the album upload after it still fails with the exact same message. I've tried changing the album names just to see but that didn't help.

root@cloudc:~# rclone copy robgs:Local-Pictures robgs-photos:upload -vv --tpslimit=1 --include="2019/05/Emerson and Maddy Joke Talen1 show.jpg"
2019/07/12 10:46:37 DEBUG : rclone: Version "v1.48.0-066-g5433021e-beta" starting with parameters ["rclone" "copy" "robgs:Local-Pictures" "robgs-photos:upload" "-vv" "--tpslimit=1" "--include=2019/05/Emerson and Maddy Joke Talen1 show.jpg"]
2019/07/12 10:46:37 DEBUG : Using config file from "/root/.rclone.conf"
2019/07/12 10:46:37 INFO  : Starting HTTP transaction limiter: max 1 transactions/s with burst 1
2019/07/12 10:46:38 DEBUG : Google Photos path "upload": List: dir=""
2019/07/12 10:46:38 DEBUG : Google Photos path "upload": >List: err=directory not found
..
.. <snip>
..
2019/07/12 10:46:46 INFO  : Google Photos path "upload": Waiting for checks to finish
2019/07/12 10:46:46 INFO  : Google Photos path "upload": Waiting for transfers to finish
2019/07/12 10:46:46 DEBUG : Google Photos path "upload": Put: src=2019/05/Emerson and Maddy Joke Talen1 show.jpg
2019/07/12 10:46:46 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Update: src=2019/05/Emerson and Maddy Joke Talen1 show.jpg
2019/07/12 10:46:49 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: >Update: err=<nil>
2019/07/12 10:46:49 DEBUG : Google Photos path "upload": >Put: 
2019/07/12 10:46:49 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/12 10:46:49 DEBUG : Google Photos path "upload": List: dir="2019/05"
2019/07/12 10:46:49 DEBUG : Google Photos path "upload": >List: err=directory not found
2019/07/12 10:46:49 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Size: Failed to read metadata: object not found
2019/07/12 10:46:49 DEBUG : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/12 10:46:49 INFO  : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: Copied (new)

[ncw]

Apologies for the delay in responding, I've been on vacation.

@calisro @badzz group reply :-)

If I want to do a full backup of my gphotos and keep the albums structure (some photos are not in albums),should I do
rclone sync remote:media/all /mnt/backup/media

I'd recommend using media:by-month probably as the directory will get a bit big.

and
rclone sync remote:album /mnt/backup/album
And I will have twice my photos that are in albums , or there is a better way ?

There isn't a better way at the moment :-( Though you could just list the albums rclone lsf -R gphotos:albums which would give you enough info to reconstruct them - you could do this with a script and a bit of hardlinking...

PS: There seems to be a bug with album names containing / character . the name stops before the /

What rclone should have done is made a sub directory so if the album
is called photos from mum/dad you'll get a directory called
photos from mum with a directory called dad inside it.

Im a little confused. I understand that videos aren't being downloaded at full resolution but what about pictures? I took a look at the limitations and it only talks about EXIF data. To me it looks like the downloads of images are being scaled so I wanted to confirm. If so, that should be added to the limitations page.

Hi the bug clearly states that the sizes are different as well. My test showsthat indeed as well rclone doc does not say it, but it is indeed the case (plus EXIF location missing) https://issuetracker.google.com/issues/112096115 du -sh web.jpg api.jpg 2.4M web.jpg 616K api.jpg

In my tests I uploaded a picture then downloaded it and the only difference was the GPS info snipped from the EXIF data. So my re-download was a few bytes shorter only.

How did you do your upload? Could it have been one done at reduced resolution?

2019/07/11 13:48:23 ERROR : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: corrupted on transfer: sizes differ 85040 vs 67499

That looks like a bug.

This doesn't happen for all files does it?

Is it something special about that file? If so any chance you could send it to me either attach it here or email it to [email protected]

[vitobotta]

Hi! I read the doc but I am confused. If I just want to periodically back up my photos from Google Photos to a local disk, which commands should I use? Thanks

[badzz]

:

Apologies for the delay in responding, I've been on vacation.

Non apologies needed. Thanks.for answering

@calisro https://github.com/calisro @badzz https://github.com/badzz
group reply :-)

If I want to do a full backup of my gphotos and keep the albums structure
(some photos are not in albums),should I do
rclone sync remote:media/all /mnt/backup/media

I'd recommend using media:by-month probably as the directory will get a
bit big.

Ok thanks

and
rclone sync remote:album /mnt/backup/album
And I will have twice my photos that are in albums , or there is a better
way ?

There isn't a better way at the moment :-( Though you could just list the
albums rclone lsf -R gphotos:albums which would give you enough info to
reconstruct them - you could do this with a script and a bit of
hardlinking...

That's what I was starting to get to in my mind. Thanks

PS: There seems to be a bug with album names containing / character . the
name stops before the /

What rclone should have done is made a sub directory so if the album
is called photos from mum/dad you'll get a directory called
photos from mum with a directory called dad inside it.

I will check it next time. But it was the original name of my album, eg
:"holidays 15/07/2015"

Im a little confused. I understand that videos aren't being downloaded at
full resolution but what about pictures? I took a look at the limitations
and it only talks about EXIF data. To me it looks like the downloads of
images are being scaled so I wanted to confirm. If so, that should be added
to the limitations page.

Hi the bug clearly states that the sizes are different as well. My test
showsthat indeed as well rclone doc does not say it, but it is indeed the
case (plus EXIF location missing)
https://issuetracker.google.com/issues/112096115 du -sh web.jpg api.jpg
2.4M web.jpg 616K api.jpg

In my tests I uploaded a picture then downloaded it and the only
difference was the GPS info snipped from the EXIF data. So my re-download
was a few bytes shorter only.

How did you do your upload? Could it have been one done at reduced
resolution?

I am using "original format" from Google photos Android app and upload from
Google photos website on chrome pc.
My photos lost 100 of kbs when downloading from website vs API (rclone)

2019/07/11 13:48:23 ERROR : 2019/05/Emerson and Maddy Joke Talen1
show.jpg: corrupted on transfer: sizes differ 85040 vs 67499

That looks like a bug.

This doesn't happen for all files does it?

Is it something special about that file? If so any chance you could send
it to me either attach it here or email it to [email protected]

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=ABR5UYNLSVE2NNCT7RBVJATQBA34VA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2WBBFI#issuecomment-514592917,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABR5UYPKF74LO2JQ6PY56D3QBA34VANCNFSM4B4HRKVA
.

[badzz]

On Wed, Jul 24, 2019, 18:10 Vito Botta [e-mail] wrote:

Hi! I read the doc but I am confused. If I just want to periodically back
up my photos from Google Photos to a local disk, which commands should I
use? Thanks

rclone sync remote:media/by-month /mnt/backup/media

—

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/369?email_source=notifications&email_token=ABR5UYJR6ESGLX7OKY3MOIDQBBWGJA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2WUVPY#issuecomment-514673343,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABR5UYJZ64SWD422CA4SJITQBBWGJANCNFSM4B4HRKVA
.

[vitobotta]

On Wed, Jul 24, 2019, 18:10 Vito Botta @.*> wrote: Hi! I read the doc but I am confused. If I just want to periodically back up my photos from Google Photos to a local disk, which commands should I use? Thanks
rclone sync remote:media/by-month /mnt/backup/media —
…
You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#369?email_source=notifications&email_token=ABR5UYJR6ESGLX7OKY3MOIDQBBWGJA5CNFSM4B4HRKVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2WUVPY#issuecomment-514673343>, or mute the thread https://github.com/notifications/unsubscribe-auth/ABR5UYJZ64SWD422CA4SJITQBBWGJANCNFSM4B4HRKVA .

Hi, what happens if I run it multiple times, will it skip the files already downloaded or? Thanks!

[badzz]

.
>

Hi, what happens if I run it multiple times, will it skip the files
already downloaded or? Thanks!

Sure, yes

[vitobotta]

.
Hi, what happens if I run it multiple times, will it skip the files already downloaded or? Thanks!
Sure, yes

Wonderful, thanks a lot!

[gedl]

Where do the "magic" Assistant creations end up in? Is there a way to retrieve only those?

My use case would be to backup from local into google photos and then download the auto creations to somewhere. Not sure if into the same tree or somewhere else.

I plan to use Google Photos to view my photos - they are already backed up somewhere else too.

[calisro]

How did you do your upload? Could it have been one done at reduced resolution?

2019/07/11 13:48:23 ERROR : 2019/05/Emerson and Maddy Joke Talen1 show.jpg: corrupted on transfer: sizes differ 85040 vs 67499

That looks like a bug.

This doesn't happen for all files does it?

Is it something special about that file? If so any chance you could send it to me either attach it here or email it to [email protected]

@ncw It happens for the vast majority of the files. Ill send you one. But it's consistent across a lot of them. I was simply uploading to an album. (Doesn't matter if I create a new one or an existing)

This is an example:
rclone copy robgs:Local-Pictures robgs-photos:album -vv --tpslimit=1 --include="2019/05/Emerson and Maddy Joke Talen1 show.jpg"

I've also tried from a local drive rather than Gdrive to gdrive.

root@cloudc:~/2019/05# rclone copy . robgs-photos:album/ -vv --tpslimit=1 
2019/07/28 18:14:00 DEBUG : rclone: Version "v1.48.0-066-g5433021e-beta" starting with parameters ["rclone" "copy" "." "robgs-photos:album/" "-vv" "--tpslimit=1"]
2019/07/28 18:14:00 DEBUG : Using config file from "/root/.rclone.conf"
2019/07/28 18:14:00 INFO  : Starting HTTP transaction limiter: max 1 transactions/s with burst 1
2019/07/28 18:14:00 DEBUG : Google Photos path "album": List: dir=""
2019/07/28 18:14:01 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/28 18:14:01 DEBUG : Google Photos path "album": Put: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:01 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Update: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:01 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Update: err=can't upload files here
2019/07/28 18:14:01 DEBUG : Google Photos path "album": >Put: 
2019/07/28 18:14:01 ERROR : Emerson and Maddy Joke Talen1 show.jpg: Failed to copy: can't upload files here
2019/07/28 18:14:01 INFO  : Google Photos path "album": Waiting for checks to finish
2019/07/28 18:14:01 INFO  : Google Photos path "album": Waiting for transfers to finish
2019/07/28 18:14:01 ERROR : Attempt 1/3 failed with 2 errors and: can't upload files here
2019/07/28 18:14:01 DEBUG : Google Photos path "album": List: dir=""
2019/07/28 18:14:01 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/28 18:14:01 DEBUG : Google Photos path "album": Put: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:01 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Update: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:01 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Update: err=can't upload files here
2019/07/28 18:14:01 DEBUG : Google Photos path "album": >Put: 
2019/07/28 18:14:01 ERROR : Emerson and Maddy Joke Talen1 show.jpg: Failed to copy: can't upload files here
2019/07/28 18:14:01 INFO  : Google Photos path "album": Waiting for checks to finish
2019/07/28 18:14:01 INFO  : Google Photos path "album": Waiting for transfers to finish
2019/07/28 18:14:01 ERROR : Attempt 2/3 failed with 2 errors and: can't upload files here
2019/07/28 18:14:01 DEBUG : Google Photos path "album": List: dir=""
2019/07/28 18:14:01 DEBUG : Google Photos path "album": >List: err=<nil>
2019/07/28 18:14:01 INFO  : Google Photos path "album": Waiting for checks to finish
2019/07/28 18:14:01 INFO  : Google Photos path "album": Waiting for transfers to finish
2019/07/28 18:14:01 DEBUG : Google Photos path "album": Put: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:01 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Update: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:01 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Update: err=can't upload files here
2019/07/28 18:14:01 DEBUG : Google Photos path "album": >Put: 
2019/07/28 18:14:01 ERROR : Emerson and Maddy Joke Talen1 show.jpg: Failed to copy: can't upload files here
2019/07/28 18:14:01 ERROR : Attempt 3/3 failed with 2 errors and: can't upload files here
2019/07/28 18:14:01 Failed to copy with 2 errors: last error was: can't upload files here
root@cloudc:~/2019/05# rclone copy . robgs-photos:album/ad -vv --tpslimit=1 
2019/07/28 18:14:07 DEBUG : rclone: Version "v1.48.0-066-g5433021e-beta" starting with parameters ["rclone" "copy" "." "robgs-photos:album/ad" "-vv" "--tpslimit=1"]
2019/07/28 18:14:07 DEBUG : Using config file from "/root/.rclone.conf"
2019/07/28 18:14:07 INFO  : Starting HTTP transaction limiter: max 1 transactions/s with burst 1
2019/07/28 18:14:07 DEBUG : Google Photos path "album/ad": List: dir=""
2019/07/28 18:14:07 DEBUG : Google Photos path "album/ad": >List: err=directory not found
2019/07/28 18:14:07 DEBUG : Google Photos path "album/ad": Put: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:07 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Update: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:07 INFO  : Google Photos path "album/ad": Waiting for checks to finish
2019/07/28 18:14:07 INFO  : Google Photos path "album/ad": Waiting for transfers to finish
2019/07/28 18:14:12 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Update: err=<nil>
2019/07/28 18:14:12 DEBUG : Google Photos path "album/ad": >Put: 
2019/07/28 18:14:12 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/28 18:14:12 DEBUG : Google Photos path "album/ad": List: dir=""
2019/07/28 18:14:12 DEBUG : Google Photos path "album/ad": >List: err=<nil>
2019/07/28 18:14:13 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/28 18:14:13 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/28 18:14:13 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/28 18:14:13 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Size: 
2019/07/28 18:14:13 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: >Size: 
2019/07/28 18:14:13 ERROR : Emerson and Maddy Joke Talen1 show.jpg: corrupted on transfer: sizes differ 85040 vs 67499
2019/07/28 18:14:13 INFO  : Emerson and Maddy Joke Talen1 show.jpg: Removing failed copy
2019/07/28 18:14:15 ERROR : Attempt 1/3 failed with 2 errors and: corrupted on transfer: sizes differ 85040 vs 67499
2019/07/28 18:14:15 DEBUG : Google Photos path "album/ad": List: dir=""
2019/07/28 18:14:16 DEBUG : Google Photos path "album/ad": >List: err=<nil>
2019/07/28 18:14:16 INFO  : Google Photos path "album/ad": Waiting for checks to finish
2019/07/28 18:14:16 INFO  : Google Photos path "album/ad": Waiting for transfers to finish
2019/07/28 18:14:16 DEBUG : Google Photos path "album/ad": Put: src=Emerson and Maddy Joke Talen1 show.jpg
2019/07/28 18:14:16 DEBUG : Emerson and Maddy Joke Talen1 show.jpg: Update: src=Emerson and Maddy Joke Talen1 show.jpg

[andreask1]

@ncw I just did some testing with the latest beta version (v1.48.0-103-g6a3e3013-beta) using sync remote:media/by-month and ran into the following issues:

• No photos before Jan 1st, 2000 were synced
• No archived photos were synced (the old Drive sync included them in their original folder)

[Conroman16]

This is looking pretty good. That said, I did some testing on the current beta (rclone v1.48.0-131-ge502be47-beta) and I too see that my archived photos are not showing up. I see a reference to it here however it doesn't appear that param has been implemented yet. Are there any plans to do so at this time? The ability to work with the archived media is a must in my use case.

[lssong99]

IT's been two years since I opened this thread... Thank you guys just make this work in time for GDrive and GPhoto's desync. (Nobody could predict then...)

The beta version still have some quarks to remove but really a good progress...

Looking at ls output, it seems the duplicated files are getting some random filename tails to make them unique... Really missed the dedupe command here.... Before Google cut tie between GDrive and GPhoto, all I need to do is dedupe the GDrive MyPhoto folder and duplicates are eliminated... Not sure if it's possible with rclone in GPhoto... (Tried with latest beta version but doesn't seems to work due to error....)

2019/08/10 19:27:10 Failed to dedupe with 20 errors: last error was: google photos API only implements removing files from albums

Any way to do this from non-album way...?

Edit even album way no work... The command doesn't return anything by exit after quite a bit time without any output...

-1 PANO_20190714_181128.vr {ANZQEuahFdXObFCdxl7GbrcwKbzxKWkxbVAf80dfadsfasdflka13ai1-wcBwa2fe3KbKrNp18k45g_XHOcfeGspQ1Zw}.jpg -1 PANO_20190714_181128.vr {ANZQEubpau27dgeUZf;lkasd;flkasdflj-F7r3dWMdaxlwtWQky-vNyjvEuVy83xrk5dLuJ4rMZE6g2umC2OkaIUNaWdtqOIE7_VOJYA}.jpg -1 PANO_20190714_200502.vr {ANZQEua-JRSn8zMFbMpp9Src6T95HMSPcpmmX6AHwFJHPgTiOGkfa;lskdf;laskflkjasefNzsov9mlzwVM0Jp8sii32DjHC4g}.jpg -1 PANO_20190714_200502.vr {ANZQEubh0LFwOem4iuM8YJayhjafslkjlskadjf;laksjf;alsfCMEuL2s09ZaUSNlQSn_PZgWmCGb1muYSZ4b-Zg}.jpg

[ncw]

IT's been two years since I opened this thread... Thank you guys just make this work in time for GDrive and GPhoto's desync. (Nobody could predict then...)

The beta version still have some quarks to remove but really a good progress...

:-)

I've almost got it approved by google, a few more hurdles to jump!

Looking at ls output, it seems the duplicated files are getting some random filename tails to make them unique... Really missed the dedupe command here.... Before Google cut tie between GDrive and GPhoto, all I need to do is dedupe the GDrive MyPhoto folder and duplicates are eliminated... Not sure if it's possible with rclone in GPhoto... (Tried with latest beta version but doesn't seems to work due to error....)

2019/08/10 19:27:10 Failed to dedupe with 20 errors: last error was: google photos API only implements removing files from albums

Unfortunately the API doesn't support renaming media items. Yes the API is really limited which is annoying :-(

So rclone can't support dedupe.

Any way to do this from non-album way...?

The API pretty much limits you to using albums if you want to read and write. You can read the media items but you can't rename/delete them.

-1 PANO_20190714_181128.vr {ANZQEuahFdXObFCdxl7GbrcwKbzxKWkxbVAf80dfadsfasdflka13ai1-wcBwa2fe3KbKrNp18k45g_XHOcfeGspQ1Zw}.jpg -1 PANO_20190714_181128.vr {ANZQEubpau27dgeUZf;lkasd;flkasdflj-F7r3dWMdaxlwtWQky-vNyjvEuVy83xrk5dLuJ4rMZE6g2umC2OkaIUNaWdtqOIE7_VOJYA}.jpg -1 PANO_20190714_200502.vr {ANZQEua-JRSn8zMFbMpp9Src6T95HMSPcpmmX6AHwFJHPgTiOGkfa;lskdf;laskflkjasefNzsov9mlzwVM0Jp8sii32DjHC4g}.jpg -1 PANO_20190714_200502.vr {ANZQEubh0LFwOem4iuM8YJayhjafslkjlskadjf;laksjf;alsfCMEuL2s09ZaUSNlQSn_PZgWmCGb1muYSZ4b-Zg}.jpg

Indeed!

The {xxxx} are the unique IDs of the media items.

[lssong99]

IT's been two years since I opened this thread... Thank you guys just make this work in time for GDrive and GPhoto's desync. (Nobody could predict then...)
The beta version still have some quarks to remove but really a good progress...

:-)

I've almost got it approved by google, a few more hurdles to jump!

Looking at ls output, it seems the duplicated files are getting some random filename tails to make them unique... Really missed the dedupe command here.... Before Google cut tie between GDrive and GPhoto, all I need to do is dedupe the GDrive MyPhoto folder and duplicates are eliminated... Not sure if it's possible with rclone in GPhoto... (Tried with latest beta version but doesn't seems to work due to error....)
2019/08/10 19:27:10 Failed to dedupe with 20 errors: last error was: google photos API only implements removing files from albums

Unfortunately the API doesn't support renaming media items. Yes the API is really limited which is annoying :-(

So rclone can't support dedupe.

Any way to do this from non-album way...?

The API pretty much limits you to using albums if you want to read and write. You can read the media items but you can't rename/delete them.

-1 PANO_20190714_181128.vr {ANZQEuahFdXObFCdxl7GbrcwKbzxKWkxbVAf80dfadsfasdflka13ai1-wcBwa2fe3KbKrNp18k45g_XHOcfeGspQ1Zw}.jpg -1 PANO_20190714_181128.vr {ANZQEubpau27dgeUZf;lkasd;flkasdflj-F7r3dWMdaxlwtWQky-vNyjvEuVy83xrk5dLuJ4rMZE6g2umC2OkaIUNaWdtqOIE7_VOJYA}.jpg -1 PANO_20190714_200502.vr {ANZQEua-JRSn8zMFbMpp9Src6T95HMSPcpmmX6AHwFJHPgTiOGkfa;lskdf;laskflkjasefNzsov9mlzwVM0Jp8sii32DjHC4g}.jpg -1 PANO_20190714_200502.vr {ANZQEubh0LFwOem4iuM8YJayhjafslkjlskadjf;laksjf;alsfCMEuL2s09ZaUSNlQSn_PZgWmCGb1muYSZ4b-Zg}.jpg

Indeed!

The {xxxx} are the unique IDs of the media items.

However, it seems is lsd a album, nothing shows up....

[badzz]

@lssong99 : normal lsd list folders. There isn't any in an album. Try lsf

[lssong99]

@lssong99 : normal lsd list folders. There isn't any in an album. Try lsf

Yes! Got it! How did I miss such a simple thing!?

It seems the size of photo are always -1 (no matter lsf or ls)... Anyway to get the size?

If I could get the size or the upload date, then it's possible to write an external script to simulate --dedupe-mode newest/largest etc.....

Also, by looking at GPhoto API page, it states
Note that you can only remove media items that your application has added to an album or that have been created in an album as part of an upload. For albums that are shared, you can only remove items added by other collaborators if you are acting on behalf of the owner of the album.

Does this means we rclone will only manage (delete) photos uploaded by rclone?

[ncw]

It seems the size of photo are always -1 (no matter lsf or ls)... Anyway to get the size?

You can use --gphotos-read-size. However it will really slow things down and I don't recommend it.

If I could get the size or the upload date, then it's possible to write an external script to simulate --dedupe-mode newest/largest etc.....

The dates on the photos are the EXIF created dates and they should be reliable. Gphotos doesn't support reading the upload date.

Also, by looking at GPhoto API page, it states
Note that you can only remove media items that your application has added to an album or that have been created in an album as part of an upload. For albums that are shared, you can only remove items added by other collaborators if you are acting on behalf of the owner of the album.

Does this means we rclone will only manage (delete) photos uploaded by rclone?

That is correct. Check out the limitations section in the docs: https://tip.rclone.org/googlephotos/#limitations

[ncw]

I'm going to close this issue since the backend is done.

Please make new issues with problems - thanks :-)

[Kreijstal]

This work is amazing

[AngelaDMerkel]

This is looking pretty good. That said, I did some testing on the current beta (rclone v1.48.0-131-ge502be47-beta) and I too see that my archived photos are not showing up. I see a reference to it here however it doesn't appear that param has been implemented yet. Are there any plans to do so at this time? The ability to work with the archived media is a must in my use case.

I realise this is _very_ belated, but I wanted to know if this has been resolved. From what I have seen using rclone mount there still isn't any good way to access the photos that were archived in Photos. I apologise if I've missed something.

[ncw]

I realise this is _very_ belated, but I wanted to know if this has been resolved. From what I have seen using rclone mount there still isn't any good way to access the photos that were archived in Photos. I apologise if I've missed something.

No this isn't resolved. The backend needs an archived parameter or --gphotos-archived flag. This would be straightforward to add if you wanted to help @AngelaDMerkel ? I'd be happy to talk you through it. The first thing would be to please make a new issue on github about it.

[AngelaDMerkel]

No this isn't resolved. The backend needs an archived parameter or --gphotos-archived flag. This would be straightforward to add if you wanted to help @AngelaDMerkel ? I'd be happy to talk you through it. The first thing would be to please make a new issue on github about it.

I'm more that happy to give it a go. I've done as you said created a new issue.