Rclone: Are we safe? Amazon Cloud Drive

[scriptzteam]

I mean could be there the same scenarion that they disable rclone app from amazon? or does the rclone handle it other way than acd_cli did?

ACD_CLI weird:

https://github.com/yadayada/acd_cli/pull/562 - "I created this pull request only to ask what happend to acd_cli's issues page?! It just vanished! "

[ncw]

It is always a possibility. However I've had various email conversations with the amazon developers in the past and I don't think they would do that without a dialogue first.

[ncw]

I was distressed to see on that thread a discussion of how to use rclone's keys to restore service. Please don't do this - it will likely lead to rclone being banned in the same way.

[mizzuri]

So, apparently there's a workaround for acd_cli using rclone's application id and secret. Won't this cause a negative effect on rclone?

Edit: Link here.
Edit 2: Now that stuff on ACD is encrypted using rclone, I'm worried that we won't be able to retrieve and decrypt the backup. :(

[ncw]

Edit 2: Now that stuff on ACD is encrypted using rclone, I'm worried that we won't be able to retrieve and decrypt the backup. :(

Provided you can retrieve the files somehow, you can decrypt them with rclone locally very easily.

[mizzuri]

Provided you can retrieve the files somehow, you can decrypt them with rclone locally very easily.

You are right. But it would still require downloading the whole directory just to get one file, right?
Because we will no longer be able to rclone lsl remote-crypted:directory

Nevertheless, it's still a peace of mind that there's still a way.
Thank you.

[fuyar]

Do you have a clue of why acd_cli key has been forbidden ?

API hammer or ToS issues or that thing about being able to access other people drive ?

[ncw]

You are right. But it would still require downloading the whole directory just to get one file, right?
Because we will no longer be able to rclone lsl remote-crypted:directory

Assuming that there existed another tool that could mount acd, then you could set it up so the above would work just fine.

[scriptzteam]

Just an update about acd_cli and amazon reply:
https://github.com/yadayada/acd_cli/pull/562#issuecomment-301816928

Thank you for providing more information.

We investigated the security issue you reported regarding Amazon Drive (reference HGXXXXXXX).

Our investigation did not reveal any incorrect handling of session tokens in our systems. However, we also reviewed the source code of a third-party application used to authenticate to Amazon Drive (https://tensile-runway-92512.appspot.com/src) and think that there is a concurrency issue present in their code which could result in a customer receiving incorrect authentication tokens.

To ensure our customers safety, we have disabled access to Amazon Drive from the third-party application "acd_cli_oa”.

We appreciate you reporting this security issue.

[mizzuri]

@scriptzteam .. So.. Does it mean that another (abusing?) app was using acd_cli's security tokens, and thus acd_cli was banned because of it? I'm a bit lost here.

[Rufflewind]

@unnfav No, https://tensile-runway-92512.appspot.com/src is acd_cli’s default authentication server. Amazon’s investigation indicates that it’s buggy and can sometimes leak tokens to the wrong user.

[mizzuri]

@Rufflewind .. Thanks for the clarification.

[ncw]

@Rufflewind wrote

No, https://tensile-runway-92512.appspot.com/src is acd_cli’s default authentication server. Amazon’s investigation indicates that it’s buggy and can sometimes leak tokens to the wrong user.

Just so everyone is clear, rclone doesn't use an external authentication server - it is all done on your computer. That is why the signup for Amazon Drive (and all the oauth providers like Google etc) is a little awkward.

I didn't want to have the responsibility of other people's credentials going through my server so I deliberately crafted rclone so that wasn't necessary. Your credentials never leave your computer!

[scriptzteam]

when i firstly installed acd_cli and there was that 3-rd party auth needed i know it was unsafe, thus i never used, i know you can trust, BUT WH THE FUCK I SHOULD TRUST SOMEONE SH*T CODE thats hosted on apppot.com lol, he could cache everything ;) thats why i preffered rclone, as @ncw said and THIS MATTER TO ME -> "Just so everyone is clear, rclone doesn't use an external authentication server - it is all done on your computer. That is why the signup for Amazon Drive (and all the oauth providers like Google etc) is a little awkward.

I didn't want to have the responsibility of other people's credentials going through my server so I deliberately crafted rclone so that wasn't necessary. Your credentials never leave your computer!"

[boblatino]

I think they also removed the rclone key:

2017/05/18 06:43:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/05/18 06:43:17 DEBUG : HTTP REQUEST (req 0x10fea400)
2017/05/18 06:43:17 DEBUG : GET /drive/v1/account/endpoint HTTP/1.1
Host: drive.amazonaws.com
User-Agent: rclone/v1.36
Authorization: XXXX
Accept-Encoding: gzip

2017/05/18 06:43:17 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/05/18 06:43:17 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/05/18 06:43:17 DEBUG : HTTP RESPONSE (req 0x10fea400)
2017/05/18 06:43:17 DEBUG : HTTP/1.1 429 Too Many Requests
Cache-Control: no-store
Connection: keep-alive
Content-Type: application/json
Date: Thu, 18 May 2017 06:43:17 GMT
Pragma: no-cache
Server: Server
Vary: User-Agent
X-Amzn-Errortype: TooManyRequests:http://internal.amazon.com/coral/com.amazon.clouddrive.service/
X-Amzn-Requestid: xxxxx

I have contacted amazon and they told me that rclone was disabled today :(

[mizzuri]

Sadly, I can also confirm that rclone is no longer working now. :((

[l3uddz]

If it was disabled, why is it still in Third Party Apps in Amazon Drive account settings? acd_cli was removed from there for me without me doing anything, rclone is still there. Seems to suggest some kind of api limit was reached perhaps?

[jdrydn]

Same here. 429s, 429 everywhere. It's sad because after acd_cli's downfall (and subsequent unusual silence) I'd decided to move to B2, except now that's going to be extremely difficult given rclone was my choice of tool to move between the two services! 😢

[ncw]

I think maybe rclone's quota has been reduced. I'm contacting Amazon to find out what is happening.

[geekcroft]

Good luck ncw 👍

[PhilipFraDIKU]

Having the "429 Too Many Requests" posts as well.

When typing # rclone lsd ACD:ACD_Encrypted in terminal I get:

2017/05/18 11:24:30 Failed to create file system for "ACD:ACD_Encrypted": failed to get endpoints: HTTP code 429: "429 Too Many Requests": response body: "{\"message\":\"Rate exceeded\"}"

[Julz2k]

Problems started for me two days ago, files could not be uploaded correctly (error 400) even after 10 tries and uploading over 800GB for the bin.

Since this morning I also face the 429 too many requests message. Sad 👎

[geekcroft]

User on the Rclone forum reporting that Amazon Support have told them that its a ban on rclone and its permanent :/

Shall wait to see what @ncw says after talking with Amazon :(

[Em31Et]

Is it because rclone don't have cache and spam amazon api for processing?

[PhilipFraDIKU]

"banned"?!?

I sure hope not, but let's see what @ncw can report back with.

[jdrydn]

@Em31Et It'll likely be because all the acd_cli users moved over to rclone when the former was banned from the ACD API, and sounds like rclone is going in a similar direction (although thank you @ncw for not being radio-silent!)

[JackDash]

Just wait for ncw's answer. It wouldn't be the first time that Amazon support gave wrong info and certainty not the first time someone would make something up.

[Em31Et]

@jdrydn
acd_cli users is the reason, but I think the main reasons is because of rclone's implementation.
rclone do not have cache and keep on retrying the api calls.

[gdomod]

its possible to make a own rclone app and register with acd ?
i thought there was too much hits from one app.

[Em31Et]

@gdomod
No, amazon do not accept new apply anymore.

[gdomod]

and now how i get my data back ? there is no more linux client avail ?

[lorenzoferrarajr]

Please let's all wait for info from @ncw

[vampywiz17]

and now how i get my data back ? there is no more linux client avail ?

https://www.cloudhq.net/

What do you think, it safe?

[jdrydn]

@lorenzoferrarajr has it right - let's wait for @ncw to get back to us with the 411

@gdomod @vampywiz17 From what I can tell there's nothing else for linux, but plenty for Win/Mac to do a migration to another service (but depending on speeds it might take a while!). I do hope I'm wrong however, because that's a major PITA!

[geekcroft]

@jdrydn I would hope that somebody writes something up for ACD - then again I've read Amazon aren't accepting new Apps at this time so presume that would cause issues :(

[vampywiz17]

@gdomod it is a service not a app, but i don't want to add my personal account to a third party... It is my question, this services, that sync two cloud drive is safe or not?

[Miladiir]

@vampywiz17 can you stop spamming this issue. this is not a forum and your discussion is not related.

[drvdijk]

While I do appreciate the design decision to have rclone do the oauth dance the way it does currently, that does mean everybody using the app effectively needs access to the application key and secret (right there in amazonclouddrive.go). And as the acd_cli guys went on using rclone's credentials, this likely (to be confirmed) caused rclone to get banned too, realizing ncw's fear.

Can we somehow come up with a way of keeping user credentials off a centralized server, while at the same time not sharing the application credentials as they currently are?

[Miladiir]

@drvdijk Remove keys from Git History and current files. Use new keys and dont check them in. Obfuscate binaries.
Maintainers can get keys to test.

This should dramatically limit the number of fraudulent key uses.

[gdomod]

rclone seems to be banned. good bye acd

[Miladiir]

Can we please stop speculating and wait for a reply.

[vampywiz17]

Can we please stop speculating and wait for a reply.

We got reply with amazon. You able to read it on forum.

[RhvsW]

Again, that is from a front line support agent at amazon who might as well be talking about faeries. wait for @ncw's response.

[Miladiir]

@vampywiz17 .@ncw Did not get a reply. For now, do not believe anything that is said anywhere, unless Nick himself posted it on a official channel (e.g. right here).

Anything else is speculation.

[Julz2k]

We also got a reply from a German supporter, which answeres are correct most of the time, cause as a german you still have access to a wide range of native 1. and 2. level supporters on the fly (main business time)
And the supporter was german Edith Schwarz

[Webreaper]

Please stop speculating. We don't have any confirmation until @ncw posts back to this thread. If you have comments to make, please make them on the forum. https://forum.rclone.org/t/acd-429-too-many-requests/1792/307

[ajkis]

Just finished call with customer support and they told me that from today rclone will not be supported anymore.

[Aldigator]

An employee of Amazon Germany just told me on the phone that they are making changes to their services and infrastructure and that rclone probably needs to be updated accordingly.

EDIT:
Scratch that, just got on their support chat to clarify after reading some more comments and the bastards actually terminated support for rclone! Apparently some office-sitter at Amazon is under the impression that informing your own support staff (let alone PAYING CUSTOMERS) about such minor changes as dropping support for an entire goddamn platform is somehow optional.

[crimsdings]

Sie nutzten für den Zugriff auf Amazon Drive ein "Rclone" von 2P. Rclone wurde jedoch ab heute deaktiviert. Daher ist kein Zugang zu Amazon Drive über Rclone mehr möglich. Es wird in Zukunft auch nicht möglich sein den Zugang über Rclone zu aktivieren.

Same answer - I specifically asked if they plan to fix it in the future and they told me no they dont

[Webreaper]

Guys, seriously. Please stop spamming the issue and use the forum.

[rpgdev]

@Webreaper I understand your inclination for make believe right now but those customer support exchanges are useful and pertinent to the issue, _not speculation by any means_: these are official company reps. Several users have reported similar exchanges already so they are credible. Now, is the ban written in stone? I hope not but in the meantime those tech support comments are informative about the current situation regarding this issue.

[Webreaper]

It's not about my inclination to 'make believe'. It's about the fact that we're all well aware of how misleading commentary from cust serv reps can be, and that situations change fast (this only happened a few hours ago). Given that ncw regularly communicates with Amazon and has done through the development of rclone, nothing is certain yet. But my key point is, whatever the outcome here, or the facts, repeatedly posting the same stuff to the github issue is pointless and just generates spam - when that's exactly what the forum is for.

[rpgdev]

@Webreaper it's common for people to post similar reports about the same thing on any given issue in order to confirm it so why should this issue be any different? It's expected that that will happen here too. Tech support comments are only inaccurate (not misleading: they are all saying the same thing right now) if the situation changes in the future, in the meantime and until we get further answers they are giving us _the company line_ which is useful and informative for this issue at this time.

[Webreaper]

Sure, knock yourself out and keep posting speculation and repeated/potentially inaccurate information while we wait for something concrete from ncw. Fine by me, I'll just unsubscribe from the issue. :)

[bugs181]

Title of this bug should be updated. I only found it through Google search. I'm a brand new user of rclone and was having the issue 429 Too Many Requests. I had no idea what this error meant and it appears to be related to ACD disabling the rclone app, is that correct? Reading all of the replies - it also appears that we are waiting for an official response from the developer/maintainer of rclone? I've subscribed to this issue so please keep us updated.

[scriptzteam]

"Title of this bug should be updated" ?? Why? i created it 3 days before, amazon "disabled" rclone just today, my subject WAS about asking if WE ARE SAFE? SO ARE WE NOW? N.O.T. :)

//meanwhile you all are waiting i just got business GDRIVE and enjoying it :D

[bugs181]

The title should be more self-explanatory. You need to properly adhere to Github's bugs rules. It's not a forum. The rule is "I am having XYZ problem" If the bug post is more concern related then something like "Could Amazon disable the rclone app?" should suffice.

[scriptzteam]

@bugs181 rule your life with your rules lol xD i have own rules, will not listen yours, peace ;)

[bugs181]

Don't play by other's rules and YOU lose that privilege, not me.

[Webreaper]

It's not a forum? Well, blow me down - with the anecdotal comments about customer service conversations and arguments about issue titles, you could have fooled me. ;-D

[scriptzteam]

Blah blah, i asked "IF WE ARE SAFE" nothing else! undertstand me?

[rpgdev]

@Webreaper should've unsubscribed before asking users to stop posting useful information. And I do hope you're right and it _ends up_ being inaccurate information but in the meantime it is not: it's the company line and pertinent to this issue.

[rueduphp]

The problem is Amazon. as long as they block to whitelist secure profiles, there will be no lasting solution for linux users.

[Miladiir]

FFS stop spamming and fighting, lets wait patiently.

On the issue: I'm glad that they made that change during my trial. I will make damn sure to stay away from closed source potentially datamining cloud storage for now. Hello encrypted Google Business!

[bugs181]

Does Google offer an unlimited plan? If not, what are other options that I can use? Looking to go with a yearly plan but will also consider a monthly fee. I have about 10TB I need to backup from an Unraid box and that storage capacity is likely to grow. Looking for offsite backup.

[rueduphp]

Not unlimited but interesting pricing https://www.backblaze.com

[henryford]

@bugs181 A little complicated, but actually a good deal - 120$/year:
https://www.reddit.com/r/DataHoarder/comments/47cln1/til_you_can_buy_an_unlimited_google_drive_account/d4i93zf/

[jdrydn]

@Miladiir I'm out of my trial but I'm going to chase for a refund - this is shocking!

@henryford Thanks for sharing that!

[zawlin]

hmm..it seems not that hard to find alternate client_id and secret keys in other software(there's quite a few ya? i have randomly tried 2 and it's very easy to get the required info) that can access acd. this information is usually left in plaintext in compiled code(or you can just inspect network packets to obtain it). You only need to know 3 fields client_id,secret_key and redirect_uri and then manually execute the login procedure to get auth token and rclone will be in business again. this is probably just a temp measure to help move data before this approach is blocked again, though effectively blocking this method of "borrowing" other software access info may not be that easy. Hope it helps.

[henryford]

@zawlin I'd advise against that. At this current point I'd wager that they disabled access for rclone precisely because acd_cli clients used its credentials to connect. I think it still says something that rclone is still showing up on the access page on acd.

[Krandor1]

Right The fact they haven't flat out told ncw straigth no and it is still on the access page are good signs. No need to do anything to antagonize them.

[ajkis]

I'd advise against that. At this current point I'd wager that they disabled access for rclone precisely because acd_cli clients used its credentials to connect

Yea I agree but we all know acd_cli guys will rip other app keys until amazon bans them all :/

[bgemmill]

@ajkis I'm one of the "acd_cli guys" and we care about the outage just as much as you do. The difference in time between our outage and yours was short enough that blaming things on malice seems hasty.

On our side of the fence we have a few users who kindly donated white-listed security profiles, but we're waiting to see what the official response is in case they get banned straight away.

We're waiting for word from @yadayada and you should wait for @ncw, and when one gets word we can all benefit.

In the mean time the only real protest is to set your acd subscription to not renew and hope they notice.

[itsrainingben]

chiming in with a 429 error code when listing directories via rclone.

i just got off with ACD help desk, she told me Amazon banned rclone. in asking why Amazon wouldn't notify users or developers, her response from 'leadership' was that they actually weren't clear who (ACD or rclone) killed it.

just gonna sit on my hands and monitor this thread for an update from @ncw

[techknowlogick]

@itsrainingben: @ncw has given an update via twitter https://twitter.com/njcw/status/865319897580097537

[itsrainingben]

@techknowlogick appreciate that

[gordan-bobic]

Several people have reported getting refunds on their ACD subscriptions, and I can confirm I just got mine. That seems to imply that ACD is burned for rclone. It may be an idea to start looking at pruning the bug list of ACD specific issues/features, and looking at removing the ACD back-end from rclone.

[rpgdev]

@gordan-bobic Amazon almost never denies a refund so this is hardly indicative of anything. @ncw hasn't received an answer from ACD team yet so removing anything ACD related from rclone is premature. His most recent tweet related to the issue: https://twitter.com/njcw/status/865583044870189056

[nextime]

I don't know if it has been already mentioned, and i'm not in any relation with it, i'm just a customer, but for me the alternative service that i'm using to sobstitute my ACD storage i was using with rclone until few days ago is Hubic, the OVH cloud storage: 50 eur/year for 10 terabyte of storage and it works great with rclone.

[Thinkscape]

Hubic is speed capped to just 10mbps.

It's in the fine print.
On Mon, 22 May 2017 at 9:51 am, Franco (nextime) Lanza <
[email protected]> wrote:

I don't know if it has been already mentioned, and i'm not in any relation
with it, i'm just a customer, but for me the alternative service that i'm
using to sobstitute my ACD storage i was using with rclone until few days
ago is Hubic, the OVH cloud storage: 50 eur/year for 10 terabyte of storage
and it works great with rclone.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/ncw/rclone/issues/1409#issuecomment-302971752, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAQgwAx2hMcY8kVA6BtHI4Su9muMP_Aeks5r8M3jgaJpZM4Nbiqz
.

>

Artur Bodera
[email protected]
http://thinkscape.pro

[jdrydn]

@Thinkscape is correct, I've just checked the Hubic "_General conditions of use_":

The bandwidth is limited to 10 Mbit/s upstream and downstream. The connection speed also depends on the quality of the Customer's internet connection.

And

OVH cannot guarantee that files exceeding 5 GB will be deposited. In the event of needing to put a file of this size online, OVH asks the Customer to contact the technical support team in order to find out the most effective method of carrying out this operation.

🤦‍♂️

[AiMAnsarie]

So is there a cli on Ubuntu other than acd_cli and rclone?

[Tvax]

What should we use now?

[gordan-bobic]

Google Drive and HubiC seems to be what most people I know are migrating to.
Last I checked there were no fuse-mountable alternatives to acd_cli and rclone.

[AiMAnsarie]

Any change in situation?

[vampywiz17]

Google Drive and HubiC seems to be what most people I know are migrating to.
Last I checked there were no fuse-mountable alternatives to acd_cli and rclone.

I thinking HubiC... It is fast?

[michael-k]

@AiMAnsarie https://forum.rclone.org/t/rclone-has-been-banned-from-amazon-drive/2314

Update 2017-05-30

I've finally heard back from Amazon about my request for new developer credentials (after 10 days!) to build an auth server

We have ended the Amazon Drive API and SDK invitation period to focus on enabling new customer experiences with current developers. Should this change, we will provide an update on our developer website: https://developer.amazon.com/amazon-drive140

So I guess that is a no :frowning:

[toddfries]

HubiC is sluggish and sometimes has auth issues randomly for no reason. I am currently paying < $5/mo for its 10T service, and treat it as a 'once I get data there, it is good' service not a 'I will wet my pants if I cannot get data uploaded 3s after I want it there' service. I'd really prefer Amazon Drive (which is the same storage mechanism as Amazon Photo Storage, free unlimited photos with amazon prime), but of course the current fiasco of rclone being banned will have to be sorted. If I can help I hope to figure out how. In the mean time, it would be nice to consolidate all the duplicate issues .. #1429, #1420, #1417, #1415 ...

[gordan-bobic]

@toddfries from what I can tell, Amazon revoked app keys from many apps, not just rclone and acd_cli, and from what I hear, they are not going to be issuing any new app keys. So I think ACD is gone for good as a useful cloud storage service.

I suspect that they underestimated both the popularity of the service and the amount of space abusers of the "unlimited" storage capacity were using.

[toddfries]

Should be noted that AWS still works (for now???)

[gordan-bobic]

@toddfries What exactly do you mean? AWS != ACD.

[mc40638]

AWS-Amazon Web Services which spans the gamut from individual users to very large enterprise clients. ACD-Amazon Cloud Drive (much like Google Drive).

ACD is built on the AWS framework but is a separate entity. AWS access is different and will depend on each organization using a published and accepted API. I would think there are commercial clients as well.

ACD has a limited number of acceptable clients, and the last time I talked to Amazon, eventually only clients publish by amazon will be available. This may not be true; just what they told me.

[mizzuri]

@gordan-bobic

What exactly do you mean? AWS != ACD.

I think @toddfries meant Amazon S3.

[ncw]

I'm going to close this issue now - for the resolution see https://forum.rclone.org/t/rclone-has-been-banned-from-amazon-drive/

[marvinpinto]

@ncw Hi! Is the resolution essentially "rclone can no-longer be used with Amazon Cloud Drive?"

(That thread was quite long and Discourse does not make it very easy to search or navigate)

[rpgdev]

@marvinpinto there is an update in the first post, you don't have to scroll the entire thread.