Azure-storage-azcopy: Issue using OAuth for blob copy operations

[nofunatall]

Which version of the AzCopy was used?

azcopy version 10.3.3

Which platform are you using? (ex: Windows, Mac, Linux)

Linux

What command did you run?

azcopy login
azcopy copy "https://${ACCOUNT}.blob.core.windows.net/${CONTAINER}/${BLOB}" "./test.jpg

What problem was encountered?

failed to perform copy command due to error: no SAS token or OAuth token is present and the resource is not public

[JohnRusk]

We've had issues on some distros with use of the session keyring, which is where azcopy login stores the auth token.

This comment on an older issue says how to check whether you are affected by the problem (with keyctl show ) and also contains a workaround that works in at least some of the affected cases. https://github.com/Azure/azure-storage-azcopy/issues/452#issuecomment-503812803

Please let us know if that comment helps, and if the workaround posted there works for you.

[nofunatall]

Session Keyring
281847720 --alswrv 1000 65534 keyring: _uid_ses.1000
515177348 --alswrv 1000 65534 _ keyring: _uid.1000

EDIT: Not meant to run with sudo

[JohnRusk]

Is that the output from keyctl show? If so, looks like you have the same problem docmented in that therad - since when it works there's a very clear indication that there's an Azcopy thing there. (I forget the name that's used, but its clearly AzCopy related). Have you tried the workaround in that thread?

[nofunatall]

I've looked at the thread and manually switching key session and running login won't work as I am integrating some scripts that are automated so I'm not sure how the login would work in an automated sense.

I would use SAS tokens generated from AZ CLI but that tool seems to generate malformed tokens or tokens that specifically only work with AZ CLI.

Azcopy is not able to use storage account keys is that right?
That would work for me as a workaround for both issues in this case.

Here are some details on my running distro.

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 19.10
Release: 19.10
Codename: eoan

Linux t460p 5.3.0-26-generic #28-Ubuntu SMP Wed Dec 18 05:37:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[nofunatall]

Confirming that the workaround mentioned in #452 does work as described but for my use case here is not a suitable work around.

Here is the output from keyctl after switching session and running azcopy login again.

Session Keyring
184649994 --alswrv 1000 1000 keyring: azcopy
722270074 --alswrv 1000 1000 _ user: AzCopyOAuthTokenCache

[JohnRusk]

Thanks for the info. That might be helpful to us in getting a better understanding of the issues with keyctl.

AzCopy v10 deliberately doesn't support accessing blob storage with account keys, because we want to encourage people to use more secure options.

Are you currently using the manual type login (the one where you put a code into a browser)? For automated scripts, you have two other options which may in fact be better than that. If running in an Azure VM, you can use Managed Service Identity which allows AzCopy to login with the identity of the VM, with no secrets to manage at all. If running outside Azure, you can user Service Principal authentication. That's a _much_ better option for scripted use than interactive login. One reason is that it lets you set up credentials specifically for your scripted operation, rather than having that operation run under the credentials of some human user.

Both of the above options still require azcopy login. But, because they don't require any human interaction at login time, you can put the whole thing into a script: switch the session, azcopy login, use azcopy, and (if you want) azcopy logout.

For info on the Managed Service Identity and Service Principal authentication options, see https://docs.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy-login . And ask here (in this issue or a new issue) if anything isn't clear or doesn't work as expected).

[nofunatall]

The service principal method you referenced is in relation to Azure AD service principals with RBAC roles yes?

e.g to create a SP you can do along the lines of:
az ad sp create-for-rbac --name "blobCleanup_${ACCOUNT}_$(date +%Y%m%d%H%M%S)" --role "Storage Account Contributor" --output yaml

Then just use the password/seceret generated from that SP creation for login with azcopy login --service-principal

[JohnRusk]

Correct, it relates to Azure AD with RBAC roles.

I haven't personally tried doing it with az ad. I made my service principal stuff through the portal when I tested the AzCopy feature. I presume they are both doing the same thing.

In the portal, I had a choice of doing a password/secret or a client certificate (which is effectively just a different form of secret, but because its a file you can apply file-level access controls to it as an additional level of security). Both the password/secret option and the certificate option are supported by AzCopy.

The documentation I referred to at the time included this page: https://docs.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals and this one https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal

[JohnRusk]

Ah, yes, I see the docs for the CLI option here: https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli?toc=%2Fazure%2Fazure-resource-manager%2Ftoc.json&view=azure-cli-latest Sounds good.

[nofunatall]

How exactly do you initiate login with service principal?

In this example code I am creating a service principal with AZ CLI.
The secret is retrieved and stored into the AzCopy predefined variable
The app id is stored in a user variable and passed through to the login command.

SP=$(az ad sp create-for-rbac --name "blobCleanup_${ACCOUNT}_$(date +%Y%m%d%H%M%S)" --role "Storage Account Contributor" --output yaml)
AZCOPY_SPA_CLIENT_SECRET=$(echo "$SP" | grep -oP "(?<=password: ).*")
APP_ID=$(echo "$SP" | grep -oP "(?<=appId: ).*")
azcopy login --service-principal "$APP_ID"

Failed to perform login command:
service principal auth requires an application ID, and client secret/certificate

[JohnRusk]

What you have looks pretty good. This doc suggests you _may_ need to set the tenant-id too, but I would guess that works the same as interactive login, when the tenant-id is optional if the user/service principal is logging into their "home" Azure AD tenant (i.e. the one where they are defined, rather than some other one in B2B auth).

So that's a long way of saying your AzCopy command line is probably fine. Do you by any chance need "export" at the start of the line that defines AZCOPY_SPA_CLIENT_SECRET? My Linux scripting is rusty, so I'm not sure, but the error message sounds like AzCopy couldn't see that env var.

[nofunatall]

Export is for use with sub processes and sub shells so is not needed in this context but either way it makes no difference as I tested with export and without export.

[JohnRusk]

Oh, doh (slaps own head)! I realise what the problem is. You have the app Id on the command line, but it's not tagged with the parameter name. Last line of your script should be

azcopy login --service-principal --application-id "$APP_ID"

Does that solve it?

[nofunatall]

Ahhh in that case then I would suggest a quick touch up to the documentation in --help
This is what it shows for the service principal examples.

Log in as a service principal by using a client secret:
Set the environment variable AZCOPY_SPA_CLIENT_SECRET to the client secret for secret based service principal auth.

• azcopy login --service-principal

Log in as a service principal by using a certificate and it's password:
Set the environment variable AZCOPY_SPA_CERT_PASSWORD to the certificate's password for cert based service principal auth

• azcopy login --service-principal --certificate-path /path/to/my/cert

There is no mention of using the switch '--application-id' but it seems it is required for SP logins.

[JohnRusk]

Oh. Thanks. I'll add that our list of things to do for release 10.4.

BTW, once you've had a chance to test using --application-id, can you let me know if that gives you a workable solution for your scripting scenario?

[nofunatall]

I used this line to get the login to work

azcopy login --service-principal --application-id "$APP_ID" --tenant-id "$TENANT_ID"

I did have to set a sleep timer for a few minutes in the script after the SP was created.

[JohnRusk]

Oh, yeah. There can be some time delay there. I think the expectation is that Service Principal accounts will be created relatively infrequently, but used often. I.e. not recreated each time. E.g. you want to do some new scripted thing. You make a service principal account once, and grant it the necessary rights. Then, in the following weeks months or years, your script uses the same service principal every time.

It would not be the expected usage pattern to re-create a brand new service principal on every invocation of the script (as your test code above is doing). That would be a bit like creating a new user account every morning for a human user. It might be _possible_, but its not generally _recommended_.

Anyway, thank you for confirming that the above worked for you. I have logged the task to get the in-app help docs corrected.

[nofunatall]

When you created your SP for testing what permission/role level did you grant it?
Owner/Contributor?

I am trying to keep permissions to a minimum for these SP's and so I am using 'Storage Account Contributor'
But I am seeing some errors:

This is an attempt to download a blob using it's absolute path.
azcopy copy "https://$ACCOUNT.blob.core.windows.net/$CONTAINER/$BLOB" "$HOME/blobs/"

failed to perform copy command due to error: cannot use directory as source without --recursive or a trailing wildcard (/*)

When using --recursive there is now a permission error
**Not sure why it's asking me to include recursive when I am referring to a blob directly for copying.

failed to perform copy command due to error: cannot start job due to error: cannot list blobs. Failed with error -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/[email protected]/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthorizationPermissionMismatch) =====
Description=This request is not authorized to perform this operation using this permission.
RequestId:4f01d280-601e-001a-6ac9-c5b6a0000000
Time:2020-01-08T02:15:08.4045037Z, Details:
Code: AuthorizationPermissionMismatch

[JohnRusk]

It needs to be Storage Blob Data Contributor. For details see https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-portal. Note that it can take up to 5 mins to take effect when you first grant the role.

[nofunatall]

Working now after two attempts with a SP set with that specific role.

[JohnRusk]

Glad to hear. Thanks. I'll close this issue now.

[nofunatall]

Can you confirm there is no way to do account to account copies without using SAS tokens?
I just attempted copying across accounts using the SP login with both accounts in the same subscription and it errors out.

failed to perform copy command due to error: a SAS token (or S3 access key) is required as a part of the source in S2S transfers, unless the source is a public resource

[JohnRusk]

Right now, yes, SAS tokens are needed for the source on account to account copies. When we release version 10.4, that will be fixed and you'll be able to use AzCopy login to cover both source and destination. It will work by AzCopy generating source sas tokens for you behind the scenes, using the User Delegation Token feature of Storage. 10.4 is our next scheduled release. We don't pre-announce dates, sorry.

[cometta]

when will 10.4 be available

[JohnRusk]

Good question @cometta . Unfortunately, due to other urgent needs, we've had to move this from 10.4 to 10.5. We don't have any date to announce for those releases at this stage. I'm sorry I don't have better news for you.

[cometta]

@JohnRusk at the moment, it is possible to use azcopy command or az-cli to retrieve the sts token ? i would like to retrieve it programmatically on a linux machine.

[nofunatall]

@cometta

@JohnRusk at the moment, it is possible to use azcopy command or az-cli to retrieve the sts token ? i would like to retrieve it programmable way on a linux machine.

By STS I assume you mean the Azure equivalent being a SAS token?
If that is what you mean this is how you can generate and format the token to work with AzCopy within a bash shell using Azure CLI and some sed.

EXPIRE=$(date -u -d "3 months" '+%Y-%m-%dT%H:%M:%SZ')
START=$(date -u -d "-1 day" '+%Y-%m-%dT%H:%M:%SZ')
SAS=$(az storage account generate-sas --account-name $ACCOUNT --account-key $KEY --start $START --expiry $EXPIRE --https-only --permissions acdlpruw --resource-types sco --services bfqt | sed 's/%3A/:/g;s/\"//g')

[brreisner]

So to clarify there is no way to programmatically sync a container between storage accounts, in the same or separate subscriptions via SPN Auth. Even if that SPN has the proper permissions set on both source and destination?

We are trying to add a step in our devops pipeline to sync a media container from account A in subscription A to account B in subscription B without user interaction.

It would appear that AZ copy is included in the hosted build agents so it can be called natively without the need to download etc. And we can precreate a SPN and assign it the necessary permissions to both accounts (even though they are in separate subscriptions the SPN permissions can span them).

But posts in this thread and everything I'm reading appears to point to a hard requirement to use SAS tokens in this scenario?

Seems silly that this is so difficult, can't use Keys like I would for most access scenarios (that would be easy), can't use a SPN. Creating SAS tokens programmatically seems to be a pain but I guess I could attempt that, or even pre-create one that is good for say a year (essentially just turning a SAS token into a Key).

If I understand correctly in 10.5 we would be able to achieve this scenario simply with an SPN? Any plans to allow the use of Keys? Did I miss understand anything?

Thanks in advance for any clarification. This has been a difficult one to track down, look forward to answering this question definitively.

[brreisner]

Additionally would the "Storage Blob Data Reader" permission not be sufficient for the source which we are only copying from?

[JohnRusk]

Use OAuth for both source and destination requires us to update AzCopy to use User Delegation Keys. You can track the relevant PR here https://github.com/Azure/azure-storage-azcopy/pull/689. It will not be included in release 10.4. After 10.4 is out, there will probably be some refactoring of the PR to align it with changes made in 10.4 and proposed in https://github.com/Azure/azure-storage-azcopy/pull/853).

I suspect that Storage Blob Data Reader probably would be enough once that PR is out.

We do not want AzCopy users to use account keys. There are better, and more secure options now. (But as this thread shows, we have a little more work to do to expose them all).

[adreed-msft]

Blob Storage Data Reader contains the Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey permission, so yes, once the PR is released, you should be able to do so.