Velero: Support for storage in Azure Germany

[starkilla2003]

Is it already possible to use a storage container in the German instance of Azure? I think the API endpoints are different for that cloud. I am running into an error when i try to setup Ark:

time="2018-01-17T12:31:33Z" level=info msg="Configuring cloud provider for backup service" logSource="pkg/cmd/server/server.go:330"
time="2018-01-17T12:31:34Z" level=info msg="Configuring cloud provider for snapshot service" logSource="pkg/cmd/server/server.go:346"
An error occurred: rpc error: code = Unknown desc = azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/6a6e8a67-b524-415c-a680-0b63e086c12d/locations?api-version=2016-06-01: StatusCode=0 -- Original Error: adal: Refresh request failed. Status Code = '400'

Is it possible to change the API endpoints used by Ark?

[ncdc]

When we've seen this Failed to refresh the Token... error in the past, it's been because the cloud-credentials secret was either missing all the required values, or one of them was incorrect. Could you double-check its contents?

@paulbouwer do you know if the version of the Azure libraries we have vendored in Ark supports Azure Germany offhand?

[starkilla2003]

I double checked all entries in the cloud-credentials. For the German Azure instance it is not possible to find the subscription via https://management.azure.com because there it is https://management.microsoftazure.de .

[ncdc]

Ok, we’ll take a look at the Azure code we’re using and see if it’s
possible to change the endpoint. I’ll be at my desk in an hour or so and
can look then.

On Wed, Jan 17, 2018 at 8:22 AM starkilla2003 [e-mail]
wrote:

I double checked all entries in the cloud-credentials. For the German
Azure instance it is not possible to find the subscription via
https://management.azure.com because there it is
https://management.microsoftazure.de .

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/heptio/ark/issues/287#issuecomment-358302355, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAABYrvAoDrQk1AgOEZq49jsAfJ-j0_Iks5tLfP6gaJpZM4RhSYF
.

[starkilla2003]

Thank you for your fast reply! You can find the full reference under the following link:

https://blogs.msdn.microsoft.com/azuregermany/2016/08/18/endpoints-in-microsoft-cloud-germany/

[ncdc]

Ok, I looked at the code, and it's going to require a code change to support alternate endpoints. We'll try to get this in for v0.7.0 due out next month.

cc @skriss @nrb @jimzim

[paulbouwer]

@ncdc Yes - the Azure libs currently vendored in Ark (master) do support the German Cloud.

This looks like a simple change to support the issue scenario. I'll submit a PR in the next day.

@jimzim @marstr - does this look right?

Azure Environments

The azure.Environment struct in the Azure Go SDK defines the cloud types available via the SDK. Currently these are:

• PublicCloud
• ChinaCloud
• GermanCloud
• USGovernmentCloud

The struct is defined in ark/vendor/github.com/Azure/go-autorest/autorest/azure/environments.go. Here is the definition for the GermanCloud environment and there is a handy method called EnvironmentFromName to return an Environment struct from the string representations I listed above.

Object Store

In the Init function of the Azure Cloud Provider Object Store code in Ark defined in ark/pkg/cloudprovider/azure/object_store.go, you need to replace the storage.NewBasicClient call with a call to storage.NewBasicClientOnSovereignCloud, which accepts an azure.Environment struct. The NewBasicClientOnSovereignCloud function is defined in ark/vendor/github.com/Azure/azure-sdk-for-go/storage/client.go.

Block Store

In the Init function of the Azure Cloud Provider Block Store code in Ark defined in ark/pkg/cloudprovider/azure/block_store.go, you need to replace the disk.NewDisksClient call with a call to disk.NewDisksClientWithBaseURI, which accepts a baseURI that can be obtained from Environment.ResourceManagerEndpoint. The NewDisksClientWithBaseURI function is defined in ark/vendor/github.com/Azure/azure-sdk-for-go/arm/disk/disks.go.

Similarly the disk.NewSnapshotsClient should be replaced with disk.NewSnapshotsClientWithBaseURI. The NewSnapshotsClientWithBaseURI function is defined in ark/vendor/github.com/Azure/azure-sdk-for-go/arm/disk/snapshots.go.

Config

Update the Ark Azure config to allow for the specification of the Azure Cloud (Environment) with a default of PublicCloud for those that don't need to access a sovereign Data Centre.

[jimzim]

So there are a couple things we need to think about.

1. Is the storage api version in German cloud the same as public cloud? This may not be an issue when we upgrade to the latest Azure Go SDK which supports multiple API versions, but I will have to deflect that question to @marstr
2. If we need to change the code in the Init functions, then we will need to think about how the end user can provide a environment variable to allow the code to switch based on the type of Azure Cloud they are in.

Btw, I am looking at what it will take to update to the latest Azure go sdk in Ark. :)

[marstr]

@jimzim says:
Is the storage api version in German cloud the same as public cloud? This may not be an issue when we upgrade to the latest Azure Go SDK which supports multiple API versions, but I will have to deflect that question to @marstr

The API Versions that are available in the German cloud may lag behind public cloud. However, once an API Version is deployed in a sovereign cloud everything _should_ behave identically to the same API Version on the public cloud. The latest Azure SDK for Go versions allow you to choose which API Version you use, so if you choose one that is available everywhere in all of the clouds you need to target, you should be good to go.

@jimzim says:
Btw, I am looking at what it will take to update to the latest Azure go sdk in Ark. :)

Feel free to reach out to me or anybody else on the Azure SDK for Go if something is unclear or you have feedback.

[ncdc]

@paulbouwer any updates on a PR for this?

[paulbouwer]

@ncdc - I'm following up internally based on comments by @jimzim and @marstr. Hopefully will have a definitive answer on api-version support across the environments and can then be confident in solution.

[ncdc]

Sounds great, thanks for the update!

[starkilla2003]

Hello, thx for the work you are putting into this. Is there any progress on this part?

[paulbouwer]

I've got confirmation that the Germany Cloud supports version 2017-03-30 for the Microsoft.Compute/disks and Microsoft.Compute/snapshots resource providers, which is the most current supported in the Public Azure Cloud. This is great for this particular issue, but I'm still following up on versions across all the clouds/environments to support this issue across more than just the Germany data centres.

The Storage Service versions have a default of 2016-05-31 in the Azure Go SDK. This should be supported across all clouds.

I think that the next steps are to move the referenced Azure Go SDK to v14 which supports an easier mechanism to target specific versions of APIs. Moving to this new version will also allow us to leverage the new long running process functionality (which will improve PV snapshot times).

Thoughts? @ncdc @jimzim @marstr

[ncdc]

Yes, let's plan to do 1 PR to upgrade to v14, and if needed, we can do additional PR(s) to add support for the sovereign clouds.

[nrb]

@paulbouwer Hi Paul - is this still in the plan for your team?

[skriss]

We'll need to confirm that the German cloud supports the current version of the API that we're using, and then we'll want to add an optional environmentName key to the Azure BackupStorageLocation and VolumeSnapshotLocation that, if specified, gets parsed and used as the environment when constructing clients per https://github.com/heptio/velero/issues/287#issuecomment-358544653

[geothery]

Hi.
Same issue for Azure China.
Not possible to pass an environmentName to use this ChinaCloud definition : https://github.com/heptio/velero/blob/master/vendor/github.com/Azure/go-autorest/autorest/azure/environments.go#L109-L131

So, it uses the default API endpoint :

time="2019-08-06T08:38:49Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:431"
An error occurred: some backup storage locations are invalid: error getting backup store for location "default": rpc error: code = Unknown desc = azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/......../listKeys?api-version=2018-02-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body

[andyzhangx]

any update for this issue? Thanks.

[andyzhangx]

I have fixed this issue by https://github.com/heptio/velero/pull/1836, someone could take a review? Thanks.