Aws-sdk-net: Operation was cancelled exception when credentials rely on profiles stored locally

[xalhera]

Steps to Reproduce (for bugs)

I'm using AWS SDK for .Net to :

• Secure my app secrets (Amazon.Extensions.Configuration.SystemsManager)
• Manage the Parameter Store (AWSSDK.SimpleSystemsManagement)
• Protect my app data (Amazon.AspNetCore.DataProtection.SSM)

I followed the instructions provided by the developer guide.

This is how I load app parameters from the Parameter Store :
``` C#
return WebHost.CreateDefaultBuilder(args)
.ConfigureAppConfiguration((context, config) =>
{
config.AddSystemsManager("/my-parameter-path");
})
.UseStartup()
.Build();

The appsettings.json looks like this :
```JSON
{
    "AWS": {
        "Profile": "my-application-profile",
        "Region": "ap-southeast-2"
    }
}

I used Visual Studio Toolkit to create my-application-profile. I can see it in the profiles dropdownlist of AWS Explorer.
It is well configured in the C:\Users\my-secret-name\.aws\credentials file :

[my-application-profile]
aws_access_key_id = <secret-key-id>
aws_secret_access_key = <secret-access-key>
region=ap-southeast-2
toolkit_artifact_guid=946a7007-d13a-4c92-b7dd-6f1a9393d826

Here is the content of GetAWSOptions() method when I debug my application:

The Credentials field stays null and the call of any AWS SDK Service throws The operation was cancelled exception.

Possible Solution (workaround)

Everything works like a charm if I specify the couple Key Id and Access Key directly in the code :
```C#
.ConfigureAppConfiguration((context, config) =>
{
config.AddSystemsManager("/my-parameter-path", new AWSOptions
{
Credentials = new BasicAWSCredentials("secret-key-id", "secret-access-key"),
Region = RegionEndpoint.APSoutheast2
});
})

I spent all the day trying to find a solution by myself :
- It isn't a timeout problem, as I am using IIS to host my app and Request Timeout is configured to 20 minutes.
- It isn't a file access problem because I allowed `everyone` to read `.aws` folder.
- I tried to load the profile from a custom location with `ProfilesLocation`, and the file was loaded (I changed the region to confirm that)

Because of all of these tests, I suppose something is wrong during the loading of credentials from the file.

## Expected Behavior
I expect that `my-application-profile` credentials could be loaded from the local store and used to retrieve data from AWS without any operation error.

## Current Behavior
You can find the Stack Trace below :

at System.Net.Http.HttpClient.HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at System.Net.Http.HttpClient.GetStringAsyncCore(Task1 getTask)
at Amazon.Runtime.Internal.Util.AsyncHelpers.<>c__DisplayClass1_11.<<RunSync>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at Amazon.Runtime.Internal.Util.AsyncHelpers.ExclusiveSynchronizationContext.BeginMessageLoop() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Internal\Util\_mobile\AsyncHelpers.cs:line 142 at Amazon.Runtime.Internal.Util.AsyncHelpers.RunSync[T](Func1 task) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Internal\Util_mobile\AsyncHelpers.cs:line 87
at Amazon.Util.AWSSDKUtils.DownloadStringContent(Uri uri, TimeSpan timeout, IWebProxy proxy) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Util\AWSSDKUtils.cs:line 1138
at Amazon.Util.EC2InstanceMetadata.GetItems(String relativeOrAbsolutePath, Int32 tries, Boolean slurp) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Util_bcl+netstandard\EC2InstanceMetadata.cs:line 513
at Amazon.Util.EC2InstanceMetadata.get_IAMSecurityCredentials() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Util_bcl+netstandard\EC2InstanceMetadata.cs:line 311
at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.RuntimeCredentials_bcl+netstandard\DefaultInstanceProfileAWSCredentials.cs:line 142
at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.RuntimeCredentials_bcl+netstandard\DefaultInstanceProfileAWSCredentials.cs:line 88
at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentialsAsync() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.RuntimeCredentials_bcl+netstandard\DefaultInstanceProfileAWSCredentials.cs:line 106
at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsyncT in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HandlersCredentialsRetriever.cs:line 90
at Amazon.Runtime.Internal.RetryHandler.InvokeAsyncT
at Amazon.Runtime.Internal.RetryHandler.InvokeAsyncT in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\RetryHandler\RetryHandler.cs:line 137
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsyncT
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsyncT
at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsyncT
at Amazon.Runtime.Internal.MetricsHandler.InvokeAsyncT
at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.GetParametersByPathAsync()
at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.GetDataAsync()
at Amazon.Extensions.Configuration.SystemsManager.SystemsManagerConfigurationProvider.LoadAsync(Boolean reload)
at Amazon.Extensions.Configuration.SystemsManager.SystemsManagerConfigurationProvider.Load()
at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()

## Context
I don't want to package my application as a docker image with sensitive data inside it.

## Your Environment
* AWSSDK.Core version used: 3.3.103.24
* Service assembly and version used:

| Package | Version |
| :--- | :--- |
| Amazon.Extensions.Configuration.SystemsManager | 1.2.0 |
| AWSSDK.SimpleSystemsManagement | 3.3.105.10 |
| Amazon.AspNetCore.DataProtection.SSM | 1.0.1 |

* Operating System and version: Windows 10 Entreprise (Version 1903)
* Visual Studio version: Microsoft Visual Studio Professional 2019 (Version 16.16)
* Targeted .NET platform: .Net Core 2.2

## .NET Core Info
* .NET Core version used for development: 2.2.401
* .NET Core version installed in the environment where application runs: 2.2.6
* Output of `dotnet --info`: 

SDK .NET Core (reflétant tous les global.json) :
Version: 2.2.401
Commit: 729b316c13

Environnement d'exécution :
OS Name: Windows
OS Version: 10.0.18362
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\2.2.401\

Host (useful for support):
Version: 2.2.6
Commit: 7dac9b1b51

.NET Core SDKs installed:
2.2.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

* Contents of project.json/project.csproj: 
```XML
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.2</TargetFramework>
    <Platforms>AnyCPU;x64</Platforms>
    <DockerComposeProjectPath>..\docker-compose.dcproj</DockerComposeProjectPath>
    <DockerDefaultTargetOS>Linux</DockerDefaultTargetOS>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Amazon.AspNetCore.DataProtection.SSM" Version="1.0.1" />
    <PackageReference Include="Amazon.Extensions.Configuration.SystemsManager" Version="1.2.0" />
    <PackageReference Include="AWSSDK.SimpleSystemsManagement" Version="3.3.105.10" />
    <PackageReference Include="Microsoft.AspNetCore.App" />
  </ItemGroup>

</Project>

[klaytaybai]

Hello @xalhera, I think we can workaround this. Can you go to your user's AppData\Local\AWSToolkit directory and find the RegisteredAccounts.json file? This is a credentials file that is created when using the Visual Studio Toolkit that takes precedence over other configuration options due to backwards compatibility concerns. If you have the credentials in this file backed up elsewhere, try deleting the file and rerunning your code. I expect it will then work. Please let me know how this works for you.

[xalhera]

Thank you @klaytaybai for your answer.

Unfortunately, I have already read something about that, there is no file named RegisteredAccounts.json in my AppData\Local\AWSToolkit directory but :

• downloadedfiles (folder)
• logs (folder)
• MiscSettings.json (file)
• NonNetSDKCredentialStoreMetadata.json (file)

The workaround you suggest is not really helpful in my case, sorry :/

[arex388]

I'd like to join the conversation as I have the same problem. In my case I used PowerShell to configure a profile, like so:

Set-AWSCredential -AccessKey "???" -SecretKey "???" -StoreAs "MyProfile"

I did this to the live server (Windows Server 2019) and to my local computer. Debugging from Visual Studio or manually launching the app (ASP.NET Core 2.2) from the CLI on my computer works fine.

On the server, launching the app from IIS fails, but manually launching it from the CLI works fine. Here's the captured exception:

System.OperationCanceledException: The operation was canceled.
   at System.Net.Http.HttpClient.HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at System.Net.Http.HttpClient.GetStringAsyncCore(Task`1 getTask)
   at Amazon.Runtime.Internal.Util.AsyncHelpers.<>c__DisplayClass1_1`1.<<RunSync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Amazon.Runtime.Internal.Util.AsyncHelpers.ExclusiveSynchronizationContext.BeginMessageLoop()
   at Amazon.Runtime.Internal.Util.AsyncHelpers.RunSync[T](Func`1 task)
   at Amazon.Util.AWSSDKUtils.DownloadStringContent(Uri uri, TimeSpan timeout, IWebProxy proxy)
   at Amazon.Util.EC2InstanceMetadata.GetItems(String relativeOrAbsolutePath, Int32 tries, Boolean slurp)
   at Amazon.Util.EC2InstanceMetadata.get_IAMSecurityCredentials()
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials()
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials()
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentialsAsync()
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.GetParametersByPathAsync()
   at Amazon.Extensions.Configuration.SystemsManager.Internal.SystemsManagerProcessor.GetDataAsync()
   at Amazon.Extensions.Configuration.SystemsManager.SystemsManagerConfigurationProvider.LoadAsync(Boolean reload)
   at Amazon.Extensions.Configuration.SystemsManager.SystemsManagerConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
   at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
   at DFenceGPS.Web.Login.Program.Main(String[] args) in E:\Visual Studio\???\???\Program.cs:line 11

I did have a RegisteredAccounts.json and I tried deleting it as suggested, but nothing changed. I also added Everyone with full access to the folder, and it didn't help either. This is my current Program.cs right now, just so I can log the exception:

public sealed class Program {
    public static async Task Main(
        string[] args) {
        try {
            await WebHost.CreateDefaultBuilder(args)
                         .ConfigureAppConfiguration(
                             c => {
                                 c.AddSystemsManager("/");
                             })
                         .UseStartup<Startup>()
                         .Build()
                         .RunAsync();
        } catch (Exception e) {
            Console.WriteLine(e);
            throw;
        }
    }
}

And the appsettings.json file:

{
    "AWS": {
        "Profile": "MyProfile",
        "Region": "us-east-1"
    }
}

This is unfortunately a critical problem for me because this project is actually split into five apps (all ASP.NET Core 2.2) and they all share the same configuration, and currently none are working. I'll try @xalhera's workaround, but like him, I don't want to publish with the credentials hard-coded in, which also defeats the purpose of the whole profile setup.

I just spent the past three days figuring out how to get the SSM Parameter Store to work across the board, and now I'm failing on an error that isn't of my own making. It's quite frustrating.

[kgaska]

I had exactly same issue and it turned out that problem was that I was using IIS instead of IIS Express. When you run application under IIS, AWS SDK is looking for credentials in different folder:
C:\Windows\System32\config\systemprofile
So once I gave IIS account permission to that folder and moved .aws folder there it started to working correctly. Hope that helps!

[xalhera]

Thank you @kgaska, I can't test your fix since I moved to Microsoft Azure Key Vault, but I trust you :)

[FakeJoe]

I had exactly same issue and it turned out that problem was that I was using IIS instead of IIS Express. When you run application under IIS, AWS SDK is looking for credentials in different folder:
C:\Windows\System32\config\systemprofile
So once I gave IIS account permission to that folder and moved .aws folder there it started to working correctly. Hope that helps!

This is awesome advice, it works like a charm, thanks!