Aws-sdk-net: How to troubleshoot "Unable to get IAM security credentials from EC2 Instance Metadata Service."?

Created on 10 Sep 2020 · 23Comments · Source: aws/aws-sdk-net

The Question

Why do I get Amazon.Runtime.AmazonServiceException: Unable to get IAM security credentials from EC2 Instance Metadata Service. in my development environment? Is the issue that my development environment is an Amazon Workspace and not an EC2 Instance, and thus does not have EC2 Instance Metadata? How do I run the sample code in Making requests using IAM user temporary credentials - AWS SDK for .NET on a non-EC2 instance like an Amazon Workspace machine (my development environment)?

My end goal is to be able to:

Not use access key/secret key, but instead resolve credentials via Delegating access across AWS accounts using IAM roles.
However, when I run the code I found Making requests using IAM user temporary credentials - AWS SDK for .NET, I get the aforementioned exception

I read the following documentation:

https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/aws-sdk-net-v3-dg.pdf - I read Page 5, which discusses creating a Dotnet-Tutorial-User. Note: This is a _bit weird_, since it seems like the tutorial is asking me to go backwards to using access key and secret key.
Realizing the tutorial seemed to be advocating bad practices, I googled until I somehow landed on Making requests using IAM user temporary credentials - AWS SDK for .NET - this seemed like exactly what I needed, but when I added this logic to my existing program, I got the above error
I also, of course, read Delegating access across AWS accounts using IAM roles, which got me started on this whole "let's get rid of insecure access keys/secret keys that can be stolen" rabbit trail
Separately, my understanding is that the AWS Extension for Visual Studio can be used to generate access, but that the extension just creates user profiles, which doesn't really get around the fact that then these credentials are stored in raw text files on my machine.

Environment

SDK Version: 3.5.0.9
Package Version:

    <PackageReference Include="AWSSDK.SecurityToken" Version="3.5.0.9" />
    <PackageReference Include="AWSSDK.S3" Version="3.5.0.9" />

OS Info: Windows Server 2016 Datacenter Edition (Amazon Workspace largest instance size)
Build Environment Visual Studio 2019 16.6.5
Targeted .NET Platform: netstandard2.1 library, called from netcoreapp3.1 executable context

This is a :question: general question

guidance modulcredentials

Source

jzabroski

All 23 comments

As a follow-up, I spied in Fiddler4 that the following sequence happens:

#  Result  Protocol  Host              URL
1  200     HTTP      169.254.169.254   /latest/api/token
2  404     HTTP      169.254.169.254   /latest/meta-data/iam/security-credentials
3  200     HTTP      169.254.169.254   /latest/api/token
4  404     HTTP      169.254.169.254   /latest/meta-data/iam/security-credentials

For the two 404 results, I get the following response body:

<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>

I'm not sure why a 404 is what is returned here. It implies like it's a bug on Amazon's side. After reading Ryan Green's Error Handling Patterns in Amazon API Gateway and AWS Lambda, it implies that the resource doesn't exist. How can my credentials not exist?

jzabroski on 10 Sep 2020

As an update, I read Yevgeniy Brikman's Authenticating to AWS with Instance Metadata, which helped me better understand how temporary access keys work. They're truly tied to the EC2 instance.

So, I guess I need guidance on how to test this API call from a non-EC2 instance like an Amazon Workspace. Is it even possible, or is it a _de facto_ drawback of doing things this way that you can only test it via some sort of integration test in a development environment EC2 instance?

jzabroski on 10 Sep 2020

Hi @jzabroski,

Good evening.

As rightly pointed in the article Authenticating to AWS with Instance Metadata, AWS exposes an Instance Metadata endpoint on every EC2 Instance at http://169.254.169.254. Hence you were getting the mentioned error in a non-EC2 instance. I'm afraid you might need to use the client/secret keys directly, may be storing these in the environment variables/credentials file/using KMS (KMS would add unnecessary complexity just for testing) and using the client constructor overloads with these credentials (again, as you pointed out, these are not the best practices). Or may be these credentials could be stored in the app settings.Development.json or config file, but don't include it as part of your deployment pipeline.

I can try to explore what other options might be available. Also, please share your use case and code snippet (with sensitive information removed) for your program/logic which you are trying to test on a non-EC2 instance.

Thanks,
Ashish

ashishdhingra on 11 Sep 2020

Hi @ashishdhingra

My use case is I work with data vendors who are removing FTP and pushing us to use S3 for file downloads. As part of this, we _could_ authenticate to their S3 bucket via access key/secret key pair, but instead we _want_ to use cross-account roles.

Cross-account roles has several advantages, above and beyond those presented by Sam Elmalak at re:Invent 2017. For example, one data vendor rolls their data license-linked access key/secret key pairs quarterly, which means we have to update it quarterly, and that manual process is risky because it creates an unnecessary blast radius. Currently, one data vendor _emails us_ the access key/secret key pair unencrypted. We pointed out to them that that this was not ideal, and if we use cross-account roles instead, we can avoid them needing to send access key/secret key pairs unencrypted. Temporary access credentials should work great in production - but it leaves open the question of how we can test this from an Amazon Workspace instance without provisioning access key/secret key pairs and creating new blast radiuses. All I could find was guidance from Netflix on how to detect a blast from outside the Direct Connect IP range.

_I do think this is a question you will see more and more of, as CSOs realize the advantages of using S3 / STS for file transfer over SFTP or FTP._

The _other thing_ that is just _bizarre_ to me is, Why does AWS, a company that makes 30B a year in revenue, store sensitive data in plaintext? Can't AWSCLI + Visual Studio AWS Extension + .NET AWSSDK use Windows Data Protection API? I realize that's not portable to Linux and Mac OS, but there are probably equivalent APIs there as part of se-linux (one would hope).

As far as code sample, here it is - it's dead simple. I just abstracted away some of the stuff in your API that I thought was not easy to test, so that I could write mock unit tests against it. However, as you can see, there is a fork in my code - one uses temporary access credentials and the other uses access key/secret key pair. It's not desirable to have a fork in my code.

```c#
using Amazon.S3;

namespace Infra.Services.FileTransferers
{
public interface IS3ClientFactory : IService
{
IAmazonS3 Make();
}
}

```c#
using System;
using System.Threading.Tasks;
using Amazon;
using Amazon.Runtime;
using Amazon.S3;
using Amazon.SecurityToken;
using Amazon.SecurityToken.Model;
using Infra.Configuration;
using Infra.Services.Configuration;

namespace Infra.Services.FileTransferers
{
    /// <summary>
    /// The purpose of this class is two-fold:
    /// 1. When working with counterparties, they may not set-up IAM roles for us to assume.
    ///    When alternative credentials to IAM roles are required (e.g. access key/secret key),
    ///    the implementor can have this factory create the S3 client the "right way".
    /// 2. Testability.
    /// </summary>
    public class S3ClientFactory : IS3ClientFactory
    {
        private readonly IConfigurationService _configurationService;

        public S3ClientFactory(IConfigurationService configurationService)
        {
            _configurationService = configurationService ?? throw new ArgumentNullException(nameof(configurationService));
        }
        public IAmazonS3 Make()
        {
            var awsClientConfig = _configurationService.GetSection<AwsClientConfiguration>();
            if (awsClientConfig?.Region == null)
                throw new ArgumentNullException(nameof(AwsClientConfiguration.Region));

            SessionAWSCredentials tempCredentials = GetTemporaryCredentialsAsync().ConfigureAwait(false).GetAwaiter().GetResult();

            // Unfortunately, the AmazonS3Client takes a concrete RegionEndpoint class instance, and does not accept an IRegionEndpoint interface.
            // Note, some Amazon code samples online still use Amazon.Internal.RegionEndpointProviderV2.GetBySystemName
            // but the API was updated to Amazon.RegionEndpoint.GetBySystemName
            var region = RegionEndpoint.GetBySystemName(awsClientConfig.Region);
            AmazonS3Client client;
            if (!string.IsNullOrWhiteSpace(awsClientConfig.AccessKey))
            {
                var awsCredentials = new BasicAWSCredentials(awsClientConfig.AccessKey, awsClientConfig.SecretKey);
                client = new AmazonS3Client(awsCredentials, region);
            }
            else
            {
                // If no access key/secret key pair is provided,
                // then use the EC2 Instance Metadata service to get temporary access credentials
                client = new AmazonS3Client(tempCredentials, region);
            }

            return client;
        }

        // Taken from: https://docs.aws.amazon.com/AmazonS3/latest/dev/AuthUsingTempSessionTokenDotNet.html
        private static async Task<SessionAWSCredentials> GetTemporaryCredentialsAsync()
        {
            using (var stsClient = new AmazonSecurityTokenServiceClient())
            {
                var getSessionTokenRequest = new GetSessionTokenRequest
                {
                    DurationSeconds = 7200 // seconds (2 hours)
                };

                GetSessionTokenResponse sessionTokenResponse =
                    await stsClient.GetSessionTokenAsync(getSessionTokenRequest);

                Credentials credentials = sessionTokenResponse.Credentials;

                var sessionCredentials =
                    new SessionAWSCredentials(credentials.AccessKeyId,
                        credentials.SecretAccessKey,
                        credentials.SessionToken);
                return sessionCredentials;
            }
        }
    }
}

c# namespace Infra.Services.Configuration { public class AwsClientConfiguration { public string AccessKey { get; set; } public string SecretKey { get; set; } public string Region { get; set; } } }

jzabroski on 11 Sep 2020

One last follow up. I discovered there is one other mechanism for accessing an S3 bucket, using private connection with no authentication

Sorry if I come across as a newb at this. I am genuinely reading a ton of documentation and watching a ton of videos to come up to speed.

jzabroski on 11 Sep 2020

Hi @jzabroski,

Good morning.

Following links provide some guidance on how credentials could be accessed in a .NET application:

I suspect we cannot use the option similar to EC2 instance credentials loading, on non-EC2 instance.

If you are looking to remotely debug a .NET application on an EC2 instance (could be staging environment), please refer the article Remote Debug an IIS .NET Application Running in AWS Elastic Beanstalk.

Following articles might help:

Sample applications that use temporary credentials lists some sample applications that use temporary credentials.
Introducing Databricks AWS IAM Credential Passthrough

Thanks,
Ashish

ashishdhingra on 11 Sep 2020

So... for most people, it's considered _normal_ to not be able to debug locally their temporary access credentials? Is that really how people do this?

It seems like a good feature request to be able to have temporary access credentials available to Amazon Workspace machines with STS privileges to things like AmazonS3FullAccess. How do I request that?

jzabroski on 11 Sep 2020

I have added feature-request label to this issue.

Thanks,
Ashish

ashishdhingra on 11 Sep 2020

👍1

@ashishdhingra I think we should close this issue and open a new, clean issue as a feature request. I'll @mention you in the new ticket and you can apply the right labels for triage.

jzabroski on 11 Sep 2020

Sure.

ashishdhingra on 11 Sep 2020

BTW, I just found https://aws.amazon.com/blogs/desktop-and-application-streaming/using-amazon-api-gateway-with-amazon-workspaces-to-interact-with-aws-resources/ which I think directly addresses this problem, If I read it correctly.

jzabroski on 11 Sep 2020

@jzabroski Yes, looking at high level, it appears to provide workaround for your problem. Please give it a try.

Also, I'm not sure if creating a feature request in .NET SDK would help since this support needs to be there at service level first, rather than SDK level. You may try posting a question on Workspaces service page, if required. But the above article appears to address the limitation of workspaces.

ashishdhingra on 11 Sep 2020

Great - where is the Workspaces service page so I can post a question?

jzabroski on 11 Sep 2020

Also, is there a code sample for the AWSSDK where I can call AssumeRole? It seems like my above code sample can be adjusted to at least not require access key/secret key stored in a config file. All I need is a role ARN. That would be great as a workaround. I can then throw an exception with a clear action if I get a 404 on both.

jzabroski on 11 Sep 2020

If you go to Amazon Workspaces page, there is a Contact Us link at the bottom of the page. It would open a case with support.

Thanks,
Ashish

ashishdhingra on 11 Sep 2020

👍1

https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/dotnet/example_code_legacy/STS appears to have some examples.

Thanks,
Ashish

ashishdhingra on 11 Sep 2020

👍1

@ashishdhingra Thank you so much - have a great weekend!

jzabroski on 11 Sep 2020

One last follow-up, in case someone finds this via Google. I found https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/net-dg-config-creds.html which explains the guidance on how to create a provider chain of responsibility pattern in the section titled Credential and Profile Resolution

Good grief, Amazon has a LOT of documentation around IAM, creds, and profiles

jzabroski on 12 Sep 2020

i have a same problem when running the application in docker container and getting assume role.
The root cause is missing AWS credentials.

So we need to have a AWS profile or add these 2 environment variables:

AWS_ACCESS_KEY_ID=my-access-key
AWS_SECRET_ACCESS_KEY=my-secret-access-key

Another sharing is i wanting to mock the amazon request with the mock server, so all need to do is set any key and secret into above environment variables.

hmm, it took around 4h to research, read many documents without result, but finally i can solved it by compare between run application with IDE and docker container lol

datphan310 on 15 Dec 2020

@datphan310 The v3 API actually has a "AWS .NET SDK Store" that does not use environment variables, and is encrypted (but they do a terrible job documenting it and I can't even remember the official name of it). However, you do need to have an AWS profile OR set-up cross-account role level access so that your whole account is trusted by the third party. If you do that, you should talk to your IT/head of security about setting up a specific account just for this purpose, as they will likely also want to set-up network security policies. The other advantage of account-level trust is then your counterparty doesn't need to refresh the access key and it's a lot easier for both sides to pass a security audit. The last advantage to doing it this way is that you don't need to reconfigure your IAM user's default role (which is painful and risky). The reason you need an AWS profile if you don't set-up cross-account role level access is that an IAM user can't assume a role.

I think some of this documentation complexity doesn't bother people who have been developing on AWS for years, but many are likely also doing things in sub-optimal and insecure ways. I've worked with two vendors trying to set up cross-account roles and neither of them got it right the first time they read the docs I sent over.

jzabroski on 15 Dec 2020

@datphan310 https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/sdk-store.html - unfortunately it is Windows only right now. Probably because they use Windows DAPI under the hood is my guess (that's how I would do it).

jzabroski on 15 Dec 2020

hi @jzabroski, is use AWS profile the best way when deploying the application? but sometime we just want to run the application alone such as docker container, we have many options (see more at https://stackoverflow.com/a/56077990/10403887) however i think use environment variables is simpliest in my case.

datphan310 on 16 Dec 2020

See: https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/net-dg-config-creds.html

Preferred approach is to use AWS .NET SDK store

jzabroski on 16 Dec 2020

Was this page helpful?

0 / 5 - 0 ratings