Aws-sdk-cpp: Add support for assuming a role via a profile

[ltn100]

Currently the C++ SDK supports using credentials from the ~/.aws/credentials file, and different profiles can be selected by setting the AWS_DEFAULT_PROFILE environment variable. However, if the profile assumes a role (as supported by the CLI) then the access/secret keys become invalid (the role_arn and source are not currently supported).

An excerpt from the logs shows:

[DEBUG] 2016-04-06 14:15:34 ProfileConfigFileAWSCredentialsProvider [140635887626240] Found profile Dev
[TRACE] 2016-04-06 14:15:34 ProfileConfigFileAWSCredentialsProvider [140635887626240] Found property role_arnfor profile Dev
[TRACE] 2016-04-06 14:15:34 ProfileConfigFileAWSCredentialsProvider [140635887626240] Found property source_profilefor profile Dev
[INFO] 2016-04-06 14:15:34 ProfileConfigFileAWSCredentialsProvider [140635887626240] Access key for profile not found.
[INFO] 2016-04-06 14:15:34 ProfileConfigFileAWSCredentialsProvider [140635887626240] Secret key for profile not found.
[INFO] 2016-04-06 14:15:34 ProfileConfigFileAWSCredentialsProvider [140635887626240] Optional session token for profile not found.

[JonathanHenson]

This is coming incredibly soon. Sorry for the delay.

[clechasseur]

This remains open, but is this supposed to be implemented now? I'm having a problem that looks similar, i.e. my .aws/credentials file is set up with a source_profile and role_arn, but I always get AccessDenied when performing S3 operations through the C++ SDK, whereas the same operations succeed with the CLI.

[pavanjains]

I am having the same problem, is this looked into?

[mdashti]

Can you give an update on this feature? Is there any other way to authenticate users via role_arn (even if it should be done explicitly and not still available in the default auth chain)?

[LantosDavid]

We wrote the following temporary workaround on our end:

// 'getProfileToUse' is copied from the ProfileConfigFileAWSCredentialsProvider constructor in the
// AWS C++ sdk (AWSCredentialsProvider.cpp)
Aws::String getProfileToUse(void)
{
    char const* const profileFromVar = getenv("AWS_DEFAULT_PROFILE");
    if(profileFromVar != nullptr) {
        return Aws::String(profileFromVar);
    } else {
        return Aws::String("default");
    }
}

Aws::Config::Profile GetCurrentProfile(void)
{
    Aws::String profileToUse = getProfileToUse();

    Aws::String credentialsFilename =
        Aws::Auth::ProfileConfigFileAWSCredentialsProvider::GetCredentialsProfileFilename();
    Aws::Config::AWSConfigFileProfileConfigLoader credentialsLoader(credentialsFilename);
    if(!credentialsLoader.Load()) {
        return Aws::Config::Profile();
    }
    auto credsFileProfileIter = credentialsLoader.GetProfiles().find(profileToUse);

    if(credsFileProfileIter != credentialsLoader.GetProfiles().end()) {
        return credsFileProfileIter->second;
    }

    Aws::String configFilename =
        Aws::Auth::ProfileConfigFileAWSCredentialsProvider::GetConfigProfileFilename();
    Aws::Config::AWSConfigFileProfileConfigLoader configLoader(configFilename);
    if(!configLoader.Load()) {
        return Aws::Config::Profile();
    }
    auto configFileProfileIter = configLoader.GetProfiles().find(profileToUse);
    if(configFileProfileIter != configLoader.GetProfiles().end()) {
        return configFileProfileIter->second;
    }

    return Aws::Config::Profile();
}

Aws::Auth::AWSCredentials GetAWSCredentials()
{
    Aws::Config::Profile profile = GetCurrentProfile();

    if(!(profile.GetCredentials().GetAWSAccessKeyId().empty()) &&
       !(profile.GetCredentials().GetAWSSecretKey().empty())) {
        return profile.GetCredentials();
    }

    Aws::String roleArn = profile.GetRoleArn();
    Aws::String sourceProfile = profile.GetSourceProfile();
    if(roleArn.empty() || sourceProfile.empty()) {
        return profile.GetCredentials();
    }

    Aws::Auth::ProfileConfigFileAWSCredentialsProvider provider(sourceProfile.c_str());
    Aws::STS::Model::AssumeRoleRequest roleRequest;
    roleRequest.SetRoleArn(roleArn);
    roleRequest.SetRoleSessionName(provider.GetAWSCredentials().GetAWSAccessKeyId());
    Aws::STS::STSClient stsClient(provider.GetAWSCredentials());
    Aws::STS::Model::AssumeRoleOutcome response = stsClient.AssumeRole(roleRequest);
    if(!response.IsSuccess()) {
        LOG_INFO() << "Failed to assume role " << roleArn;
        return provider.GetAWSCredentials();
    }

    LOG_INFO() << "Successfully assumed role " << roleArn;
    Aws::STS::Model::Credentials roleCredentials = response.GetResult().GetCredentials();
    return Aws::Auth::AWSCredentials(roleCredentials.GetAccessKeyId(),
                                     roleCredentials.GetSecretAccessKey(),
                                     roleCredentials.GetSessionToken());
}

Moving on to a more proper fix, I was looking through the AWS C++ sdk source code to see where this could possible fit in. DefaultAWSCredentialsProviderChain seems pretty interesting. It seems like I'd have to create a new AWSCredentialsProvider (there seems to be something like this in the Java SDK, called "ProfileAssumeRoleCredentialsProvider") and add it with AddProvider(). I suppose that if ProfileConfigFileAWSCredentialsProvider returned blank, it might make sense to try a ProfileAssumeRoleCredentialsProvider. One issue here is: I'd have to call AddProvider() in the AWS core module, but the provider itself would need to call AssumeRole() from the STS module. Could the provider be defined in STS and then added from core? It seems like this would cause a circular dependency (ie. core needs STS for the provider object, and presumably STS needs core for all kinds of stuff). Perhaps someone from the AWS team could advise on this.

[LantosDavid]

Here is a slightly cleaned up and corrected version:

/**
 * getProfileToUse() finds and returns the name of the current AWS profile.
 * This function is copied from the ProfileConfigFileAWSCredentialsProvider constructor in the AWS
 * C++ sdk (AWSCredentialsProvider.cpp).
 * @return the name of the current profile
 */
static Aws::String getProfileToUse(void)
{
    char const* const profileFromVar = getenv("AWS_DEFAULT_PROFILE");
    if(profileFromVar != nullptr) {
        return Aws::String(profileFromVar);
    }

    return Aws::String("default");
}

/**
 * GetCurrentProfile() acquires the current profile name, and then finds and returns that profile's
 * associated data (ie. access key, secret key, role ARN, etc).
 * @return the current profile, if found.  Otherwise, a blank profile
 */
static Aws::Config::Profile GetCurrentProfile(void)
{
    Aws::String profileToUse = getProfileToUse();

    Aws::String credentialsFilename =
        Aws::Auth::ProfileConfigFileAWSCredentialsProvider::GetCredentialsProfileFilename();
    Aws::Config::AWSConfigFileProfileConfigLoader credentialsLoader(credentialsFilename);
    if(credentialsLoader.Load()) {
        auto credsFileProfileIter = credentialsLoader.GetProfiles().find(profileToUse);
        if(credsFileProfileIter != credentialsLoader.GetProfiles().end()) {
            return credsFileProfileIter->second;
        }
    }

    Aws::String configFilename =
        Aws::Auth::ProfileConfigFileAWSCredentialsProvider::GetConfigProfileFilename();
    Aws::Config::AWSConfigFileProfileConfigLoader configLoader(configFilename);
    if(configLoader.Load()) {
        auto configFileProfileIter = configLoader.GetProfiles().find(profileToUse);
        if(configFileProfileIter != configLoader.GetProfiles().end()) {
            return configFileProfileIter->second;
        }
    }

    return Aws::Config::Profile();
}

Aws::Auth::AWSCredentials GetAWSCredentials()
{
    // First things first: try the default credentials provider chain.  If that returns something,
    // obviously just use that.
    Aws::Auth::DefaultAWSCredentialsProviderChain defaultProviderChain;
    Aws::Auth::AWSCredentials credentials = defaultProviderChain.GetAWSCredentials();
    if(!credentials.GetAWSAccessKeyId().empty() && !credentials.GetAWSSecretKey().empty()) {
        LOG_VERBOSE() << "Using AWS access key: " << credentials.GetAWSAccessKeyId();
        return credentials;
    }

    // The default credentials provider chain didn't return usable credentials.  Let's check if
    // the current profile contains a role ARN and source profile:
    Aws::Config::Profile profile = GetCurrentProfile();
    Aws::String roleArn = profile.GetRoleArn();
    Aws::String sourceProfile = profile.GetSourceProfile();
    if(roleArn.empty() || sourceProfile.empty()) {
        // the current profile does not contain a role ARN and source profile.  Nothing else we
        // can do here.
        LOG_ERROR() << "No default AWS credentials, and current profile contains no role ARN "
                       "and source profile.  Unable to obtain AWS credentials";
        return Aws::Auth::AWSCredentials("", "");
    }

    // The current profile contains a role ARN and source profile.  Let's try assuming that
    // role:
    Aws::Auth::ProfileConfigFileAWSCredentialsProvider provider(sourceProfile.c_str());
    Aws::STS::Model::AssumeRoleRequest roleRequest;
    roleRequest.SetRoleArn(roleArn);
    roleRequest.SetRoleSessionName(provider.GetAWSCredentials().GetAWSAccessKeyId());
    Aws::STS::STSClient stsClient(provider.GetAWSCredentials());
    Aws::STS::Model::AssumeRoleOutcome response = stsClient.AssumeRole(roleRequest);
    if(!response.IsSuccess()) {
        LOG_ERROR() << "Failed to assume role " << roleArn;
        return Aws::Auth::AWSCredentials("", "");
    }

    LOG_VERBOSE() << "Successfully assumed role " << roleArn;
    Aws::STS::Model::Credentials roleCredentials = response.GetResult().GetCredentials();
    return Aws::Auth::AWSCredentials(roleCredentials.GetAccessKeyId(),
                                     roleCredentials.GetSecretAccessKey(),
                                     roleCredentials.GetSessionToken());
}

This seems to do what we need. I'd be interested in getting feedback from an AWS C++ sdk dev on how something like this could be fit into the original sdk. It would be nice to get this added.

[wps132230]

Hi @LantosDavid, thank you for your contribution!
Could you create a pull request since you've already have the code?

[clechasseur]

Since there wasn't a reply to the last message, I'm assuming this is still not supported?

[LantosDavid]

I finally had a chance to try implementing this. Here is my pull request:

https://github.com/aws/aws-sdk-cpp/pull/1025

[lnetherton]

This functionality does still not work with the latest version. Here is some simple code I am testing with:

#include <iostream>
#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/HeadObjectRequest.h>
#include <aws/core/utils/StringUtils.h>
#include <aws/core/utils/logging/AWSLogging.h>
#include <aws/identity-management/auth/STSProfileCredentialsProvider.h>

using namespace std;

int main() {
    Aws::SDKOptions options;
    Aws::InitAPI(options);

    Aws::Client::ClientConfiguration config;
    config.region = "eu-west-1";
    Aws::S3::S3Client client(config);

    Aws::S3::Model::HeadObjectRequest headObjectRequest;
    headObjectRequest.SetBucket("my-bucket");
    headObjectRequest.SetKey("my_file.txt");

    auto headObjectOutcome = client.HeadObject(headObjectRequest);

    if (headObjectOutcome.IsSuccess())
    {
        cout << headObjectOutcome.GetResult().GetContentLength() << "\n";
    }
    else
    {
        cout << headObjectOutcome.GetError().GetMessage() << "\n";
    }

    return 0;
}

When I run it with the environment variable AWS_DEFAULT_PROFILE=default, then I get a success and it prints the file length.

But when I use AWS_DEFAULT_PROFILE=role, where the "role" profile is defined like this (actual account number changed):

[role]
role_arn = arn:aws:iam::123456789012:role/role
region = eu-west-1
source_profile = default

I get this error:

No response body. with address : 52.218.98.96

My credentials/config are correct because it works fine from the command line:

$ AWS_DEFAULT_PROFILE=role aws s3 ls s3://my-bucket/my_file.txt
2019-05-22 14:45:55      11156 my_file.txt

[singku]

Hi, unlike CLI, assume role is a feature of STSProfileCredentialsProvider, which is a high level library in AWS C++ SDK. It's not in the default credentials resolution chain, which is part of aws-cpp-sdk-core library. To utilize it, you have to opt-in, you can specify the provider when constructing your service client.
Something like

    STSProfileCredentialsProvider credsProvider;
    Aws::S3::S3Client client(config, credsProvider);

[lnetherton]

Hi @singku, yes I did manage to get it to work like that, however this is not very useul when writing code that needs to work on an EC2 instance and locally (outside of AWS). We would have to maintain 2 different versions of the code or configurations when using the service. None of the other language SDKs behave in this way (the CLI uses the python SDK on the back end).

Can the STSProfileCredentialsProvider be added to the default credentials provider chain?

[surry]

Having support for assumed roles by default would also be useful for cases where we are not direct users of the C++ SDK. TensorFlow for example uses the C++ SDK to support reading and writing directly to S3 in addition to local filesystems. Since we can't easily change the TensorFlow code using the C++ SDK, we aren't able to use assumed roles via profiles with TensorFlow (to save training checkpoints to S3 for example).

[LantosDavid]

Adding support for assumed roles to the default provider chain would make the AWS C++ lib consistent with the AWS Java lib (see 'fromProfile()' in ProfilesConfigFile.java, which checks whether the profile is role-based), as @lnetherton mentioned above.
The original pull request we submitted added assumed roles to the default provider chain. There was a circular dependency we had to get around. Is that the reason you didn't add it?

[singku]

Instead of adding it into DefaultCredentialsProviderChain, which is part of core, It's a design choice we made to put STS related logic inside identity-management.

The reasons are:

• Unlike EC2InstanceCredentialsProvider and STSAssumeRoleWithWebIdentityCredentialsProvider we implemented in core, which are relatively simple and don't include request signing, STSAssumeRole is more complicated. We'd have to implement another version of STSClient in core, which would make core logic/code ugly and difficult to manage.
• AssumeRole is implemented in high level library identity-management, if we'd like to add this into the default credentials provider chain. There would be circular dependencies you have to find a way to avoid/get rid of (core depend on identity-management, identity-management depend on core). IMHO, this is tricky, risky and error prune.

However, you can actually utilize this without having to manage different versions of your application code.

Implement your own CredentialsProviderChain, just do this as what we do in DefaultCredentialsProviderChain. And add STSProfileCredentialsProvider into the chain. Then, when creating your service client, you specify this provider chain in your code.

For example: (Copied from DefaultCredentialsProviderChain in core)

class MyAWSCredentialsProviderChain : public AWSCredentialsProviderChain
{
public:
    MyAWSCredentialsProviderChain();
};

MyAWSCredentialsProviderChain::MyAWSCredentialsProviderChain() : AWSCredentialsProviderChain()
{
    AddProvider(Aws::MakeShared<EnvironmentAWSCredentialsProvider>(DefaultCredentialsProviderChainTag));
    AddProvider(Aws::MakeShared<ProfileConfigFileAWSCredentialsProvider>(DefaultCredentialsProviderChainTag));
    //++++++++++++++++>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
    AddProvider(Aws::MakeShared< STSProfileCredentialsProvider>(DefaultCredentialsProviderChainTag));
    //<<<<+++++++++++++++++++++++++++++++++++++++++++
    AddProvider(Aws::MakeShared<STSAssumeRoleWebIdentityCredentialsProvider>(DefaultCredentialsProviderChainTag));

    //ECS TaskRole Credentials only available when ENVIRONMENT VARIABLE is set
    const auto relativeUri = Aws::Environment::GetEnv(AWS_ECS_CONTAINER_CREDENTIALS_RELATIVE_URI);
    AWS_LOGSTREAM_DEBUG(DefaultCredentialsProviderChainTag, "The environment variable value " << AWS_ECS_CONTAINER_CREDENTIALS_RELATIVE_URI
            << " is " << relativeUri);

    const auto absoluteUri = Aws::Environment::GetEnv(AWS_ECS_CONTAINER_CREDENTIALS_FULL_URI);
    AWS_LOGSTREAM_DEBUG(DefaultCredentialsProviderChainTag, "The environment variable value " << AWS_ECS_CONTAINER_CREDENTIALS_FULL_URI
            << " is " << absoluteUri);

    const auto ec2MetadataDisabled = Aws::Environment::GetEnv(AWS_EC2_METADATA_DISABLED);
    AWS_LOGSTREAM_DEBUG(DefaultCredentialsProviderChainTag, "The environment variable value " << AWS_EC2_METADATA_DISABLED
            << " is " << ec2MetadataDisabled);

    if (!relativeUri.empty())
    {
        AddProvider(Aws::MakeShared<TaskRoleCredentialsProvider>(DefaultCredentialsProviderChainTag, relativeUri.c_str()));
        AWS_LOGSTREAM_INFO(DefaultCredentialsProviderChainTag, "Added ECS metadata service credentials provider with relative path: ["
                << relativeUri << "] to the provider chain.");
    }
    else if (!absoluteUri.empty())
    {
        const auto token = Aws::Environment::GetEnv(AWS_ECS_CONTAINER_AUTHORIZATION_TOKEN);
        AddProvider(Aws::MakeShared<TaskRoleCredentialsProvider>(DefaultCredentialsProviderChainTag,
                    absoluteUri.c_str(), token.c_str()));

        //DO NOT log the value of the authorization token for security purposes.
        AWS_LOGSTREAM_INFO(DefaultCredentialsProviderChainTag, "Added ECS credentials provider with URI: ["
                << absoluteUri << "] to the provider chain with a" << (token.empty() ? "n empty " : " non-empty ")
                << "authorization token.");
    }
    else if (Aws::Utils::StringUtils::ToLower(ec2MetadataDisabled.c_str()) != "true")
    {
        AddProvider(Aws::MakeShared<InstanceProfileCredentialsProvider>(DefaultCredentialsProviderChainTag));
        AWS_LOGSTREAM_INFO(DefaultCredentialsProviderChainTag, "Added EC2 metadata service credentials provider to the provider chain.");
    }
}

    auto credsChain = Aws::MakeShared<MyAWSCredentialsProviderChain>(Tag);
    Aws::S3::S3Client client(chain, config);

[ltn100]

Thanks @singku. However, your example is incorrect. You should instead use:

AddProvider(Aws::MakeShared<STSProfileCredentialsProvider>(DefaultCredentialsProviderChainTag));

to get the desired behaviour.

I think it is a shame that it is impossible to amend the provider order in the AWSCredentialsProviderChain, so that we do not have to duplicate the many lines of code that you give in your example above.

Also, the implementation STSProfileCredentialsProvider is itself broken! This is probably due to a recent API change. I have submitted a PR (referenced above) to fix this.

[singku]

@ltn100 Thanks for your reply. You are right that I used the incorrect class in the example. We will review your PR and have a discussion if it's appropriate to manipulate the order of providers.

[duog]

This is very annoying. Rather than adding this credentials provider to the default chain, as it seems all the other language sdks do(are there any other exceptions?), you require every user of the sdk to reimplement the default credentials provider chain? Any application who's author has failed to read this issue will not do this, and will mysteriously fail to AssumeRole, where the cli and applications in other languages will succeed.

[frivard-coveo]

I tried the fix mentionned here, but it didn't work right away. This is due to the fact that the current implementation only looks at the config file, not the credential file. But the any profile with access keys has to be in the credentials file, so it only make sense to read that and not the config.
So I made the change, tested it localling, it worked! So I made a PR to bring the fix in the mainline : #1603