Aws-lambda-dotnet: Q: Clarify use of AWSOptions / GetAWSOptions

[wdolek]

I followed documentation Configuring the AWS SDK for .NET with .NET Core which describes use of AWSOptions as it would be possible to override cloud environment by placing required values into appsettings.Development.json.

However when I deployed my service to ECS, nothing worked. Long story short, I ended up with this additional code in my Startup.cs:

        private AWSOptions GetAwsOptions()
        {
            var awsOptions = _configuration.GetAWSOptions();

            if (_env.IsDevelopment())
            {
                var credentialProfileStoreChain = new CredentialProfileStoreChain();
                if (credentialProfileStoreChain.TryGetAWSCredentials(awsOptions.Profile, out var credentials))
                {
                    awsOptions.Credentials = credentials;
                }
            }

            awsOptions.Credentials = awsOptions.Credentials ?? new ECSTaskCredentials();
            awsOptions.Region = awsOptions.Region ?? new EnvironmentVariableAWSRegion().Region;

            return awsOptions;
        }

Notice that I had to default to ECSTaskCredentials and EnvironmentVariableAWSRegion.Region - because IConfiguration.GetAWSOptions extension is returning empty AWSOptions if config section is not present (and I don't want this section on cloud). See implementation of GetAWSOptions.

(I'm using DI to resolve IAmazonSecretsManager, and manually instantiating AmazonSimpleNotificationServiceClient providing access/secret key)

Is there any example how this API should be used correctly? I was expecting GetAWSOptions will take care of detecting on which environment code runs, so I can seamlessly use same code both on local env and cloud.

[3GDXC]

@wdolek the following works for me

…. Startup.cs - ConfigureServices
services.AddAWSService(
Configuration.GetAWSOptions());
….
with the appsettings.development.json having the AWS section with the "Region" value
after you have configured the aws cli tools with a users access_key and secret_key
to allow running the code locally, deployment also worked when using a cloudformation template and IAM roles

[3GDXC]

@wdolek note ACCESS_KEY and SECRET_KEY can be supplied as ENVIRONMENT_VARIBLES in your deployment configuration

[wdolek]

@3GDXC we are using CDK for deployment. However I'm bit lost.

• yes, I can confirm code works fine locally, since appsettings.Development.json and ~/.aws/credentials are present (this is not the issue)
• but when I deploy this to ECS (via CDK), it doesn't work any more - environment is production, appsettings.json doesn't contain any AWS section and code is not capable getting info from ECS Task or ENV vars automatically as I would expect (I have to assign values into empty AWSOptions instance created by GetAWSOptions)

Forgot to mention:

• netcoreapp2.1
• AWSSDK.Extensions.NETCore.Setup 3.3.101

[3GDXC]

@wdolek are you able to enumerate and output the values contained in Configuration (non-production only) to allow for debugging? sounds as if the ENV values aren't set correctly

[wdolek]

@3GDXC Sorry, I'm not sure if I understand.

Checked our CDK and those vars: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION are set on production (cloud). My point is, I still have to assign instance of _reader_ of those values into AWSOptions instance because it is not picking it up (it only tries to read it from application config file).

So even if I enumerate Environment.GetEnvironmentVariables(), it still doesn't mean AWSOptions will be set up properly.

Is there any example how this should be used - both for local development (using appsettings.Development.json) and production (ECS)?

[wdolek]

@3GDXC another way how to reproduce, in Startup.cs:

        public void ConfigureServices(IServiceCollection services)
        {
            var awsOptions = Configuration.GetAWSOptions();
            services.AddDefaultAWSOptions(awsOptions);

            // both will throw NRE: Region is null, Credentials is null
            var region = awsOptions.Region.SystemName;
            var accessKey = awsOptions.Credentials.GetCredentials().AccessKey;

            // ... rest of stuff
        }

[wdolek]

@3GDXC another way how to reproduce, in Startup.cs:

        public void ConfigureServices(IServiceCollection services)
        {
            var awsOptions = Configuration.GetAWSOptions();
            services.AddDefaultAWSOptions(awsOptions);

            // both will throw NRE: Region is null, Credentials is null
            var region = awsOptions.Region.SystemName;
            var accessKey = awsOptions.Credentials.GetCredentials().AccessKey;

            // ... rest of stuff
        }

[3GDXC]

@wdolek the application configuration loads the environment variable as default; these are used to populate the AwsOptions which should be returned by the Configuration extension method 'GetAwsOptions()' I have used this with success and I'm not seeing the same issue as you describe; can you share the code and/or the repo that reproduces your problem?

[3GDXC]

@wdolek which version of the AWSSDK.Extensions.NETCore.Setup package are you using?

see the source https://github.com/aws/aws-sdk-net/blob/master/extensions/src/AWSSDK.Extensions.NETCore.Setup/ConfigurationExtensions.cs

[wdolek]

@3GDXC Hi, I'm investigating what's going on with env vars in our deployment pipeline - maybe that this is really the place where something is wrong. Unfortunately I won't be able to disclose any source code.

I'm using: AWSSDK.Extensions.NETCore.Setup Version 3.3.101

When checking source code of GetAWSOptions(IConfiguration, string), I'm still bit unsure with "reading env vars by default". What am I missing (see comments in snippet)?

        public static AWSOptions GetAWSOptions(this IConfiguration config, string configSection)
        {
            var options = new AWSOptions();

            IConfiguration section;

            if (string.IsNullOrEmpty(configSection))
                section = config;
            else
                // https://docs.microsoft.com/en-us/dotnet/api/system.configuration.configuration.getsection?view=netframework-4.8&viewFallbackFrom=netcore-2.1#returns
                section = config.GetSection(configSection);

            // since I stick with default config key "AWS" (branching to else above), and there's no "AWS" key on my prod config, this is `true`, thus it returns immediately empty options
            if (section == null)
                return options;

            // ... rest of the method
          }

[3GDXC]

@wdolek the aspnetcore code (unless you've overridden it) will load environment values into configuration as its default behaviour, this is why I suggested enumeration over the Configuration values (as in the example output from our services)

> Configuration values:
> _AWS_XRAY_DAEMON_ADDRESS = 
> _AWS_XRAY_DAEMON_PORT =
> ASPNETCORE_ENVIRONMENT = Development
> AWS = 
> AWS_ACCESS_KEY_ID = ***
> AWS_DEFAULT_REGION = eu-west-1
> AWS_REGION = eu-west-1
> AWS_SECRET_ACCESS_KEY = ***
> AWS_XRAY_CONTEXT_MISSING = LOG_ERROR
> AWS_XRAY_DAEMON_ADDRESS = 169.254.79.2:2000
> AWS:Region = eu-west-1
> ENVIRONMENT = Development

[wdolek]

@3GDXC wait, are you referring to Development environment? I don't have problem with dev, but with production - where I don't override AWS config key obviously.

From ECS Task definition, I see these env variables are set (I'm not able to do remote debugging and I don't want secrets to leak into logs):

{
  ...
  "containerDefinitions": [
    {
      ...
      "environment": [
        {
          "name": "AWS_ACCESS_KEY_ID",
          "value": ...
        },
        {
          "name": "AWS_DEFAULT_REGION",
          "value": ...
        },
        {
          "name": "AWS_SECRET_ACCESS_KEY",
          "value": ...
        },
        ...
      ],
      ...
    }
  ],
  ...
}

For full picture, configuration is set to add env vars too (but I guess AWS SDK is accessing env vars trough Environment anyway):

            WebHost
                .CreateDefaultBuilder(args)
                .ConfigureAppConfiguration(
                    (_, configurationBuilder) =>
                    {
                        configurationBuilder.AddEnvironmentVariables();
                    })
                .UseStartup<Startup>();

[grahamehorner]

@wdolek the environment make no difference at all it is a simple ENV variable, have you tried setting the AWS__REGION note double underscore is what .netcore configuration uses to create a configuration section

[wdolek]

@grahamehorner @3GDXC I got confused by Development in example.

Anyway, I feel like total idiot now. I stepped back and started from scratch and it works. However I removed all AWS_ env variables from being set from our CDK, those are already present in ECS by default. I'm suspecting I was passing wrong value to AWS_ env var and thus it was failing for me (we obtained this project from another team some time ago and didn't check if everything fits together - apparently not). I also refactored Startup and related components so it does not access AWSOptions right away - effectively eliminating another place where NRE could be thrown.

I'm closing this ticket. @3GDXC I'm sorry you wasted so much time with me, patiently explaining me how it should work, thank you! 🙇

[3GDXC]

@wdolek Glad you got your issue sorted; and if I helped in anyway to identify your route cause then it's time well spent as far as I'm concerned, I'm a developer that likes to try and help where I can and time is never wasted it's always good to bounce around a few suggestions and/or have someone try and reproduce issues. That's the wonderful would of open source and the open source community