Aws-sdk-java: Is profile prefix deprecated or required?

[AlainODea]

You folks say that profile prefix is deprecated:
https://github.com/aws/aws-sdk-java/blame/46c00efdd9240d2d20d734f806dcc71506af6e0f/aws-java-sdk-core/src/main/java/com/amazonaws/auth/profile/internal/BasicProfileConfigLoader.java#L96-L99

botocore (AWS Python SDK) says that the profile prefix is required:
https://github.com/boto/botocore/blob/d00ee27528efad227ad46d3db9fd2f80f802803f/botocore/configloader.py#L208-L222

Between you, it is seemingly impossible to write ~/.aws/config in a compliant way.

[millems]

This is a definite pain point. It's surprisingly difficult to create a file that makes both the CLI and Java SDK happy. This is fixed in the Java SDK 2.0, but is hard to fix in 1.11, because 1.11 uses the same code path for the ~/.aws/config (that MUST have the prefix) and the ~/.aws/credentials (that MUST NOT have the prefix).

See this issue for more discussion on incompatibilities: https://github.com/aws/aws-sdk-java/issues/803

If you're willing to work with pre-GA software, you could add a dependency on V2's auth module and adapts its ProfileCredentialsProvider to a 1.11.x credentials provider.

Otherwise we can update the warning log to not say it's deprecated (since it's not technically deprecated anymore - it was at one point which points to the chaos around these files), but we'll still need the log to say to use the 'profile' prefix when specifying the credentials provider.

[AlainODea]

@millems thank you for the clarification. I really appreciate it.

I totally understand how this can happen. Lots of fast-moving independent groups doing awesome things and then an undocumented file format sneaks in between you all.

I think updating the message to say it's not deprecated would reduce confusion.

[millems]

How is "Your profile name includes a 'profile ' prefix. This prefix is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when it is referenced from your Java code."?

[AlainODea]

That’s great wording! Thank you.

[dagnir]

We have a fix for this upstream and it should be part of the next release.

[joelittlejohn]

Hi @millems. I don't use profiles via the Java SDK (I only use them with other CLI tools). Since this warning was introduced, I now see it repeated 10 times every time I run an app or library that uses the SDK:

Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.
Nov 24, 2019 10:09:10 PM com.amazonaws.auth.profile.internal.BasicProfileConfigLoader loadProfiles
WARNING: Your profile name includes a 'profile ' prefix. This is considered part of the profile name in the Java SDK, so you will need to include this prefix in your profile name when you reference this profile from your Java code.

I understand there was a desire to make things clear, but the fix is really obtrusive. Configuring log4j in every app/library I use to turn this off isn't really an option.

Could this warning be replaced with a change in the docs somewhere? Even the comments here are likely enough that Google will now provide the answer if anyone is confused.

[mibollma]

I agree with @joelittlejohn.
This annoying warning is spamming all my logs and has almost no utility.
I'm also not using profiles with the Java SDK but instead to switch between stages on the CLI.
Please either follow the suggestion of @joelittlejohn, or limit the output to occur only once e.g. through a static variable or provide a simple global way to mute the whole library.

[laymain]

I have to use both v1 and v2 SDKs in the same application with a custom profile, that is the only way I've found to make it work:

• Setting the environment variable AWS_PROFILE to custom
• Declare two sections in the ~/.aws/config file

[custom]
region=eu-west-1

[profile custom]
region=eu-west-1

But I'm having annoying contradictory warn messages from both SDKs

• SDK v1

WARN  [com.amazonaws.auth.profile.internal.BasicProfileConfigLoader] - The legacy profile format requires the 'profile ' prefix before the profile name. The latest code does not require such prefix, and will consider it as part of the profile name. Please remove the prefix if you are seeing this warning.

• SDK v2

WARN  [software.amazon.awssdk.profiles.internal.ProfileFileReader] - Ignoring profile 'custom' on line 4 because it did not start with 'profile ' and it was not 'default'.

Is there anything that can be done to avoid this?

I think there's also an issue in the BasicProfileConfigLoader: it builds a map of profile properties using the profile name as key, but it does not remove the profile prefix.
So when you have set AWS_PROFILE=custom, it does not load your profile because it stores profile custom in the map of profiles.

[millems]

@laymain Sorry that this is annoying. V2 had to be changed to be more compatible with other SDKs, which broke some compatibility with V1 itself.

For a temporary workaround, when you're using both in the same application, you could use the V2 profile credentials provider with the V1 SDK by implementing V1's AWSCredentialsProvider and delegate to the V2 DefaultCredentialsProvider?

We could see about adding a "forward compatibility" flag with the V1 SDK that allows it to behave consistently with the V2 SDK/CLI/etc.

[laymain]

@millems Thank you for your answer.

To implement your workaround, I also have to implement a wrapper for V1's AwsRegionProvider,
but V1's AwsClientBuilder abstract class does not allow its injection:

https://github.com/aws/aws-sdk-java/blob/1f21df5e3574070cd1ccc94d545d592912758af6/aws-java-sdk-core/src/main/java/com/amazonaws/client/builder/AwsClientBuilder.java#L69-L95

[millems]

@laymain Yeah, unfortunately region providers are an internal concept right now. Since regions don't expire like credentials, we didn't really ever plan to make them external, since configuring a region provider should be the same as configuring the region from that provider.

[laymain]

@millems For now I will stick with my _ugly_ workaround.
But I have to disable warnings in the log4j.properties of every application that uses either SDK v1 or v2:

log4j.logger.software.amazon.awssdk.profiles.internal.ProfileFileReader=ERROR
log4j.logger.com.amazonaws.auth.profile.internal.BasicProfileConfigLoader=ERROR

It will be nice to have this "forward compatibility" flag.

[umeshksingla]

Oh god, so true