Athens: Use AWS ECS credentials

Created on 12 Jun 2019  ·  60Comments  ·  Source: gomods/athens

Is your feature request related to a problem? Please describe.

When running athens in AWS ECS (Fargate), the environment is provided credentials through a different avenue than the usual environment variables (i.e.: AWS_ACCESS_KEY_ID). I have a sidecar process go fetch the credentials and inject them into the config file:

for s in $(wget -qO- 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI | jq -r \"to_entries|map(\\\"\\(.key)=\\(.value|tostring)\\\")|.[]\"); do
  export $s;
done && \
  sed -i'' -e 's|MY_AWS_ACCESS_KEY_ID|'\"$AccessKeyId\"'|' /config/config.toml && \
  sed -i'' -e 's|MY_AWS_SECRET_ACCESS_KEY|'\"$SecretAccessKey\"'|' /config/config.toml && \
  sed -i'' -e 's|MY_AWS_SESSION_TOKEN|'\"$Token\"'|' /config/config.toml

The problem is that these credentials expire and I don't have a way to have athens fetch the updated credentials in the running instance.

Describe the solution you'd like
I'd like athens to be able to fetch these credentials from the environment automatically.

Describe alternatives you've considered
Right now, I have a health check on the athens instances that fetch a package that is stored on S3. When that fails, the health check fails and the instance is destroyed and a new one is provisioned in its place with valid credentials.

enhancement question
Source
picture johnjelinek

Most helpful comment

Yep! I have an idea on what to do. I am working on a PR soon and expect it in a couple of days.

picture manugupt1 on 26 Jun 2019
🎉2

All 60 comments

@johnjelinek
Just for my understanding - the problem is that currently Athens fetches the credentials only once (either from the config file or environment variables) while starting and there is no way to provide new credentials?

marpio picture marpio on 12 Jun 2019

Yes, I bootstrap it out of process and have no way to send updates although
that URL is available during the lifetime of the container and is always
available to get new values. The AWS SDK should have support to make
fetching these credentials seamless if you wanted to provide another way
for Athens to learn about its IAM task role.

On Wed, Jun 12, 2019, 2:02 PM marpio notifications@github.com wrote:

@johnjelinek https://github.com/johnjelinek
Just for my understanding - the problem is that currently Athens fetches
the credentials only once (either from the config file or environment
variables) while starting and there is no way to provide new credentials?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AAGVJCX3RC6KOZHQOW2BXSTP2FB3DA5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXRPGOI#issuecomment-501412665,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGVJCUJOKCJC2VZGERT2E3P2FB3DANCNFSM4HXM6B5Q
.

johnjelinek picture johnjelinek on 12 Jun 2019

Athens s3 config file contains the UseDefaultConfiguration option. If it's set to true then, instead of using credentials.NewStaticCredentials the defaults are being used.
Fom docs:

// The credentials object to use when signing requests. Defaults to a
// chain of credential providers to search for credentials in environment
// variables, shared credential file, and EC2 Instance Roles.

Would that be of any help to you?

marpio picture marpio on 12 Jun 2019

No, I first tried setting that to true and it didn't find the credentials.
That's why I made that bootstrapping process to inject values into the
config file.

On Wed, Jun 12, 2019, 2:58 PM marpio notifications@github.com wrote:

Athens s3 config file contains the UseDefaultConfiguration option. If
it's set to true then, instead of using credentials.NewStaticCredentials
the defaults are being used.
Fom docs:

// The credentials object to use when signing requests. Defaults to a
// chain of credential providers to search for credentials in environment
// variables, shared credential file, and EC2 Instance Roles.

Would that be of any help to you?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AAGVJCTIMOA6S7PKOQ7N2ELP2FIMRA5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXRUAXQ#issuecomment-501432414,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGVJCVYIWATK3PC7CU46VLP2FIMRANCNFSM4HXM6B5Q
.

johnjelinek picture johnjelinek on 12 Jun 2019

Hi,

What happens when you do not add credentials to the config file? If Athens
does not start, I think we should fix this as we can get credentials from

A) Profile (default / otherwise)
B) Config
C) Env (for security token)

Cheers,

Manu Gupta

On Wed, Jun 12, 2019 at 2:09 PM John Jelinek IV notifications@github.com
wrote:

No, I first tried setting that to true and it didn't find the credentials.
That's why I made that bootstrapping process to inject values into the
config file.

On Wed, Jun 12, 2019, 2:58 PM marpio notifications@github.com wrote:

Athens s3 config file contains the UseDefaultConfiguration option. If
it's set to true then, instead of using credentials.NewStaticCredentials
the defaults are being used.
Fom docs:

// The credentials object to use when signing requests. Defaults to a
// chain of credential providers to search for credentials in environment
// variables, shared credential file, and EC2 Instance Roles.

Would that be of any help to you?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
https://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AAGVJCTIMOA6S7PKOQ7N2ELP2FIMRA5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXRUAXQ#issuecomment-501432414
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AAGVJCVYIWATK3PC7CU46VLP2FIMRANCNFSM4HXM6B5Q

.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AADAXDYECMP6YO2ILGF7AXDP2FQY3A5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXRZ4OA#issuecomment-501456440,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADAXD6WYFE7UMH645NX37TP2FQY3ANCNFSM4HXM6B5Q
.

manugupt1 picture manugupt1 on 12 Jun 2019

If I do not add credentials, athens starts and I can get the metrics
endpoint, but all artifact requests respond with 500 because it can't get
the credentials of the ECS task.

On Wed, Jun 12, 2019, 4:29 PM Manu Gupta notifications@github.com wrote:

Hi,

What happens when you do not add credentials to the config file? If Athens
does not start, I think we should fix this as we can get credentials from

A) Profile (default / otherwise)
B) Config
C) Env (for security token)

Cheers,

Manu Gupta

On Wed, Jun 12, 2019 at 2:09 PM John Jelinek IV notifications@github.com
wrote:

No, I first tried setting that to true and it didn't find the
credentials.
That's why I made that bootstrapping process to inject values into the
config file.

On Wed, Jun 12, 2019, 2:58 PM marpio notifications@github.com wrote:

Athens s3 config file contains the UseDefaultConfiguration option. If
it's set to true then, instead of using
credentials.NewStaticCredentials
the defaults are being used.
Fom docs:

// The credentials object to use when signing requests. Defaults to a
// chain of credential providers to search for credentials in
environment
// variables, shared credential file, and EC2 Instance Roles.

Would that be of any help to you?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<

https://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AAGVJCTIMOA6S7PKOQ7N2ELP2FIMRA5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXRUAXQ#issuecomment-501432414

,
or mute the thread
<

https://github.com/notifications/unsubscribe-auth/AAGVJCVYIWATK3PC7CU46VLP2FIMRANCNFSM4HXM6B5Q
>

.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<
https://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AADAXDYECMP6YO2ILGF7AXDP2FQY3A5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXRZ4OA#issuecomment-501456440
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AADAXD6WYFE7UMH645NX37TP2FQY3ANCNFSM4HXM6B5Q

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AAGVJCSVUQZQWJS45O4W6Y3P2FTDPA5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXR3KCQ#issuecomment-501462282,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGVJCTEBUCXRVE3OGQBMTTP2FTDPANCNFSM4HXM6B5Q
.

johnjelinek picture johnjelinek on 13 Jun 2019

@johnjelinek since you're fetching the credentials from an endpoint currently - we could support https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/endpointcreds/

cc: @manugupt1

marpio picture marpio on 13 Jun 2019

That should work. The endpoint will always be the same in AWS ECS (Fargate).

On Thu, Jun 13, 2019, 6:24 AM marpio notifications@github.com wrote:

@johnjelinek https://github.com/johnjelinek since you're fetching the
credentials from an endpoint currently - we could support
https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/endpointcreds/

cc: @manugupt1 https://github.com/manugupt1

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AAGVJCU2VL5BXM3U7AAIDDTP2IU7BA5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXTMFSA#issuecomment-501662408,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGVJCVROOVNX4WINMUDEVDP2IU7BANCNFSM4HXM6B5Q
.

johnjelinek picture johnjelinek on 13 Jun 2019
👍1

Yep! that should work. I would like to work on it. I should have a PR next week.

manugupt1 picture manugupt1 on 13 Jun 2019
👍2

I just had to add this kind of support to another project. This works:

if ECSCredentialsURI, exists := os.LookupEnv("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"); exists {
    endpoint := fmt.Sprintf("169.254.170.2%s", ECSCredentialsURI)
    awsSession := session.New(aws.NewConfig().WithRegion("us-east-1").WithMaxRetries(3))
    awsCreds, err = creds.GetFromEndpoint(*awsSession.Config, awsSession.Handlers, endpoint)
    if err != nil {
        return nil, err
    }
}
johnjelinek picture johnjelinek on 14 Jun 2019
👍1

apparently session.New is deprecated ... maybe use session.NewSession() instead.

johnjelinek picture johnjelinek on 14 Jun 2019

@marpio: any updates?

johnjelinek picture johnjelinek on 24 Jun 2019

@manugupt1 are you still on it?

marpio picture marpio on 24 Jun 2019

yep! sorry did not notice it got assigned to me. I looked into it a bit now.

It seems like we can end up using a combination of Static and Env credentials and if we get a 403, then we re-try by fetching new credentials rather than enforcing an HTTP handler.

This will enable users to inject new credentials in whatever way they see fit (HTTP / otherwise)

What do you think?

Ref: https://github.com/aws/aws-sdk-go/blob/master/aws/credentials/env_provider.go#L36
https://github.com/aws/aws-sdk-go/blob/master/aws/credentials/static_provider.go

manugupt1 picture manugupt1 on 25 Jun 2019

@manugupt1
I was thinking about supporting the endpoint provider b/c it "knows" when the creds expire. That way we wouldn't need to wait for a 403.
Please see https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/endpointcreds/ and https://github.com/gomods/athens/issues/1283#issuecomment-501912410 for an example.

What is your opinion @johnjelinek ?

marpio picture marpio on 25 Jun 2019

Precisely. It knows when it's expired and when you call Get it knows how to renew.

johnjelinek picture johnjelinek on 25 Jun 2019

I am currently in the same situation as @johnjelinek. That feature would be highly appreciated.

cudle picture cudle on 26 Jun 2019

Yep! I have an idea on what to do. I am working on a PR soon and expect it in a couple of days.

manugupt1 picture manugupt1 on 26 Jun 2019
🎉2

FYI: Ran into some issues. Here is a draft PR: https://github.com/gomods/athens/pull/1291

Please start putting in comments so that I can address as I make changes to it.

manugupt1 picture manugupt1 on 28 Jun 2019

@manugupt1 What kind of issues?

marpio picture marpio on 28 Jun 2019

@marpio I have added a PR that is ready for review. https://github.com/gomods/athens/pull/1291/files

Do you think we should also add a unit test that spins up a server and retrieves creds from that server and see if the test succeeds?

manugupt1 picture manugupt1 on 30 Jun 2019

any new updates on this @marpio?

johnjelinek picture johnjelinek on 16 Jul 2019

Hi I have a patch PR, I was travelling and am on-call, so I will try to get
to it asap.
Cheers,

Manu Gupta

On Tue, Jul 16, 2019 at 11:38 AM John Jelinek IV notifications@github.com
wrote:

any new updates on this @marpio https://github.com/marpio?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AADAXD43ZRVMOJEDJPZGYX3P7YISJA5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2BYNHA#issuecomment-511936156,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADAXD7L2H5MK3VZWRUN6D3P7YISJANCNFSM4HXM6B5Q
.

manugupt1 picture manugupt1 on 16 Jul 2019

@marpio / @manugupt1: I'm having difficulty getting this to work. Here's my config change:

CredentialsEndpoint = "http://169.254.170.2/v2/credentials/0357419a-4eea-4eef-8510-16c5c451ba33"

but I'm getting an exception:


  2019-07-22 22:33:05.413265 I | Starting application at port :3000
  panic: runtime error: invalid memory address or nil pointer dereference
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x68aa42]

  goroutine 20 [running]:
  net/http.(*Client).deadline(0x0, 0xc00000e3a0, 0x0, 0x40bad9)
    /usr/local/go/src/net/http/client.go:187 +0x22
  net/http.(*Client).do(0x0, 0xc0003c8f00, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:527 +0xab
  net/http.(*Client).Do(...)
    /usr/local/go/src/net/http/client.go:509
  github.com/aws/aws-sdk-go/aws/corehandlers.sendFollowRedirects(0xc000448c00, 0x154cc80, 0xc0003c8e00, 0xc000448c00)
    /go/pkg/mod/github.com/aws/[email protected]/aws/corehandlers/handlers.go:120 +0x3c
  github.com/aws/aws-sdk-go/aws/corehandlers.glob..func3(0xc000448c00)
    /go/pkg/mod/github.com/aws/[email protected]/aws/corehandlers/handlers.go:112 +0x7f
  github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc000448da8, 0xc000448c00)
    /go/pkg/mod/github.com/aws/[email protected]/aws/request/handlers.go:213 +0x98
  github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc000448c00, 0x0, 0x0)
    /go/pkg/mod/github.com/aws/[email protected]/aws/request/request.go:493 +0x19e
  github.com/aws/aws-sdk-go/aws/credentials/endpointcreds.(*Provider).getCredentials(0xc0004c85c0, 0x1339a04, 0x6, 0x1886101)
    /go/pkg/mod/github.com/aws/[email protected]/aws/credentials/endpointcreds/provider.go:156 +0x2ef
  github.com/aws/aws-sdk-go/aws/credentials/endpointcreds.(*Provider).Retrieve(0xc0004c85c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7f56660b0008, ...)
    /go/pkg/mod/github.com/aws/[email protected]/aws/credentials/endpointcreds/provider.go:114 +0x63
  github.com/aws/aws-sdk-go/aws/credentials.(*ChainProvider).Retrieve(0xc0004ad020, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/pkg/mod/github.com/aws/[email protected]/aws/credentials/chain_provider.go:77 +0xd6
  github.com/aws/aws-sdk-go/aws/credentials.(*Credentials).Get(0xc0004b9b90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/pkg/mod/github.com/aws/[email protected]/aws/credentials/credentials.go:221 +0x157
  github.com/aws/aws-sdk-go/aws/signer/v4.Signer.signWithBody(0xc0004b9b90, 0x0, 0x1836ee0, 0xc00039e6c0, 0x10100, 0x154ff38, 0x0, 0xc0003c8d00, 0x1847100, 0xc00047a2c0, ...)
    /go/pkg/mod/github.com/aws/[email protected]/aws/signer/v4/v4.go:339 +0x28e
  github.com/aws/aws-sdk-go/aws/signer/v4.signSDKRequestWithCurrTime(0xc000448800, 0x154ff38, 0x0, 0x0, 0x0)
    /go/pkg/mod/github.com/aws/[email protected]/aws/signer/v4/v4.go:478 +0x300
  github.com/aws/aws-sdk-go/aws/signer/v4.SignSDKRequest(0xc000448800)
    /go/pkg/mod/github.com/aws/[email protected]/aws/signer/v4/v4.go:424 +0x48
  github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc000448988, 0xc000448800)
    /go/pkg/mod/github.com/aws/[email protected]/aws/request/handlers.go:213 +0x98
  github.com/aws/aws-sdk-go/aws/request.(*Request).Sign(0xc000448800, 0x14154ddd7e, 0x23dcb20)
    /go/pkg/mod/github.com/aws/[email protected]/aws/request/request.go:384 +0xb0
  github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc000448800, 0x0, 0x0)
    /go/pkg/mod/github.com/aws/[email protected]/aws/request/request.go:486 +0x152
  github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsWithContext(0xc00039e6d0, 0x7f5663e4d840, 0xc0003557d0, 0xc0003a6380, 0x0, 0x0, 0x0, 0x185ed80, 0xc0003557d0, 0xc00007ecc0)
    /go/pkg/mod/github.com/aws/[email protected]/service/s3/api.go:4012 +0x199
  github.com/gomods/athens/pkg/storage/s3.(*Storage).Exists(0xc0004c8640, 0x185ed80, 0xc0003557a0, 0xc0003de9c0, 0x24, 0xc0003de9f0, 0x22, 0x185ed00, 0x0, 0x0)
    /go/src/github.com/gomods/athens/pkg/storage/s3/checker.go:29 +0x31e
  github.com/gomods/athens/pkg/storage/s3.(*Storage).Info(0xc0004c8640, 0x185ed80, 0xc000355770, 0xc0003de9c0, 0x24, 0xc0003de9f0, 0x22, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/gomods/athens/pkg/storage/s3/getter.go:21 +0x128
  github.com/gomods/athens/pkg/download.(*protocol).Info(0xc0004c8880, 0x185ed80, 0xc0003556e0, 0xc0003de9c0, 0x24, 0xc0003de9f0, 0x22, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/gomods/athens/pkg/download/protocol.go:159 +0x131
  github.com/gomods/athens/pkg/download/addons.(*withpool).Info.func1()
    /go/src/github.com/gomods/athens/pkg/download/addons/with_pool.go:73 +0x8a
  github.com/gomods/athens/pkg/download/addons.(*withpool).listen(0xc0004d2320)
    /go/src/github.com/gomods/athens/pkg/download/addons/with_pool.go:46 +0x49
  created by github.com/gomods/athens/pkg/download/addons.(*withpool).start
johnjelinek picture johnjelinek on 23 Jul 2019

One of the tricky things about running this in AWS ECS Fargate is that the URL changes every time. I have to do something like this in a sidecar sed -i'' -e 's|MY_AWS_CREDENTIALS_ENDPOINT|http://169.254.170.2'\"$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI\"'|' /config/config.toml to populate the config file before starting athens. I'm not sure if you know of a way to get env vars to interpolate when the value of an env var itself contains an env var, i.e.: AWS_CREDENTIALS_ENDPOINT=169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI.

johnjelinek picture johnjelinek on 23 Jul 2019

Is it possible to see how you have set up on farget and I would love to set it up and actually fix this issue instead of going through the docs and fixing it

Cheers
Manu

From: John Jelinek IV notifications@github.com
Sent: Monday, July 22, 2019 3:48 PM
To: gomods/athens
Cc: Manu Gupta; Mention
Subject: Re: [gomods/athens] Use AWS ECS credentials (#1283)

One of the tricky things about running this in AWS ECS Fargate is that the URL changes every time. I have to do something like this in a sidecar sed -i'' -e 's|MY_AWS_CREDENTIALS_ENDPOINT|http://169.254.170.2'\"$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI\"'|' /config/config.toml to populate the config file before starting athens. I'm not sure if you know of a way to get env vars to interpolate when the value of an env var itself contains an env var, i.e.: AWS_CREDENTIALS_ENDPOINT=169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AADAXD7Y3MFY4I7H3VPPOS3QAY2NNA5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2RMK2Y#issuecomment-513983851, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AADAXD52T3BK7A37T3P2VVTQAY2NNANCNFSM4HXM6B5Q.

manugupt1 picture manugupt1 on 23 Jul 2019

I'm reopening the issue to track your problem @johnjelinek

marpio picture marpio on 23 Jul 2019

This was my original proposal, looks like that will be the easiest way to get the URL and add it to the handler. https://github.com/gomods/athens/issues/1283#issuecomment-505278544

We can add a force override for changing URLs so that we can use it when absolutely require: for example FORCE_SIDE_CAR_CREDSENDPOINT

manugupt1 picture manugupt1 on 24 Jul 2019

What do you think @johnjelinek @marpio

manugupt1 picture manugupt1 on 26 Jul 2019

@johnjelinek @manugupt1
I don't have much experience with AWS so just for my understanding - the problem seems to be that not only do the credentials expire but so does the endpoint providing the credentials, right?
I'm also wondering about the segmentation fault - does it happen when the endpoint expires or on the first request?

Currently we're setting the credentials endpoint at the start of Athens and so when it expires we're out of luck.
Is there a way to know that the endpoint changed? I couldn't find anything regarding that in the sdk but I might have missed it.
Otherwise could we check some env var and reinitialise the endpoint address?

marpio picture marpio on 26 Jul 2019

The endpoint that provides the credentials does not expire, it is available for the life of the Fargate service, but the panic should be caught.

johnjelinek picture johnjelinek on 26 Jul 2019

I do not have much experience either. I will try and set it up and see if I can replicate the panic.

Cheers
Manu

From: John Jelinek IV notifications@github.com
Sent: Friday, July 26, 2019 5:31 AM
To: gomods/athens
Cc: Manu Gupta; Mention
Subject: Re: [gomods/athens] Use AWS ECS credentials (#1283)

The endpoint that provides the credentials does not expire, it is available for the life of the Fargate service, but the panic should be caught.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AADAXD76U7W5ZMQ7DUHYZSLQBLVARA5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD24N7PQ#issuecomment-515432382, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AADAXDZH4FI726VTDRH3SRTQBLVARANCNFSM4HXM6B5Q.

manugupt1 picture manugupt1 on 26 Jul 2019

It should be pretty easy to mock, any HTTP endpoint that serves a few key-pairs in JSON should help you repro a panic.

johnjelinek picture johnjelinek on 26 Jul 2019

Let me try that and get back. I really appreciate your patience. Thank you

Cheers
Manu

From: John Jelinek IV notifications@github.com
Sent: Friday, July 26, 2019 7:53 AM
To: gomods/athens
Cc: Manu Gupta; Mention
Subject: Re: [gomods/athens] Use AWS ECS credentials (#1283)

It should be pretty easy to mock, any HTTP endpoint that serves a few key-pairs in JSON should help you repro a panic.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AADAXD3BF7RAGX56BFJWK2TQBMFWXA5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2424NI#issuecomment-515485237, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AADAXD4H3XY3PLQASXLHBD3QBMFWXANCNFSM4HXM6B5Q.

manugupt1 picture manugupt1 on 26 Jul 2019

@manugupt1: how'd the mock HTTP endpoint work for you?

johnjelinek picture johnjelinek on 4 Aug 2019

Looks like #1291 was supposed to close this but it didn't. I'm going to close now.

@johnjelinek if this is still not panic-ing (https://github.com/gomods/athens/pull/1291#issuecomment-513987374) or not working, can you open a new issue?

arschles picture arschles on 5 Aug 2019

@arschles: I don't understand. #1291 was not implemented in a way that would address this issue. That's what this communication was addressing. So you want a new issue to mention the same problem?

johnjelinek picture johnjelinek on 5 Aug 2019

@arschles #1291 didn't address this issue (we're getting a panic). I think we can leave it open until we fix the current implementation.

marpio picture marpio on 5 Aug 2019

@johnjelinek I have been able to get a 403 on this with endpoint creds. Can you verify if it works with real ones: https://github.com/gomods/athens/pull/1325

manugupt1 picture manugupt1 on 6 Aug 2019

Cool! I think that looks good. Can you push this to the canary container so I can pull it in Fargate?

johnjelinek picture johnjelinek on 6 Aug 2019

@johnjelinek canary should be pushed on every master build so you should be able to test it.

marpio picture marpio on 6 Aug 2019

@marpio: works great! :shipit: The only other part that would make this nicer is if it could figure out the endpoint in process instead of me having to use a sidecar to generate the URL to pass into the config.

johnjelinek picture johnjelinek on 8 Aug 2019

@johnjelinek glad to hear that!
Would using an environment variable be an option for you?
https://github.com/gomods/athens/blob/a336c5552ff5ad03ebd0b269ac0792a2b8ed8d56/pkg/config/s3.go#L11

marpio picture marpio on 8 Aug 2019

@marpio: no, the env var would have to be set a deploy-time, but AWS generates it dynamically, so I generate the URL at run-time in a sidecar, modifying the config file in a shared volume, and then I start the athens container -- however, the URL is always discoverable at: http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, but I can't pass in a reference to another env var to AWS_CREDENTIALS_ENDPOINT.

Do you think it makes sense to add support for resolving other env vars so I could set it like this:

AWS_CREDENTIALS_ENDPOINT=http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

Or do you think it makes more sense to embed this endpoint as a default URL, since it's always available here when running in AWS Fargate (and then I don't have to pass in an endpoint at all)?

johnjelinek picture johnjelinek on 8 Aug 2019

@johnjelinek Can you be a bit more clear?

Is it a new URI that it generates or do the tokens expire after a while?

manugupt1 picture manugupt1 on 11 Aug 2019

A service, in Fargate, is an essential container and a collection of sidecars that run to provide functionality. Every time a service starts in Fargate, the environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI changes. The environment variable is the same for all containers of the service. So, right now, I have a sidecar that fetches the value of AWS_CONTAINER_CREDENTIALS_RELATIVE_URI and appends it to http://169.254.170.2 and then I take all that and modify the config file, so when the essential athens container starts, it's already configured with where to fetch credentials from. That provided endpoint already has everything needed to refresh credentials as long as the Fargate service does not end. The URL will never change for that instance of the service.

To simplify, if athens had a default to make AWS_CREDENTIALS_ENDPOINT point to http://169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, then I could remove my sidecar container from boot and have the same functionality. Hopefully, this explanation makes more sense. Let me know if you have further questions @manugupt1.

johnjelinek picture johnjelinek on 12 Aug 2019

@johnjelinek to keep track I am reopening the issue.

Can you look at this PR and see if it makes sense: https://github.com/gomods/athens/pull/1337/files

manugupt1 picture manugupt1 on 13 Aug 2019

I'm not sure this makes sense. It doesn't look like it's linking to anything from the environment. I think what you're trying to implement is an environment variable alias. The $AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is already an environment variable, it's not part of the URL. It might make more sense to see if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI exists as an env var and if so, evaluate AWS_CREDENTIALS_ENDPOINT to whatever that evaluates to instead of requiring and config input.

johnjelinek picture johnjelinek on 13 Aug 2019

We use envconfig that should help override env vars.

https://github.com/kelseyhightower/envconfig/blob/master/README.md

That should work! Is there an easy way for you to test this branch? If not, let’s figure something out

Cheers
Manu

From: John Jelinek IV notifications@github.com
Sent: Tuesday, August 13, 2019 5:53 AM
To: gomods/athens
Cc: Manu Gupta; State change
Subject: Re: [gomods/athens] Use AWS ECS credentials (#1283)

I'm not sure this makes sense. It doesn't look like it's linking to anything from the environment. I think what you're trying to implement is an environment variable alias. The $AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is already an environment variable, it's not part of the URL. It might make more sense to see if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI exists as an env var and if so, evaluate AWS_CREDENTIALS_ENDPOINT to whatever that evaluates to instead of requiring and config input.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHubhttps://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AADAXD6S6RQQTOSX52GIV4TQEKVDPA5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4FRZYA#issuecomment-520821984, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AADAXD7FZDTSJLHO7HVT4PTQEKVDPANCNFSM4HXM6B5Q.

manugupt1 picture manugupt1 on 13 Aug 2019

@manugupt1: If you could push your latest changes to the canary container release, I can pull it in and try it out ... I suspect it won't work, but let's find out!

johnjelinek picture johnjelinek on 13 Aug 2019

@manugupt1: I left comments on your PR

johnjelinek picture johnjelinek on 13 Aug 2019

@johnjelinek It should work as I tested out printing out exported env vars. I will open the PR for review and see if everyone is okay to merge it in.

manugupt1 picture manugupt1 on 14 Aug 2019
👍1

cool, when it gets to the canary container, I'll try it out

johnjelinek picture johnjelinek on 14 Aug 2019

It should be on canary now can you test it John! Thanks

Cheers
Manu

From: John Jelinek IV notifications@github.com
Sent: Wednesday, August 14, 2019 9:51 AM
To: gomods/athens
Cc: Manu Gupta; Mention
Subject: Re: [gomods/athens] Use AWS ECS credentials (#1283)

cool, when it gets to the canary container, I'll try it out

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AADAXDZPYK3EDGY7DLGGYS3QEQZY5A5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4JNGSI#issuecomment-521327433, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AADAXD64J43K44CQMZBU4N3QEQZY5ANCNFSM4HXM6B5Q.

manugupt1 picture manugupt1 on 21 Aug 2019

Excellent, I should be able to test it tomorrow

johnjelinek picture johnjelinek on 21 Aug 2019

@manugupt1: canary works great. I removed all my sidecars and it still works with this env var:

  {
    "name": "AWS_CREDENTIALS_ENDPOINT",
    "value": "http://169.254.170.2"
  }

I think can be closed now. It might be nice to document somewhere that http://169.254.170.2 is the default Fargate endpoint for everyone.

johnjelinek picture johnjelinek on 22 Aug 2019
🎉1

Is it a global endpoint or is it region specific?

Cheers
Manu

From: John Jelinek IV notifications@github.com
Sent: Thursday, August 22, 2019 9:32 AM
To: gomods/athens
Cc: Manu Gupta; Mention
Subject: Re: [gomods/athens] Use AWS ECS credentials (#1283)

@manugupt1https://github.com/manugupt1: canary works great. I removed all my sidecars and it still works with this env var:

{
"name": "AWS_CREDENTIALS_ENDPOINT",
"value": "http://169.254.170.2"
}

I think can be closed now. It might be nice to document somewhere that http://169.254.170.2 is the default Fargate endpoint for everyone.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/gomods/athens/issues/1283?email_source=notifications&email_token=AADAXD2TMQOEZOCQNTCNWHTQF25SRA5CNFSM4HXM6B52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD45U46I#issuecomment-523980409, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AADAXD35QFHU6SWC4JR6TZTQF25SRANCNFSM4HXM6B5Q.

manugupt1 picture manugupt1 on 22 Aug 2019

global

johnjelinek picture johnjelinek on 22 Aug 2019

Makes sense! I looked at the docs. It does not get there specifically. However, I created two issues for these:

https://github.com/gomods/athens/issues/1349
https://github.com/gomods/athens/issues/1350

Let's track them there!

thanks!

manugupt1 picture manugupt1 on 23 Aug 2019

@johnjelinek : I want to use the aws credentials in AWS Fargate. I am using a golang docker container to read some AWS SSM Params and need credentials. Can you please help me with this snippet..

sess := session.Must(session.NewSessionWithOptions(session.Options{
        SharedConfigState: session.SharedConfigEnable,
    }))
    svc := ssm.New(sess)

I am unable to access any ssm secrets.

venkz picture venkz on 31 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

zkry picture zkry  ·  3Comments
robjloranger picture robjloranger  ·  4Comments
arschles picture arschles  ·  4Comments
komuw picture komuw  ·  3Comments
chriscoffee picture chriscoffee  ·  3Comments