Athens: Proxy: use in-mem auth instead of .netrc

[marwan-at-work]

A Proxy needs to authenticate with git or whatever vcs upstream to pull private company repos. Currently we do this through the .netrc file but this means that your credentials are chillin on disk. For development, this is fine and is probably already set for users through Access Keychain or .netrc without them realizing it. But for deployment we should make sure we have a clear/secure way of authenticating an Athens Proxy. https://confluence.atlassian.com/bitbucketserver/permanently-authenticating-with-git-repositories-776639846.html

[fedepaol]

Working on this one.

I tinkered a bit with cache credential helper. It's not possible to force it to have an infinite timeout.
The docs does not tell which is the max timeout but looking at the sources (https://github.com/git/git/blob/53f9a3e157dbbc901a02ac2c73346d375e24978c/credential-cache.c#L100) it's an int, so it might vary between 2^31 and 2^63 depending on the compiler git was compiled with.

@marpio suggested to implement the git-credential api by ourlselves (https://git-scm.com/docs/git-credential) which let git call an executable that will give back the credentials.

A draft of the workflow could be:

• the proxy itself (the executable) implements the api if launched with a given parameter
• before starting, the proxy (the long running server) checks for the credentials in a path (filled by a secret) and stores them in memory (as the git cache would do)
• the proxy (the server one) instructs git to use it's executable as a credential helper using git config --global credential.helper 'path/to/proxy -enablegitcredentialapiparameter' which calls the proxy (the server one)

In this way all the logic is contained within the proxy, the credentials would be stored inside the memory of the long running proxy as they would be in the cache deamon (https://github.com/git/git/blob/53f9a3e157dbbc901a02ac2c73346d375e24978c/credential-cache--daemon.c )

The two processes can communicate via a system socket the same way the git cache command does with the credential cache daemon. I would also try to steal the same permission logic.

An alternative would be to use the cache with the longest avaliable timeout (2^31 worst case scenario), but the way it works is by caching the credentials filled by the user the first time it pulls from a repo, so I think it would not suit our use case.

The api to implement is pretty simple and @marpio give us a good pointer to take inspiration blatantly copy from which is https://github.com/kelseyhightower/hub-credential-helper

Not sure I like having the proxy implement a completely different functionality under a parameter (the credential store) but this has the advantages of being able to know where the executable is and not having to ship something else together with the proxy.

Also, not sure if the secret file needs to be deleted (using shred maybe) or if it's ok if we leave it where we find it.

As a side note, we could also try this https://stackoverflow.com/questions/5343068/is-there-a-way-to-skip-password-typing-when-using-https-on-github/18362082#18362082
but the setup would be more trickier, and it would require to use the passphrase the first time we go get which is not ideal. It would be a lot less code to write but it's less elegant imho.

[fedepaol]

Updated the comment with the results of the discussion with @marpio

[fedepaol]

Dug a little more on secrets handling.
@marwan-at-work suggested to read a file (k8s way) or through vault.
While k8s provides them as a file (or env variables), azure, gcp, vault and aws provide them through rest calls (or via an sdk).
With @michalpristas we agreed that having the proxy binary serving as credential helper sacrifices readability, so the idea is having the git helper as a separate binary.

What we need is an executable that responds to the git credential helper api .
It's an executable that is invoked every time a go get is executed and must returns the credentials to the std output. Every request may be forwarded directly to the secret manager of the platform (or read the secret every time), or we could have a daemon baking (and caching) the requests. The daemon can be the proxy itself.

[marpio]

I think we could read the secret on each request. This would allow us to change the credentials without restarting the proxy (if daemon is the proxy). Maybe we could implement different secret providers (k8s, vault, gcp etc.) and choose one just by passing the flag to the binary.
'path/to/credential-helper -k8s/vault/gcp'

[arschles]

@fedepaol talked about this a bunch on slack just now, and he agreed to let me submit my proposal here based on what we talked about, even thought it says a lot of what he already said (thanks @fedepaol!) Here it is:

1. First, I think we should only support files. To get this working, we could tell the user to configure a git credential file store before they start Athens
   
   
   
   • This would satisfy a lot of use cases on Kubernetes because you can mount a secret into a file (and you can back secrets with Vault)
   
   
   • You wouldn't be able to integrate with any API based systems though
   
   
2. Next, I think we should support systems that use REST APIs. To get this working, I still think we should tell the user to configure a git credential helper before they run athens, but I propose we implement external binaries that support the credential API and keep them logically separate from Athens.
   
   
   
   • I wonder if some of these already exist?
   
   
   • For the ones that still exist, I propose we implement them in separate repositories inside of the gomods organization
   
   

The final part is that I would like to make the user responsible for configuring the credential store or helper before they run Athens. That means that we'll need to write more documentation (as @fedepaol said on Slack). Overall I think it's worth doing the credential helpers separately from Athens because they let us keep a cleaner abstraction between Athens and the credential handling

@fedepaol you are asleep now (I hope!) but I know that next time you're free for Athens-ing, you're going to look at this. For everyone else - that was a log of "I think" in there - nothing here is definitive and I want to hear thoughts!

[fedepaol]

Agree with all the line.
@arschles should I mess with the docs in order to document the process of configuring a credential store mapped as a secret?

[fedepaol]

Update from my experiments:

• it's possible to feed the store with a handcrafted store file. The file has a line for each remote, with the following format: https://fedepaol:[email protected]
• we need to be careful, because the get is blocking if it requires authentication, but if we set GIT_TERMINAL_PROMPT=0 the get will fail instead of asking for credentials. In case of privare repos, if the store file is already filled it works as expected and the clone works like a charm.

TL;DR:

• set a store file with the format I included above
• configure the store helper like this : git config --global credential.helper "store --file /path/to/store/mystore.store"
• have the store file mapped as a secret
• set GIT_TERMINAL_PROMPT=0 in order to avoid blocking prompts to the user if the proxy tries to fetch from an unauthorized private repo

[fedepaol]

Just another quick question: we currently pass a .netrc file which is not in the home to the proxy that copies it into the home (https://github.com/gomods/athens/blob/master/cmd/proxy/actions/app.go#L59).

Shouldn't the user be in charge of this? Wouldn't have been easier for the user to place it directly inside the home instead of passing another parameter to athens?
With the store helper, does it make sense to remove the .netrc behaviour completely?

Finally, where should I document this? Inside the "Installs" section?

[marwan-at-work]

@fedepaol no, if you mount .netrc directly into the home, all the sibling files in that directory get removed (if you're on K8S). So we have to have it in a temp fs then copy it onto disk.

Once we have the in-mem option figured out, we'll see whether it's good enough to remove the .netrc option for the next couple of months we should definitely keep .netrc.

[fedepaol]

@marwan-at-work thanks for the explanation!
Re: in-mem, the idea was not to keep the secret in memory but reading it every time from the mounted secret, since keeping it in memory would need a long running daemon serving a custom made git credential helper. Does this make sense to you?

[marwan-at-work]

@fedepaol will that be any more secure than .netrc?

[fedepaol]

@marwan-at-work not sure anymore, which is the reason I asked if we could avoid using the store credential helper and relying directly on a secret backed .netrc, which is what we are currently doing.

The only advantage of not moving the secret from where k8s place it would be the fact that it is read every time and for this reason athens would not need to be restarted in order to change the credentials.
Apart from that, if we are on k8s, I don't think there would be much difference from a security point of view between the two approaches since I expect the home fs to be ephemeral.

I still think that this should documented properly (including GIT_TERMINAL_PROMPT=0) and that custom git credential helpers would help (sic) in case we won't be on k8s but we will want to rely on platform dependent secret apis, as discussed with @arschles

[marpio]

I think the difference would be that on k8s the file containing secrets would stay on tempfs (RAM-backed filesystem). If we could use the .netrc file from tempfs it wouldn't make much difference, I guess. Does .netrc file have to be in the home dir?

[fedepaol]

I think so (re .netrc file in the home).
Also, I am not sure where the home gets mounted but the common practice is not to map it with persistent volumes, so it would go away when the pod gets shut down and the only way to access it would be to have access to the container.

[marwan-at-work]

i'm closing this as we are going with the .netrc file. If someone feels like this is not enough, feel free to reopen.