Athens: CI/CD Not Building Proxy Image Correctly

[arkadyb]

Hi,

Id like to use athens for corp level go modules proxy, but cant seem to find a way get it authenticated against our private repos in http://github.com/[CompanyName]
Is there some way to use my GitHub deploy key in netrc file? Or should i be using some other auth type? Thanks for help!

[michalpristas]

Hey @arkadyb, thanks for raising an issue.

please check this doc page https://docs.gomods.io/install/install-on-kubernetes/#give-athens-access-to-private-repositories-via-github-token-optional
and let me know if this answers your question

[arkadyb]

Hey @michalpristas , thanks for quick answer, but it does not seem to work for me. im getting
time="2018-10-29T07:19:27Z" level=error msg="v0.0.0-20180926053524-0abb149210af.mod not found in /tmp/athens603234636/pkg/mod/cache/download/github.com/[company-name]/utils-go/@v" http-method=GET http-path="/github.com/[company-name]/utils-go/@v/v0.0.0-20180926053524-0abb149210af.info" http-url="/github.com/[company-name]/utils-go/@v/v0.0.0-20180926053524-0abb149210af.info" kind="Internal Server Error" module=github.com/[company-name]/utils-go operation=download.versionInfoHandler ops="[download.versionInfoHandler protocol.Info protocol.fillCache goget.Version goGetFetcher.Fetch module.getSources module.checkFiles]" version=v0.0.0-20180926053524-0abb149210af handler: GET /github.com/[company-name]/utils-go/@v/v0.0.0-20180926053524-0abb149210af.info [500]

Is there some way to track down reasons behind Internal Server Error?

[michalpristas]

Have you restarted the proxy?
this setting should prepare netRc file. are you able to check this
@marwan-at-work you have this working right?

[fedepaol]

Also @arkadyb , could you manually throw a .netrc file in your home to check if that works?
The .netrc file should be filled like this

[michalpristas]

also, one more important question comes to mind. which OS are you using?
our netrc implementations are not windows friendly

[arkadyb]

@fedepaol, @michalpristas , ive tried to run on local (macos) and Docker. .netrc file has been created at home, but i still face same issues, i get back 500 error when athens tries to access my private repos. Does it metter repo belongs to org and not my private user?

Just to note, when i pass ATHENS_GITHUB_TOKEN into docker container run command, it does not create .netrc file in my home folder.

[arkadyb]

I was finally able to run it. Problem is - when u run docker image from dockerhub, it does not seem to create required .netrc file even when i pass respective git hub security token. i was able to build image from Dockefile in repo and this one works.
Seem to be an issue of DockerHub image...

[marwan-at-work]

@arkadyb what is the image that used from docker hub? Also, how did you pass the GITHUB_TOKEN or NETRC_PATH variables to the image? Can you please provide a reproduction example. Thanks

[arkadyb]

@marwan-at-work , docker hub image - https://hub.docker.com/r/gomods/proxy/
as per command it is: docker run -d ... -e ATHENS_GITHUB_TOKEN=[my-github-token] --name athens-proxy --restart always -p 3000:3000 gomods/proxy:latest

[arkadyb]

To repro simply use:
export ATHENS_STORAGE=~/athens-storage mkdir -p $ATHENS_STORAGE docker run -d -v $ATHENS_STORAGE:/var/lib/athens \ -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \ -e ATHENS_GITHUB_TOKEN=[my-github-token] \ -e ATHENS_STORAGE_TYPE=disk \ --name athens-proxy \ --restart always \ -p 3000:3000 \ gomods/proxy:latest

Then exec into running container and make sure file is not there. Be sure to not use image from ur docker cache when trying to repro.

[michalpristas]

what i think you need to do is to copy the token into your container. during build, there is no such process except copying of config file.
so i suggest to share a dir with your container pass it as volume -v /var/lib/tokens:/my/path/tokens
and then set -e ATHENS_GITHUB_TOKEN=/var/lib/tokens/github_token
i actually havent tried this. it's just from top of my head

[marwan-at-work]

I think the docker hub image is an old or incorrect one. If you curl localhost:3000/ against the docker hub container, you get the old html response instead of the simple string one.

[arkadyb]

@michalpristas, it works just fine with docker image built from sources.
@marwan-at-work, with docker hub image i get huge logo and

Welcome to the Athens proxy
github.com/gomods/athens © 2018

With the one build from sources just "Welcome to The Athen Proxy"

[marwan-at-work]

@arkadyb exactly. I think our CI/CD is not correctly building the docker image.

@michalpristas with the latest source code, providing the ATHENS_GITHUB_TOKEN should be good enough as it will create an appropriate .netrc file inside the container itself.

[michalpristas]

oh i forgot it is a token itself not a path, sorry my bad

[marwan-at-work]

cc: @arschles

[arschles]

oh no! I'll check into this and see what's going on here, thanks for bringing this up everyone!

[arschles]

I did some investigation. As everyone knows, every commit push to master (i.e. PR merges), a canary image and an image with the SHA as a tag get pushed. I looked at https://github.com/gomods/athens/commit/539072c6e35ef6afae643efffe5cafd097a2cf52, since that changed the home page from Welcome to The Athen Proxy to Welcome to The Athens Proxy.

Right now, gomods/proxy:canary and gomods/proxy:539072c should both output Welcome to The Athens Proxy (i.e. the change should have been pushed), and the locally built image should also have the same change (which would be gomods/proxy:latest).

Here's the output after I ran make docker from 657c9d0:

➜  athens git:(log-push-script) docker pull gomods/proxy:canary && docker pull gomods/proxy:539072c
canary: Pulling from gomods/proxy
Digest: sha256:a9e940945858261d9ddc39489feec9bb5e161cdd7752e6370f2f3ac1408b3cb8
Status: Image is up to date for gomods/proxy:canary
539072c: Pulling from gomods/proxy
Digest: sha256:f85b5d32cf17b20a3acd91b6c0dab42152f735881772c7620be2513c57663e26
Status: Image is up to date for gomods/proxy:539072c
➜  athens git:(log-push-script) docker run -d --rm -p 3000:3000 --name "sha" gomods/proxy:539072c
a0a8482eed258b27cedd247b161fc6c3dd345550b775491877d0de9625ac0246
➜  athens git:(log-push-script) curl localhost:3000
"Welcome to The Athens Proxy"
➜  athens git:(log-push-script) docker rm -f sha
sha
➜  athens git:(log-push-script) docker run -d --rm -p 3000:3000 --name "canary" gomods/proxy:canary
3a971455914a392656c5576318aefd79de22947182ac94bdd394074c628bc184
➜  athens git:(log-push-script) curl localhost:3000
"Welcome to The Athens Proxy"
➜  athens git:(log-push-script) docker rm -f canary
canary
➜  athens git:(log-push-script) docker run -d --rm -p 3000:3000 --name "latest" gomods/proxy:latest
f1cd44e1bbbc059bb6d7e24a1e3d8b9fc8832db12625d3682ae2185b1bd8b551
➜  athens git:(log-push-script) curl localhost:3000
"Welcome to The Athen Proxy"
➜  athens git:(log-push-script) docker rm -f latest
latest

This isn't really crazy-scientific, but it shows at least that the content of the / route is the same on local (as of 539072c6e35ef6afae643efffe5cafd097a2cf52, which is the most recent right now) as it is in the image where it should have changed last.

I've created #841 to help us find out if there's something else going on that I missed, and I created #842 to help us know what code every binary and every image came from.

One possible problem here is that canary is, so you have to pull them after every master commit, because CI pushes them then. Otherwise you'll be using an outdated image from who-knows-when.

Another thing is that gomods/proxy:latest seems to be pull-able, but I'm pretty sure it shouldn't be because it's not listed here - Dockerhub might be automatically assigning latest to the most recently pushed (which would be lame). Because of both of these things, we should encourage (and fix any docs) folks to use the immutable images with SHA tags.

Does anyone see anything I missed?

cc/ @marwan-at-work @michalpristas @arkadyb @fedepaol

[marpio]

@arschles
The "latest" tag on dockerhub points to our v0.1.0 release.

https://github.com/gomods/athens/blob/657c9d04a35197a23667a064804deda38bcc0256/scripts/push-docker-images.sh#L24-L25
push-docker-images.sh tags the the last version - in this case v0.1.0 - with the latest tag additionally to v0.1.0

So it actually works as designed. The question is do we wont to leave it that way. If so we should document it.

[chriscoffee]

Just a bit of further information, latest was last tagged on 2018-08-27T14:45:24.541023Z.

Also, here's a list of everything in the registry, along with when they were last updated.

"canary 2018-11-01T02:31:58.725978Z"
"ed05805 2018-11-01T02:31:57.258963Z"
"8f4e3dc 2018-10-31T22:27:23.921407Z"
"657c9d0 2018-10-31T17:15:21.828329Z"
"c6c5c5a 2018-10-30T23:09:19.106762Z"
"539072c 2018-10-30T20:53:02.082194Z"
"e9dd2f2 2018-10-30T13:44:01.668527Z"
"2f41b2f 2018-10-29T13:22:31.349513Z"
"7421695 2018-10-26T20:18:31.801536Z"
"b600753 2018-10-26T20:00:11.569262Z"
"3a190df 2018-10-26T18:56:43.111318Z"
"rename-proxy 2018-10-26T18:26:23.621789Z"
"40673b3 2018-10-26T18:26:22.279054Z"
"b90ca97 2018-10-26T18:17:46.744512Z"
"5fbf278 2018-10-26T06:37:21.517052Z"
"dbac943 2018-10-25T22:46:34.912418Z"
"322ff26 2018-10-24T21:41:55.841331Z"
"117698d 2018-10-24T20:06:55.937107Z"
"01104d3 2018-10-24T19:28:50.638536Z"
"24fb4b7 2018-10-24T11:57:00.469674Z"
"a542115 2018-10-24T11:50:37.112410Z"
"2d589f1 2018-10-24T09:06:10.194491Z"
"d26b99d 2018-10-23T23:56:31.203993Z"
"ccf5940 2018-10-23T23:10:15.447139Z"
"1dc33bd 2018-10-22T21:45:52.102117Z"
"532e93e 2018-10-22T20:27:24.320652Z"
"7b65736 2018-10-19T00:03:28.211771Z"
"f665b0c 2018-10-18T21:41:38.442788Z"
"639bd23 2018-10-18T21:31:52.659497Z"
"87a53db 2018-10-18T19:52:50.067619Z"
"8ed0099 2018-10-18T19:40:00.716098Z"
"aeb39bf 2018-10-18T19:21:39.907923Z"
"f6088cd 2018-10-17T14:42:10.436925Z"
"480d8c8 2018-10-16T01:51:26.873078Z"
"f5259a3 2018-10-16T00:39:46.304145Z"
"13a2e1b 2018-10-16T00:34:50.014770Z"
"4cda44c 2018-10-15T12:11:30.178313Z"
"13083b9 2018-10-15T09:20:44.424208Z"
"4b6da42 2018-10-11T20:25:43.153681Z"
"c751032 2018-10-11T14:05:30.428238Z"
"dbbfdf8 2018-10-11T11:15:46.354775Z"
"73a2013 2018-10-11T11:07:31.417760Z"
"73db12a 2018-10-10T16:54:05.419133Z"
"a2dfe47 2018-10-10T16:45:51.166388Z"
"48e73f7 2018-10-09T14:22:09.686010Z"
"fc9486b 2018-10-08T21:39:46.833246Z"
"2768c58 2018-10-08T20:03:11.182035Z"
"65a3600 2018-10-08T19:04:35.291804Z"
"6d6ff27 2018-10-08T11:35:12.236732Z"
"b795f8b 2018-10-06T02:46:36.423919Z"
"89d9dd2 2018-10-05T16:00:15.122353Z"
"c31dfcb 2018-10-05T13:42:43.573578Z"
"6753def 2018-10-04T23:40:40.875148Z"
"97d8013 2018-10-04T23:21:44.158583Z"
"337488c 2018-10-04T22:52:54.475555Z"
"73e45cb 2018-10-04T02:53:05.285197Z"
"75b3bd7 2018-10-03T21:30:13.556922Z"
"dfe939e 2018-10-03T18:19:16.759841Z"
"eac2b82 2018-10-03T16:53:13.650002Z"
"31b66be 2018-10-03T16:26:53.491105Z"
"39d10b3 2018-10-03T16:21:29.611803Z"
"b1502ba 2018-10-03T15:59:38.387459Z"
"d370b5a 2018-10-02T11:43:59.781868Z"
"cdced92 2018-10-01T19:03:34.551300Z"
"bc4e533 2018-10-01T18:58:49.483050Z"
"4123c32 2018-10-01T18:22:11.726243Z"
"7ee25d5 2018-10-01T07:14:38.319123Z"
"e5b3073 2018-09-29T17:31:20.675501Z"
"89c09b1 2018-09-28T18:35:34.564403Z"
"963ecdd 2018-09-28T13:59:30.496696Z"
"187ab9d 2018-09-26T21:16:59.309023Z"
"d663e73 2018-09-25T12:08:35.486396Z"
"524912a 2018-09-24T22:59:24.076537Z"
"536257c 2018-09-22T05:41:42.741551Z"
"2c3d72d 2018-09-22T04:33:39.177321Z"
"83281a8 2018-09-21T20:36:56.187745Z"
"7a32858 2018-09-21T06:43:23.846193Z"
"e574d28 2018-09-20T20:54:20.925477Z"
"bcc896f 2018-09-20T20:43:38.044237Z"
"29b6b6f 2018-09-20T19:37:00.420717Z"
"5449649 2018-09-20T17:23:49.494054Z"
"5fd3e12 2018-09-20T16:58:40.105032Z"
"d314c78 2018-09-20T16:45:15.513524Z"
"7e881bf 2018-09-20T15:41:32.148206Z"
"5dcaeb8 2018-09-20T15:11:42.412307Z"
"e1b7fbc 2018-09-20T15:03:48.940583Z"
"d6a1cda 2018-09-20T14:37:08.877935Z"
"8174795 2018-09-20T07:04:00.459460Z"
"3db7a66 2018-09-20T03:49:24.743309Z"
"d03eb7b 2018-09-19T08:35:51.276337Z"
"2883921 2018-09-19T06:49:26.011707Z"
"29cb5c9 2018-09-02T05:59:07.590435Z"
"6bf0de7 2018-09-02T00:06:11.117753Z"
"fe63c38 2018-09-01T19:50:51.267844Z"
"goreleaser 2018-09-01T19:40:16.160946Z"
"6065ae7 2018-09-01T19:40:14.692859Z"
"ff5eac1 2018-09-01T19:24:59.013168Z"
"249a1a6 2018-09-01T18:44:22.813389Z"
"a89552f 2018-09-01T17:44:04.893737Z"
"a2a2d93 2018-09-01T10:01:50.206733Z"
"5bb28a9 2018-08-31T20:25:08.150530Z"
"7c745fb 2018-08-31T18:34:41.467610Z"
"df14f2b 2018-08-31T16:44:31.669357Z"
"ea1503f 2018-08-30T23:35:32.689955Z"
"f96d1f8 2018-08-30T23:01:10.652859Z"
"992ee93 2018-08-30T22:36:52.298097Z"
"3a76afb 2018-08-30T22:15:09.961783Z"
"0e411d7 2018-08-30T22:07:35.898142Z"
"e2cd124 2018-08-30T09:01:53.564093Z"
"697a7a5 2018-08-30T07:57:09.159598Z"
"robjloranger-patch-1 2018-08-29T19:19:11.026857Z"
"417a487 2018-08-29T19:19:09.260812Z"
"7af04ff 2018-08-28T21:26:22.314683Z"
"0b9624a 2018-08-28T16:45:45.021407Z"
"e5ce3f7 2018-08-28T10:06:03.368455Z"
"69a5340 2018-08-28T06:55:48.660562Z"
"b62ebdd 2018-08-27T21:28:17.606565Z"
"37f0687 2018-08-27T16:57:38.763436Z"
"5c3af51 2018-08-27T16:10:25.875907Z"
"a953f11 2018-08-27T16:04:17.963567Z"
"7962cac 2018-08-27T15:29:47.746387Z"
"0f218d0 2018-08-27T15:19:42.011755Z"
"latest 2018-08-27T14:45:24.541023Z"
"v0.1.0 2018-08-27T14:45:22.917513Z"
"d66b481 2018-08-27T13:32:01.615786Z"

[marpio]

@arkadyb would it be okay for you if we close the issue? The fix will be applied with #843 Please use the canary tag to try things out for now.

[arkadyb]

@marpio ok with me. I do builds from source codes for now anyway.