Athens: CDN metadata in memory driver

[robjloranger]

the last piece of #147 is the in memory driver.

This could be as simple as using the existing file system implementation but passing in an afero.NewMemMapFs to the (./pkg/cdn/metadata/fs).NewStorage when in memory is desired.

[michalpristas]

also once somebody is on it. it would be great to make it default for Dev env

[mydiemho]

I can work on this.

One question:
1) I am working on a fork, but the code is looking for some package in gomods/athens. Do I just have to also clone the gomods/athens repo?

[marpio]

Thanks @mydiemho ! That's awesome.
As for your question - I think the easiest way is to add a remote.
You can follow i.e http://blog.campoy.cat/2014/03/github-and-go-forking-pull-requests-and.html?m=1
Also you can use https://github.com/gomods/athens/blob/master/pkg/storage/mem/mem.go
as a reference for the implementation if that's a help.

[mydiemho]

@robjloranger @marpio I'm a little confuse on what I'm suppose to do here.

@robjloranger you said above that I just need to pass in afero.NewMemMapFs to the (./pkg/cdn/metadata/fs).NewStorage method

but it looks like we're already doing that in https://github.com/gomods/athens/blob/master/pkg/storage/mem/mem.go#L34

[michalpristas]

Hey @mydiemho
For a reference take a look at https://github.com/gomods/athens/tree/master/pkg/cdn/metadata/fs
This is basically what you need to achieve, but with in-memory store

The code you posted uses github.com/gomods/athens/pkg/storage/fs while the goal of this task is to have the implementation in github.com/gomods/athens/tree/master/pkg/cdn

The difference is in what each package stores.
While pkg/storage/* stores the actual bit, source code with mod and info files
pkg/cdn stores a metadata about the packages stored using pkg/storage especially redirect URI to CDN.

Think of storage as a local DB and CDN as a tool taking care of global distribution.

• storage.Get - returns bits from local. so if you're from another part of the world it might be sluggish
• CDN.Get on the other hand gets you a redirect URI, this uri gets you to CDN on top of your local DB (configured using your cloud provider) and CDN decides which replica is the best for you.

Does this make sense?

[mydiemho]

@michalpristas could you take a look to see if I am on the right path before I make more changes?

Also, how should I test this? I have look at the tests in pkg/cdn and pkg/storage and they don't seem to use the same strucutre

[robjloranger]

For this you can use package level unit tests. Have a look in packages other than storage for examples. The storages all share similar functionality which is why they share a test suite.

I will have a proper look on the morning. PST. 😴

[mydiemho]

@robjloranger I looked through other packages and a few don't have tests either.

What do you mean by package level unit tests? Could you link to an example?

[robjloranger]

For example the error package, https://github.com/gomods/athens/blob/master/pkg/errors/errors_test.go