Athens: Implement Distributed Deduplication Mechanisms

[arschles]

Is your feature request related to a problem? Please describe.

308 was for making Athens not do two go get calls for the same module@version at the same time. That meant that if N go get mod@ver calls came in at the same time, only one go get would get issues on the backend. https://github.com/gomods/athens/pull/573 went and fixed #308 by introducing "single flight" - a kind of mutual exclusion mechanism built in memory.

Now, we have an Athens that will only run go get once for a given mod@ver, but if you want to run a fleet of Athens servers (i.e. behind a load balancer in the cloud, in pods in a Kubernetes cluster, etc...), the different Athens processes don't all single-flight together. If a go get mod@ver request goes to athensA and another go get mod@ver request goes to athensB at exactly the same time, they will both still run go get mod@ver.

Describe the solution you'd like

We should build new download.Wrappers (like the one here) that use distributed mutual exclusion mechanisms, so that a fleet of Athens servers can single-flight together.

I'd like us to have multiple drivers eventually. Some ideas:

• etcd locks
• Consul locks
• Cloud proprietary solutions

Describe alternatives you've considered

I tried to think of other ways to de-duplicate go gets across lots of Athens servers, but can't think of others besides distributed mutual exclusion.

Additional context

[marwan-at-work]

I'd like to work on this (I've done in implementation for it back in the day so I just need to dig it up).

I'd like to emphasize that this should only be worked on, reviewed, and merged after beta. So that we focus our efforts on achieving the beta milestone :v:

[arschles]

@marwan-at-work are you thinking that we can have multiple stasher implementations that use singleflight? if so, I have a few in mind:

• use groupcache to fill, then store in the backing storage (don't really know about this one but it does a lot of the work for us)

• create an interface like this:

  type singleFlightLease interface {
      // acquire a lease for mod@ver. the caller has dur time to do its work
      // (i.e. fetch the module and save it into storage) and call Release.
      // after that, it loses the lease and another caller will be able to acquire it
      //
      // this function blocks until the lease could be acquired or there was an error.
      // when the lease is acquired, the returned string will be the ID of the lease.
      // you can use it to release the lease later
      Acquire(mod, ver string, dur time.Duration) (string, error)
  
      // release the lease with the given ID. return a non-nil error if there was a 
      // problem releasing the lease
      Release(id string) error
  

  and then use it in a single flight stasher to do mutual exclusion on fetching modules and storing them. we can do a bunch of different implementations for this like the ones in the OP (not sure but I think we can use pub/sub systems for this too)

I'm really interested in this problem. For running a public Athens we'll need a deduplication mechanism that can handle lots of workers, so I'm assuming probably a cloud thing we don't have to manage.

Maybe there are some other options I'm missing in the list. What do you think about all this?