Athens: Cast your votes here! (1) Remove RDBMS support from the proxy? and (2) How should the registry redirect to CDNs?

[arschles]

About

This issue has had lots of awesome discussion - so much that it's actually split into two topics 😆!

• Whether or not we should remove RDBMS support from the proxy
• How the registry should redirect to CDN locations

All the discussions are in the thread in this isssue, but I'm summarizing the discussions on each topic here, and put each to a vote. In each section, you'll see links to do voting.

Voting Instructions

To cast your vote, put a 👍 reaction (🎉 s are cool too 😄) on the comment for your preferred option. Pleasepleaseplease be cool and upvote just one of the two options per topic. Also, I'm only going to be counting upvotes, not downvotes.

The voting deadline is 11am US Pacific Time on August 16th (that's when our next dev meeting is)

Removing RDBMS Support From the Proxy?

The proxy stores module metadata and source code in RDBMS's and the registry uses RDBMS's to store module metadata. When we wrote the first prototype of the proxy, I used Buffalo's built-in RDBMS drivers to store source code, go.mod, and .info files. More recently, @marwan-at-work has found that storing big dependencies in RDBMS's is problematic (at best).

A few notes:

• This issue is a proposal for removing RDBMS storage drivers from the proxy only.
• The data that we're planning on having the registry store is different from what the proxy stores, and we'll discuss that in another issue. The overview is in REGISTRY.md

Cast your votes here:

• keep RDBMS support in the proxy
• remove RDBMS support from the proxy

How to Make the Registry Redirect the go Tool to a CDN

The original RDBMS discussion led us to discuss where the registry will store code, go.mod, and *.info files, and how it's different from the proxy. In most cases, we're planning on storing these artifacts in a CDN. We're planning to do this because we expect that the global Go community will be requesting modules and CDNs are a system designed specifically for this purpose - delivering static content quickly to users in any part of the world (they also do this relatively inexpensively).

The question that came up after we discussed , then, is: how should the registry redirect requests to the CDN?

There are two options:

1. For every API call, make the registry return a HTTP 301 Redirect to the CDN location
   
   
   
   • The go toolchain currently supports this behavior and does the right thing
   
   
   • We would want to document this explicitly in the download protocol specification as well
   
   
2. Add some design to the go toolchain to make it do a ?go-get=1 request (sometimes called a "meta tag redirect")
   
   
   
   • The toolchain expects an HTML response with a <meta> tag in it that looks similar to this: <meta name="go-import" content="gopkg.in/yaml.v1 git https://gopkg.in/yaml.v1">
   
   
   • The registry can direct the toolchain to make all subsequent download protocol requests against a root URL
   
   
   • The registry would return the root URL of the CDN of its choosing
   
   

Cast your votes here:

• Use 301 redirects
• Use "meta tag redirects"

[robjloranger]

I think it's the other way around. Doesn't the proxy only store meta data and the registry stores source code?

[robjloranger]

I guess it caches a copy. I think the discussion was regarding performance of reading and writing that source file into a database. Which I think would be pretty poor, in comparison to gridFS or cloud blob storage.

[robjloranger]

I found it, it was in #383

[marwan-at-work]

I was under the impression that, whether it's Proxy or Registry, we always save all three pieces of a module (RevInfo, go.mod, zip) in its storage backend. Is that not correct? :)

[robjloranger]

I think that's right. I get confused. I think the proxy also stores meta data but the registry just stores the files mod,info,zip.

[marwan-at-work]

@robjloranger i thought by metadata, you meant go.mod and RevInfo. If not, what kind of metadata does the proxy store?

[arschles]

So the proxy stores mod, zip, info in the storage backend - no other metadata. But the registry stores those things in a CDN, and serves a "meta redirect" to the go cli. The metadata that the registry stores is the log entries for provenance / synchronization / auth and so forth

(we should have docs for all of this, sorry for that. I think @michalpristas is working on them now as part of https://github.com/gomods/athens/issues/390)

Let me know if I made sense?

[robjloranger]

Oh, that makes more sense. I always read that as some other metadata.

[marwan-at-work]

@arschles i get it now. But why can't the registry just respond with 301 -> CDN and then the client (cmd/go) will get stuff from cdn directly as well?

And side note, as of now, cmd/go will not work with serving HTML if cmd/go has GOPROXY set up.

Related conversation: https://github.com/gomods/athens/issues/350

[arschles]

We discussed two options in the Aug 9, 2018 meeting, so let's vote on there in here:

[arschles]

Vote here if you want us to work on getting go get to respect ?go-get=1 requests (and enabling us to do redirects) when GOPROXY is set (requires cmd/go itself to be updated as it wouldn't work right now.)

[arschles]

Vote here if you want the proxy to issue 301's to a CDN storage location for all the download protocol endpoints. We will also add support for 301s into the download protocol specifications standardize that 301s are possible

[michalpristas]

I voted for 301, but maybe we should keep the code (unused) for meta tags so we can reuse it with Sparta reborn

[marwan-at-work]

I think the issue still stands that, can anyone make RDBMS work? I think I'm the only one who tried it on a big repo and failed.

[arschles]

@marwan-at-work you're right, we got off track in this issue. This issue is now basically two things in one:

1. should we remove RDBMS support from the proxy?
2. how should we have the registry "redirect" the go toolchain to a CDN
   
   
   
   • So far, we have 4 votes for redirecting with 301 status codes, 0 for "meta tag redirects". Cast your votes here and here
   
   

I'm going to open votes for the RDBMS support (in item 1) in this thread as well and we'll finalize the decisions from both (1) and (2) in next week's dev meeting (on August 13, 2018) - so vote before then! Use the Github reactions feature to give a 👍 to upvote. No downvotes please 😄

I'll create follow-up votes for each item, and close this issue after the votes are done.

[arschles]

Vote here if you'd like the proxy to keep RDBMS support

[arschles]

Vote here if you'd like us to remove RDBMS support from the proxy

[marwan-at-work]

I did a little more reading on why metadata tag was introduced in the original vgo papers. The use case for this functionality is not necessarily meant for Proxies. It's actually meant for users who want to abstract proxies under their own domains import path.

For example:

1. I write my package under marwan.io/mycoolpackage
2. Then cmd/go does GET https://marwan.io/mycoolpackage?go-get=1
3. My server returns an html with this meta <meta name="go-import" content="marwan.io/mycoolpackage mod https://athens.com">
4. Athens.com, can then just do a 301

Later on if I don't like Athens, I can just switch my meta tag to point to another mod server or cloud storage.

So us voting away from HTML redirects makes even more sense to me at this point.

Furthermore, we have a potential bug:
If a user does that example above, we can end up stuck in an infinite loop. Will create an issue to investigate if that's true.

[arschles]

Since we voted to remove RDBMS, I've created https://github.com/gomods/athens/issues/494 for that work, and since we voted to do 301s on proxy misses, I've created https://github.com/gomods/athens/issues/495.

I'm gonna close this issue now. Thanks everyone for voting!!! ❤️