Athens: Do we need CDN package?

[michalpristas]

To put you into context. At this time we have two moving parts when talking about Olympus.

• Module Storage (also referred as storage)
• CDN metadata store (pkg/cdn)

I was wondering do we need the second one?

Imagine a scenario.

You setup Olympus with a store (no cloud provided CDN on top of it just yet)
When you ask for a package it will return one from Olympus.

Then you experience some slower communication with other parts of the world because your storage is not geo-replicated. You set up a CDN on top of it.
Now you have your CDN and configure Olympus to use that CDN.

Olympus with each new module stored will store a redirect URI intoCDN Metadata store as well.
When the user asks for a package it will get 301 to CDN based on redirectURI stored in a CDN metadata store.
The user is happy, download is faster.

You add a domain for your CDN or you will use another CDN provider. Then you'll update Olympus configuration to use this new domain.

Now new modules will store redirectURI with domain aware redirect URI,
old ones have still domainless redirect URI (which may be invalid at this point).
All modules sit in the same DB.

---

Wouldn't it be easier, to skip saving metadata into CDN storage, and redirect based on the CDN settings? (Composing redirectURI on the fly)
The process will be easier we will get rid of a problem scatched above and people won't be confused about so many storage interfaces.

/cc @marwan-at-work @arschles

[michalpristas]

Vote here to kick CDN out

[michalpristas]

Vote here to keep it

[marwan-at-work]

@michalpristas you mention that a user sets up a CDN on top of Olympus, can a user actually do that?

[marpio]

Related https://github.com/gomods/athens/issues/299

[michalpristas]

@marwan-at-work CDN is part of your infra,
examples
https://aws.amazon.com/cloudfront/
https://docs.microsoft.com/en-us/azure/cdn/cdn-overview

It is not our goal to provide a cdn and tackle with all the problems related to it e.g routing
So I as an Azure user, choose to use Azure Blob storage for modules and then i set up azure cdn on top of it.
Then i will configure olympus with cdn URI, then all the request for info, mod and zip will be redirected there.

[marwan-at-work]

I was under the impression that Olympus will be public, in other words “we”
create/control the storages and CDNs, not the user. The user can only
configure a Proxy. So when you say, “I an Azure user use Azure Blob
storage” do you mean that any one can configure www.olympus.com to use my
own Azure storage?

On Fri, Aug 17, 2018 at 9:45 AM Michal Pristas [e-mail]
wrote:

@marwan-at-work https://github.com/marwan-at-work CDN is part of your
infra,
examples
https://aws.amazon.com/cloudfront/
https://docs.microsoft.com/en-us/azure/cdn/cdn-overview

It is not our goal to provide a cdn and tackle with all the problems
related to it e.g routing
So I as an Azure user, choose to use Azure Blob storage for modules and
then i set up azure cdn on top of it.
Then i will configure olympus with cdn URI, then all the request for info,
mod and zip will be redirected there.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/503#issuecomment-413870749, or mute
the thread
https://github.com/notifications/unsubscribe-auth/APihdQnCtf7G-8oy40G32JZxjkSjTLs-ks5uRsjkgaJpZM4WBjyy
.

[michalpristas]

oh that's what you mean, by user i mean an Ops person responsible for running it. sorry for the confusion.
As brian mentioned there are few companies which agreed to run Olympus on their infra. So they will need to set it up accordingly using the stack they have available.

Edit; now i know your mobile number. you're in troubles ☎️

[robjloranger]

@michalpristas this is actually what I thought we would do. Which I why I kept getting confused when we discussed the CDN and Storage together. I couldn't figure out what the CDN was supposed to be for.

[marwan-at-work]

@michalpristas lmao thanks for the call out, I removed it. Although I'm pretty sure it's on my website too.

I think it's safe to say most of us want to remove the CDN :)

@arschles curious about your input.

[marpio]

There was this idea (I think @arschles mentioned it) of having the replication of modules between Olympuses consist of two steps - first saving the CDN URL of another Olympus and downloading and storing to it's own storage in step 2.
So let say we have Olympus MS (OM) and Olympus Amazon (OA).
OM checks whats new from OA and finds out that OA has module Mx which OM doesn't have. It then saves the OA-CDN-URL of the module and from that point OM can serve requests of Mx using the OA-CDN. Then OM can download Mx, save it in it's storage and from that point serve Mx using it's own OM-CDN-URL.
This would minimize the time where OM and OA aren't consistent.

I don't know if this idea have been thrown away or not but I guess that would require us to leave the CDN pkg in place.

[michalpristas]

Yes, this is a valid scenario, but it was brought when we hadn't redirects or sync cache fill from VCS.
When this CDN solution was designed we had Olympus in mind with workers and pull sync.
Now we think of push-pull synchronization and sync vcs fill on top of that.
As I think about it sync VCS fill is not really redirect friendly, whether it's to another olympus or to other CDN.

[michalpristas]

Closing a vote in favor of code removal