Athens: We should have a private module filter for the proxy
We have an exclude list which tells the proxy to not return a module that matches a pattern on the list now. We also need to implement a list of patterns that the proxy can use to determine if a module is "private" - this means that it shouldn't ask for the module from upstream (the registry) and instead should directly fetch them from the VCS (with go get)
arschles
All 14 comments
Can I try to work on this?
fedepaol
on 20 Jul 2018
@fedepaol relevant issue: https://github.com/gomods/athens/issues/96 and PR https://github.com/gomods/athens/pull/212
marpio
on 20 Jul 2018
@fedepaol please do! and unlike last time, this one is not going to go away or become obsolete 😄
arschles
on 20 Jul 2018
I gave a quick look at https://github.com/gomods/athens/pull/212 . What I would try to do is to extend the domain of the current filter (now +/- ) with a p maybe?
Something like
- a
+ a/b/c
p a/b/c/d
Would mean that a/b/c/d will never be looked for in olympus an fetched directly from the vcs.
Does it make sense? To me it feels better than having a different filter file.
Can I try to proceed with this implementation?
fedepaol
on 21 Jul 2018
New question, sorry @arschles :-(
I am looking at https://github.com/gomods/athens/blob/db7a318a6f1944640b4474f3df3a83332d74f271/cmd/proxy/actions/app_proxy.go#L21 and https://github.com/gomods/athens/blob/db7a318a6f1944640b4474f3df3a83332d74f271/cmd/proxy/actions/cache_miss_handler.go#L25
My understanding of the current status of the proxy is that when there is a miss in the local storage, alignments (notify and fetch from the registry) are queued asynchronously but the request that caused the miss gets a 404.
Is that correct?
If so, what part should I implement with this issue? Just leaving the thing as they are (trying to asynchronously fill the local cache directly from the vcs) or something different?
fedepaol
on 22 Jul 2018
Hey Federico - I’m traveling today and going to be offline tomorrow morning
so hopefully someone else can give more detail.
We’re going to begin filling caches synchronously soon, so if there’s any
way you can design the filter so that it can be applied later after the
synchronous implementation gets merged, that would be the best option.
There are a few PRs open that show how the async => sync work is coming but
I’m on my phone so I don’t have them handy
On Sun, Jul 22, 2018 at 14:41 Federico Paolinelli notifications@github.com
wrote:
New question, sorry @arschles https://github.com/arschles :-(
I am looking at
https://github.com/gomods/athens/blob/db7a318a6f1944640b4474f3df3a83332d74f271/cmd/proxy/actions/app_proxy.go#L21
and
https://github.com/gomods/athens/blob/db7a318a6f1944640b4474f3df3a83332d74f271/cmd/proxy/actions/cache_miss_handler.go#L25My understanding of the current status of the proxy is that when there is
a miss in the local storage, alignments (notify and fetch from the
registry) are queued asynchronously but the request that caused the miss
gets a 404.Is that correct?
If so, what part should I implement with this issue? Just leaving the
thing as they are (trying to asynchronously fill the local cache directly
from the vcs) or something different?—
You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/309#issuecomment-406899040, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEU0cdwlt_94YTj2msqidy1rhWYPE56ks5uJPGggaJpZM4VXMwU
.
arschles
on 23 Jul 2018
Yeah they pointed me to the pr. Will give it a look and try to see what I
can do!
Il lun 23 lug 2018, 17:58 Aaron Schlesinger notifications@github.com ha
scritto:
Hey Federico - I’m traveling today and going to be offline tomorrow morning
so hopefully someone else can give more detail.We’re going to begin filling caches synchronously soon, so if there’s any
way you can design the filter so that it can be applied later after the
synchronous implementation gets merged, that would be the best option.There are a few PRs open that show how the async => sync work is coming but
I’m on my phone so I don’t have them handyOn Sun, Jul 22, 2018 at 14:41 Federico Paolinelli <
[email protected]>
wrote:New question, sorry @arschles https://github.com/arschles :-(
I am looking at
My understanding of the current status of the proxy is that when there is
a miss in the local storage, alignments (notify and fetch from the
registry) are queued asynchronously but the request that caused the miss
gets a 404.Is that correct?
If so, what part should I implement with this issue? Just leaving the
thing as they are (trying to asynchronously fill the local cache directly
from the vcs) or something different?—
You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/309#issuecomment-406899040,
or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AAEU0cdwlt_94YTj2msqidy1rhWYPE56ks5uJPGggaJpZM4VXMwU.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/309#issuecomment-407108953, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAMPcN3Dt9jwB1vlkSQrUJZuNa-DeCQoks5uJfKsgaJpZM4VXMwU
.
fedepaol
on 23 Jul 2018
Awesome! Buona fortuna 😊
On Mon, Jul 23, 2018 at 12:25 Federico Paolinelli notifications@github.com
wrote:
Yeah they pointed me to the pr. Will give it a look and try to see what I
can do!Il lun 23 lug 2018, 17:58 Aaron Schlesinger notifications@github.com ha
scritto:Hey Federico - I’m traveling today and going to be offline tomorrow
morning
so hopefully someone else can give more detail.We’re going to begin filling caches synchronously soon, so if there’s any
way you can design the filter so that it can be applied later after the
synchronous implementation gets merged, that would be the best option.There are a few PRs open that show how the async => sync work is coming
but
I’m on my phone so I don’t have them handyOn Sun, Jul 22, 2018 at 14:41 Federico Paolinelli <
[email protected]>
wrote:New question, sorry @arschles https://github.com/arschles :-(
I am looking at
and
https://github.com/gomods/athens/blob/db7a318a6f1944640b4474f3df3a83332d74f271/cmd/proxy/actions/cache_miss_handler.go#L25
>My understanding of the current status of the proxy is that when there
is
a miss in the local storage, alignments (notify and fetch from the
registry) are queued asynchronously but the request that caused the
miss
gets a 404.Is that correct?
If so, what part should I implement with this issue? Just leaving the
thing as they are (trying to asynchronously fill the local cache
directly
from the vcs) or something different?—
You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/309#issuecomment-406899040,
or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAEU0cdwlt_94YTj2msqidy1rhWYPE56ks5uJPGggaJpZM4VXMwU
>.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/309#issuecomment-407108953,
or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AAMPcN3Dt9jwB1vlkSQrUJZuNa-DeCQoks5uJfKsgaJpZM4VXMwU.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/309#issuecomment-407172581, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEU0Y5O-SK8Lz_txyAfPOSOBK2e8DOHks5uJiMagaJpZM4VXMwU
.
arschles
on 23 Jul 2018
Grazie!
Also, does my suggestion of extending the current configuration instead of
adding a new one make sense (
https://github.com/gomods/athens/issues/309#issuecomment-406823918)?
Il lun 23 lug 2018, 21:26 Aaron Schlesinger notifications@github.com ha
scritto:
Awesome! Buona fortuna 😊
On Mon, Jul 23, 2018 at 12:25 Federico Paolinelli <
[email protected]>
wrote:Yeah they pointed me to the pr. Will give it a look and try to see what I
can do!Il lun 23 lug 2018, 17:58 Aaron Schlesinger notifications@github.com
ha
scritto:Hey Federico - I’m traveling today and going to be offline tomorrow
morning
so hopefully someone else can give more detail.We’re going to begin filling caches synchronously soon, so if there’s
any
way you can design the filter so that it can be applied later after the
synchronous implementation gets merged, that would be the best option.There are a few PRs open that show how the async => sync work is coming
but
I’m on my phone so I don’t have them handyOn Sun, Jul 22, 2018 at 14:41 Federico Paolinelli <
[email protected]>
wrote:New question, sorry @arschles https://github.com/arschles :-(
I am looking at
and
>
My understanding of the current status of the proxy is that when
there
is
a miss in the local storage, alignments (notify and fetch from the
registry) are queued asynchronously but the request that caused the
miss
gets a 404.Is that correct?
If so, what part should I implement with this issue? Just leaving the
thing as they are (trying to asynchronously fill the local cache
directly
from the vcs) or something different?—
You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub
<https://github.com/gomods/athens/issues/309#issuecomment-406899040
,
or mute
the thread
<>
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/309#issuecomment-407108953,
or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAMPcN3Dt9jwB1vlkSQrUJZuNa-DeCQoks5uJfKsgaJpZM4VXMwU
>.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/309#issuecomment-407172581,
or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AAEU0Y5O-SK8Lz_txyAfPOSOBK2e8DOHks5uJiMagaJpZM4VXMwU.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/309#issuecomment-407172959, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAMPcMcZ7JARDFgdzvvROuDJXGBb9UPvks5uJiNzgaJpZM4VXMwU
.
fedepaol
on 23 Jul 2018
Update: I gave a look at all the stuff attached to the big Kahuna (https://github.com/gomods/athens/issues/290) and after discussing with @marpio I am pretty convinced that the big stuff is not implementing the new filter but putting the feeding the proxy with the local vcs in place.
For this reason, I would try to pick something else and wait for this one.
PS: thought that first bugs were easier :-)
fedepaol
on 23 Jul 2018
Ok, sorry again about this. This issue should be easy after the other stuff
is in, I swear! Ping me on slack when you can and We’ll get you set up with
an issue you can do right now 😊
On Mon, Jul 23, 2018 at 16:05 Federico Paolinelli notifications@github.com
wrote:
Update: I gave a look at all the stuff attached to the big Kahuna (#290
https://github.com/gomods/athens/issues/290) and after discussing with
@marpio https://github.com/marpio I am pretty convinced that the big
stuff is not implementing the new filter but putting the feeding the proxy
with the local vcs in place.
For this reason, I would try to pick something else and wait for this one.PS: thought that first bugs were easier :-)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/309#issuecomment-407200877, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEU0coBiPlRGvIdkn3-JLiF0dP8w-O7ks5uJjqDgaJpZM4VXMwU
.
arschles
on 24 Jul 2018
Lol no worries! I won't give up!
Il mar 24 lug 2018, 16:27 Aaron Schlesinger notifications@github.com ha
scritto:
Ok, sorry again about this. This issue should be easy after the other stuff
is in, I swear! Ping me on slack when you can and We’ll get you set up with
an issue you can do right now 😊On Mon, Jul 23, 2018 at 16:05 Federico Paolinelli <
[email protected]>
wrote:Update: I gave a look at all the stuff attached to the big Kahuna (#290
https://github.com/gomods/athens/issues/290) and after discussing with
@marpio https://github.com/marpio I am pretty convinced that the big
stuff is not implementing the new filter but putting the feeding the
proxy
with the local vcs in place.
For this reason, I would try to pick something else and wait for this
one.PS: thought that first bugs were easier :-)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/309#issuecomment-407200877,
or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AAEU0coBiPlRGvIdkn3-JLiF0dP8w-O7ks5uJjqDgaJpZM4VXMwU.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gomods/athens/issues/309#issuecomment-407426463, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAMPcOAfhzWOlN-Dg-wMRZ4DTD0K48i0ks5uJy7WgaJpZM4VXMwU
.
fedepaol
on 24 Jul 2018
@fedepaol #290 is done now - sorry for not mentioning that sooner. you're clear to work on this if you still want to!
arschles
on 30 Jul 2018
Will work on this
fedepaol
on 31 Jul 2018
Related issues
weitangli
·
3Comments
arschles
·
3Comments
robjloranger
·
4Comments
zkry
·
3Comments
marpio
·
4Comments
Most helpful comment
Will work on this