Is your feature request related to a problem? Please describe.
There are still interesting storage options which we do not support i.e. B2. We cannot possibly support all storages in our code base but we can make it possible for the user to provide one i.e. through https://golang.org/pkg/plugin/
Describe alternatives you've considered
Implementing them like we do now but it will make the maintenance difficult.
cc: @marwan-at-work since you mentioned that on slack
marpio
All 8 comments
Another alternative would be is to provide an HTTP interface to storages, almost an extension of the download protocol. And also we can look at how K8s does plugin extensions
marwan-at-work
on 12 Mar 2019
I've implemented an http storage type in an effort to support an integration with a new storage backend (Artifactory) in a more generic way than simply integrating with that backend (this came out of a Slack conversation with @arschles). It works essentially as @marwan-at-work described, in that it maps the storage operations to generic HTTP GET/PUT/DELETE calls with BasicAuth, and looks pretty much just like the download protocol with some extensions. Perhaps that would be applicable or useful here?
My intention is to open an proposal issue and then a PR, though I've been held up somewhat (internal review by my employer). However that's nearly resolved鈥攂arring any surprises I expect to do that this week.
tylerchr
on 12 Mar 2019
@tylerchr so awesome! Looking forward to this
I'm +1 on this over the Go plugin interface, so that folks can build storage interfaces and have flexibility to run them either on localhost or over a network
arschles
on 13 Mar 2019
I opened #1130 (issue describing my needs) and #1131 (prototype implementation of an HTTP storage type).
Would be interested to hear your thoughts (@marpio, @marwan-at-work) especially since you're coming from the "storage plugins" angle which I hadn't initially considered.
tylerchr
on 16 Mar 2019
Missed this when it was first discussed, but there's an option to use https://github.com/hashicorp/go-plugin and define a gRPC service as the interface. Then, storage plugins can be created and maintained in a separate repo, so long as they fulfill the service definition.
Kunde21
on 29 May 2019
@Kunde21 the readme says:
While the plugin system is over RPC, it is currently only designed to work over a local [reliable] network. Plugins over a real network are not supported and will lead to unexpected behavior.
Is that outdated?
marwan-at-work
on 29 May 2019
@marwan-at-work the readme kind of buries the gRPC option:
Cross-language support. Plugins can be written (and consumed) by almost every major language. This library supports serving plugins via gRPC. gRPC-based plugins enable plugins to be written in any language.
Kunde21
on 29 May 2019
Is everyone ok with sticking with HTTP for now? It's less specification work for us because we can use the module download protocol we all know and love already.
More conversation at https://github.com/gomods/athens/pull/1131#issuecomment-496509839 and https://github.com/gomods/athens/issues/1110#issuecomment-471779673
arschles
on 8 Jun 2019
Related issues
komuw
路
3Comments
marpio
路
4Comments
robjloranger
路
3Comments
fedepaol
路
4Comments
Haiyung
路
3Comments
Most helpful comment
I've implemented an
httpstorage type in an effort to support an integration with a new storage backend (Artifactory) in a more generic way than simply integrating with that backend (this came out of a Slack conversation with @arschles). It works essentially as @marwan-at-work described, in that it maps the storage operations to generic HTTP GET/PUT/DELETE calls with BasicAuth, and looks pretty much just like the download protocol with some extensions. Perhaps that would be applicable or useful here?My intention is to open an proposal issue and then a PR, though I've been held up somewhat (internal review by my employer). However that's nearly resolved鈥攂arring any surprises I expect to do that this week.