Rkt: Support for OCI runtime and image specification.

[sgotti]

I don't know if someone is already thinking/working on this but I'd like to share my thoughts and hear yours.

Since, after the oci runtime spec, also the oci image spec is taking shape, I started thinking on how to implement this in rkt.

Possible steps:

OCI Image spec

Currently the OCI image spec is made of different media types that should be fetched (the OCI image spec doesn't cover distribution) and stored.

• Implement an OCIStore. The current rkt store is built around the appc/spec ACI images, it should probably be enhanced to also handle the OCI image format but probably, since the two specs are quite different, a dedicated store will be cleaner. It will have multiple diskv stores and a db to save the various media types by their digest (related #2479, #2313)
• Implement OCI image rendering (from multiple layers).
• Implement inside the OCIStore a TreeStore that will render the OCI image (similar to the current ACI treestore) (and put the OCI runtime config.json at its root).
• Implement OCI image fetching. Actually this should just cover only fetching from a docker v2 registry:
  
  
  
  • Implement a DockerV2RegistryFetcher
  
  

Probably big parts of the docker2aci code should be used (copied inside rkt) for implementing the above steps.

OCI Runtime spec

One idea will be to convert the OCI runtime config to an ACI manifest (using oci2aci?) so the podmanifest generation and handling done in the stage0 and the various stage1s shouldn't be changed.

Questions

• since the scope of the OCI image spec is to be able to run an image by its image name string (like the ACI image string (https://github.com/opencontainers/image-spec#cooperation-with-oci-runtime-project) how to distinguish in rkt if an user want to run an appc/spec or an OCI image? Probably something like https://github.com/coreos/rkt/issues/715#issuecomment-175058054 is needed.
• Actually I'll omit fetching/running images by file/url since it's not clear if there'll be an image bundle format (containing all the image layers, manifests etc...) or how to do this using multiple media types files/url (for example: how to fetch the layers if I only provide the image manifest? The ACI rkt implementation is different/simpler since given an aci it just goes to the store or does meta discovery to retrieve dependencies)
• How to show the different media types in the OCI Store?
  
  
  
  • Use rkt image list with an additional type column? What to do if there're different fields between aci images and OCI images that makes this difficult to show in a single table?
  
  
  • Keep them separated and use a flag to choose between ACI or OCI images?
  
  
  • What OCI image spec media type to show by default (they also will probably have different fields)?
  
  
  • Just show the image manifest (since the layers are detached from it) and add some options to also show the layers?
  
  
• How to handle docker images for V1 and V2 repositories?
  
  
  
  • keep using docker2aci special docker:// urls for both docker V1/V2 registries until it'll be deprecated by the OCI spec?).
  
  
  • keep using docker2aci special `docker:// only for V1 registries and move V2 handling inside the OCIStore?
  
  

[s-urbaniak]

@sgotti It just happened that I am supposed to work on this effectively starting today ;-) The first step for me is to implement a oci-image-tool, see https://github.com/opencontainers/image-spec/issues/43.

I was thinking of implementing an OCI image library hosted maybe in a (not-yet-existing) github.com/opencontainers/image repository which we could vendor here in rkt leaving the implementation details, like the OCIStore, rendering, validation outside of the rkt code base.

Regarding all your other questions I'm also still in research mode, so maybe we can clarify those iteratively.

[sgotti]

@s-urbaniak Thanks! Good to know.

I think that an external library for rendering and validation (like done for the appc/spec) will be good.
To be more accurate in appc/spec the acirenderer will resolve the required files for every dependency and it's up to rkt to render them https://github.com/coreos/rkt/blob/master/pkg/aci/render.go but the OCI image spec is simpler since there isn't a DAG tree but just a list of layers.

I think that the store should live inside rkt (like the current store for ACIs) and probably also fetching (since OCI image spec doesn't cover distribution) should be done inside rkt like the current fetching implementations since they will be very tied to the various rkt options.

Should we keep this issue as a base discussion point for the implementations or do you have other ways to keep in sync?

[s-urbaniak]

@sgotti great, many thanks for the pointers!

Let's keep this issue strictly as "implement OCI image/runtime support in rkt" and we can sync via irc, I'm sur on freenode and online in the #rkt-dev channel.

[dgonyeo]

Mentioning here because it's relevant: I recently made a PR against docker2aci that adds docker v2.2 and OCI support: https://github.com/appc/docker2aci/pull/176.

This would result in rkt having the same support for OCI as it does for docker, so it's nothing like the native AppC support rkt has today but if you're interested in copying logic from docker2aci for fetching and/or manifest conversion, it's a starting place.

[sgotti]

To keep track, the ongoing work for natively supporting multiple image types and related requirements is here:

2953 #3101 Distribution point concept (the base for all ongoing work)

3071 Reference based image handling and related store

3039 Pod manifest extension for different digest types.

2964 Fetchers refactor, they'll become distribution handlers and will use transport plugins. The docker registry distribution handler will fetch images from a docker registry (for docker images with a v2.2 manifest and oci images)