Openapi-specification: extension registry management

[dret]

i just discovered the extension registry and it's great to see this getting started (as per #1351, i guess). my suggestion would be to decouple this from the spec and turn it into a repo and site of its own. one of the main motivations for registries is to decouple spec and extension management, and managing these two things separately usually is a good idea.
another thing i would suggest is to track registry updates, and to have clear extension policies. maybe https://tools.ietf.org/html/draft-wilde-registries-01 can help with better understanding the issues that a registry should follow if it is expected to be reliable and robust over time.

[MikeRalphson]

Thanks @dret We did debate having a separate repo for this, but came down on the side of using the gh-pages branch as a pseudo-separate repository with its own history. We can revisit this decision easily if it becomes necessary. The registry has a site of its own at spec.openapis.org/registry.

The gh-pages branch can (and should) have its own CONTRIBUTING.md guidelines, and I'll be sure to work through the IETF draft you linked to. This is one reason we haven't removed the large "DRAFT" watermark from the site yet.

[dret]

On 2019-01-13 22:42, Mike Ralphson wrote:

Thanks @dret https://github.com/dret We did debate having a separare
repo for this, but came down on the side of using the |gh-pages| branch
as a pseudo-separate repository with its own history. We can revisit
this decision easily if it becomes necessary. The registry has a site of
its own at spec.openapis.org/registry https://spec.openapis.org/registry.

ok. better than nothing, i guess. i still think that these things should
be entirely decoupled. after all, form example there may be extensions
for AsyncAPI (if that ever gets to be a proper OpenAPI-based thing)
which then maybe also should be listed in that registry?

The |gh-pages| branch can (and /should/) have its own |CONTRIBUTING.md|
guidelines, and I'll be sure to work through the IETF draft you linked
to. This is one reason we haven't removed the large "DRAFT" watermark
from the site yet.

one thing is the mechanical aspect. the more important one are the
semantics: once you register, you cannot make breaking changes. things
cannot be unregistered, they will simply be marked as deprecated.
submissions must be in some specific format. there must be associated
online resources for submissions. the list goes on, but i think it would
be essential to define these things before the registry gets populated.

if you feel like there is anything wrong/missing in the draft, please
ping me or even better, raise an issue. i haven't worked on it in a
little while, but it is on my list of things i'd like to pick up again.

[MikeRalphson]

once you register, you cannot make breaking changes. things cannot be unregistered, they will simply be marked as deprecated.

Yep, I think that's understood by all.

submissions must be in some specific format

Submissions will be made by GitHub PR (by the requester, or by an @OAI/tsc member on their behalf), a GitHub PR template will capture all mandatory metadata (such as a base_type for a format, a schema for an extension etc).

there must be associated online resources for submissions

Could you expand on the need for "online resources", if external, that sounds fragile.

@tedepstein @earth2marsh I think you've both mentioned the need for process around this, it would be really helpful if you could start to work up a draft CONTRIBUTING.md / Pull Request template, as this would help to stop this looking like a one-man crusade :smile:

[MikeRalphson]

@dret

there may be extensions for AsyncAPI (if that ever gets to be a proper OpenAPI-based thing)

Just FYI, at the moment AsyncAPI is pursuing a course with their own foundation outside the OAI (which sidesteps issues with our charter scope).

[dret]

On 2019-01-14 02:54, Mike Ralphson wrote:

@dret https://github.com/dret

here may be extensions for AsyncAPI (if that ever gets to be a
proper OpenAPI-based thing)

Just FYI, at the moment AsyncAPI is pursuing a course with their own
foundation outside the OAI (which sidesteps issues with our charter scope).

i know. i just wanted to use an example that at least semi-practically
makes clear why separation of concerns is a good thing here.

[dret]

On 2019-01-14 01:16, Mike Ralphson wrote:

once you register, you cannot make breaking changes. things cannot
be unregistered, they will simply be marked as deprecated.

Yep, I think that's understood by all.

great. i haven't looked at how the registry is defined and described in
the spec, and how the registry page talks about submissions, but this
should be very prominent in both places.

submissions must be in some specific format

Submissions will be made by GitHub PR (by the requester, or by an
@OAI/tsc https://github.com/orgs/OAI/teams/tsc member on their
behalf), a GitHub PR template will capture all mandatory metadata (such
as a |base_type| for a format, a |schema| for an extension etc).

very good. most importantly, there must be an "abstract schema" for what
submissions must contain (fields, their types, mandatory or optional).
how to represent and submit also matters, but is secondary to that.

there must be associated online resources for submissions

Could you expand on the need for "online resources", if external, that
sounds fragile.

yes it is. but that's how things are in decentralized systems. it is
unlikely that your submissions will capture all data. so you will
probably also ask for a URI to a persistent resource that fully
documents the extension. this could disappear, that's true, but part of
the submission requirements would be that it would supposed to be
persistent and stable (i.e., it doesn't get changed in terms of the
semantics).

@tedepstein https://github.com/tedepstein @earth2marsh
https://github.com/earth2marsh I think you've both mentioned the need
for process around this, it would be really helpful if you could start
to work up a draft |CONTRIBUTING.md| / Pull Request template, as this
would help to stop this looking like a one-man crusade 😄

sounds good. let me know if you guys need an extra set of eyes. i think
this is the opposite of a one-man crusade. it is what happens when
standards grow up: people get invested enough that they want to use but
extend the standard. good standards are really careful about enabling
and managing that part of the standards journey carefully.

i've seen at least four registries already being considered, right?
that's a good sign, i think.

here's an idea: these four registries should all use the same general
approach. anybody interested in using this as a great real-life example
for coming up with something that would support the registry mechanics
we're talking about here, but that would be reusable?

[MikeRalphson]

Thanks @dret, that's all very helpful and I really appreciate the offer of an extra set of eyes.

The four initial registries all use the same basic jekyll static site collections to achieve their persistent URLs from metadata + markdown.

The registry site itself could be easily re-used (say by JSON Schema for their own formats registry, or by AsyncAPI) and is all Apache-2.0 licensed like the spec. itself. It wouldn't take very much to make it a white-box solution.

[dret]

On 2019-01-14 10:04, Mike Ralphson wrote:

Thanks @dret https://github.com/dret, that's all very helpful and I
really appreciate the offer of an extra set of eyes.

The four initial registries all use the same basic jekyll static site
collections to achieve their persistent URLs from metadata + markdown.

The registry site itself could be easily re-used (say by JSON Schema for
their own formats registry, or by AsyncAPI) and is all Apache-2.0
licensed like the spec. itself. It wouldn't take very much to make it a
white-box solution.

that looks great. any interest in actually doing this? i'd be on board.

my claim is that many orgs actually could use registries but that they
are a bit hard to get off the ground. having a repo i can fork and brand
and then simply configure (i want these five registries, and here is my
config data for them) would be very helpful.

[tedepstein]

@MikeRalphson wrote:

@tedepstein @earth2marsh I think you've both mentioned the need for process around this, it would be really helpful if you could start to work up a draft CONTRIBUTING.md / Pull Request template, as this would help to stop this looking like a one-man crusade 😄

Happy to help. I'll try to have a draft by end of this week. @earth2marsh , please ping me if you'd like to pair-program, tag-team, or whatever. :-)

[tedepstein]

@MikeRalphson, I'm starting on the Pull Request template and CONTRIBUTING.md guide for the registries.

I'm not clear yet on what a PR for a new registry entry should look like, and I assume we don't yet have an example of such a PR. I have the impression, based on what you've written here and in related issues, that the PR format would be fairly obvious if I understood how Jekyll works; I just don't have any experience with Jeckyll.

If Jeckyll is the right place to start, I'm happy to do some background reading, and hopefully the pieces will all fall in place. You pointed me to the Jeckyll docs in response to an earlier question. Should I start there?

[MikeRalphson]

I'm not clear yet on what a PR for a new registry entry should look like, and I assume we don't yet have an example of such a PR

Exactly so. The PR should add one new markdown file with YAML front-matter, where the template section is based on an existing entry for the same registry. (The template is a mixture of values brought in from the YAML front-matter and other Jekyll collection/site settings and free-form text like examples).

The PR template would have a number of check-boxes to ensure the relevant data has been collected. I'm not sure whether one generic PR template would do, or if one PR template per registry is better... I've just thought of one wrinkle, how does the PR author add the issue YAML property linking to a PR they haven't yet finished creating? This could be solved by always having an issue first then a linked PR, with the PR and the YAML referencing the original issue number.

You pointed me to the Jeckyll docs in response to an earlier question. Should I start there?

Yes, I think that's a good starting point. For googling other resources, drop the stray c from Jekyll. :grin:

Once you've read the basics, have a look at Jekyll collections (https://jekyllrb.com/docs/collections/) which is the feature we're exploiting to build the registries out.

[dret]

On 2019-01-16 16:11, Ted Epstein wrote:

I'm not clear yet on what a PR for a new registry entry should look
like, and I assume we don't yet have an example of such a PR. I have the
impression, based on what you've written here and in related issues,
that the PR format would be fairly obvious if I understood how Jekyll
works; I just don't have any experience with Jeckyll.

i wouldn't spend time learning jekyll right now. jekyll simply has a way
of picking up structured data from pages, and this structured data has
to be provided in the preamble (the initial part of the page) and has to
use YAML format.

jekyll then parses this data and makes it available to scripting. some
of the data has built-in support in jekyll or certain packages, other is
generic data that is parsed and simply made available to jekyll template
code (which uses a language call liquid).

If Jeckyll is the right place to start, I'm happy to do some background
reading, and hopefully the pieces will all fall in place. You pointed me
to the Jeckyll docs in response to an earlier question. Should I start
there?

again, i'd highly recommend to first have the data model of what's
required, and i think we discussed that already somewhere. it basically
should be everything that matters for representing the registry, so that
i could extract and rebuild the registry somewhere else simply by using
the data i can get through the API.

[MikeRalphson]

so that i could extract and rebuild the registry somewhere else simply by using the data i can get through the API.

I'm so not sure that's an actual use-case we would want to enable. :slightly_smiling_face: Though it may be a worthwhile intellectual exercise to ensure we have moved all the relevant data into the YAML front-matter.

[tedepstein]

@MikeRalphson , are registries considered part of the 3.x development scope?

@darrelmiller, should we add this to #1466, the Big List of Possibilities, so we can keep it on our radar?

[MikeRalphson]

Ted: I added a discussion item for this exact topic to this week's agenda earlier today.