Openapi-specification: Proposal: x-oas-draft-alternativeSchemas

[darrelmiller]

A long standing request has been to add support for alternate schemas. See #764 #1443

There are many concerns about the effect this will have on the tooling ecosystem, however, there is also a strong desire from the community to support both more recent versions of JSON Schema and other formats such as XSD Schema, Protobuf Schema.

Instead of simply adding this feature to the specification and hoping that tooling vendors will implement it. I propose we use the draft feature (PR #1531 ) process to validate the value and feasibility of this feature.

openapi: 3.0.2
info:
  title: A sample using real JSON Schema and xsd
  version: 1.0.0
paths:
  /:
    get:
      responses:
        '200':
          description: Ok
          content:
            application/json:
              x-oas-draft-alternate-schema:
                type: json-schema
                externalValue: ./rootschema.json
            application/xml:
              x-oas-draft-alternate-schema:
                type: xml-schema
                externalValue: ./rootschema.xsd

• The property is called alternate-schema because it can be used in instead of, or in addition to the OAS Schema object. If both are present then both schemas must be respected.
• The type field is required and must match one of the values identified in an alternate schema registry which will be created as part of this proposal. The alternate schema registry will provide a link to a specification for the schema file.
• It is recommended that tools provide users with warnings when they encounter an alternate schema type that they do not support. Tools should not fail to process a description, unless the schema is essential to the operation. Ideally they should continue as if the alternate-schema is not present in a form of graceful degradation.
• The externalValue property is required and must be a relative or absolute URL.
• The alternate-schema property can be used anywhere the schema property can be used.

[philsturgeon]

Where did we land on alternate schemas various versions? Saying "json-schema" here might be fine, as JSON Schema files contain "$schema": "http://json-schema.org/draft-XX/schema#", and Protobuf has syntax = "proto3";, but other formats might not?

It would be up to tool vendors to sniff out the right version, which might cause trouble.

Adding the version in the file does add a lot more work for everyone. When users upgrade their JSON Schema files, they would have to update all the OpenAPI references too, which doesn't sound ideal. Tool vendors would also need to update their registries of "supported alternate schemas" just to support a newer JSON Schema draft, which the underlying tools might already support just fine.

I'm conflicted. Probably leave them out and assume that formats declare their versions?

[handrews]

Please, please, PLEASE do not make people specify the JSON Schema draft version. Right now that's fairly straightforward, but as we move towards full vocabulary support including all of the things that will facilitate code generation, this will become far too complex. The whole point is to allow schema authors and vendors more flexibility in using and detecting support for vocabularies. Please don't duplicate the entire effort inside of OpenAPI.

[darrelmiller]

@philsturgeon Can we identify existing schema languages that are not clearly self-description as to version? I'm going to guess all the XML ones will use a namespace. JCR doesn't seem to have any way to version stuff, but then it is still draft.

Personally I'm a fan of formats that identify their own version. There is something icky about relying on external metadata. So, I wouldn't be opposed to just not offering a way to specify a version.

[philsturgeon]

I am no longer conflicted. 👍🏼

[handrews]

@darrelmiller do people ask about JCR in OpenAPI? It's on draft-09 but still lists only one complete implementation. No need for a big tangent here, just curious as to whether it comes up a lot or is just the first JSON-related example you thought of :-)

[darrelmiller]

@handrews It's the only other schema language in JSON that I know of. And no, nobody has ever asked for support.

[MikeRalphson]

There is also schematype for Yaml which has come up before as a possible alternate schema-description language, though I'm not sure about requests for support.

[handrews]

the only other schema language in JSON

Technically "for JSON", as its syntax is not JSON, merely _"JSON-like", possessing the conciseness and utility that has made JSON popular._ (oh wait, I'm being pedantic again, oops...)

[MikeRalphson]

As the guidance for the json schema $schema keyword is

The "$schema" keyword SHOULD be used in a root schema. It MUST NOT appear in subschemas.

I don't think we need any additional recommendation that people declare their json schema version. Presumably the $schema keyword is 'in scope' / visible the same way $refs outside the fragment being $ref'd by the OpenAPI document are resolvable.

Unless the fragment being $ref'd from the OpenAPI document counts as a "root schema"?

[handrews]

@MikeRalphson "root schema" is the root of a "schema document" which is a whole resource. If you are referencing a fragment other than "#" then you're referencing a subschema.

Certain concepts, most notably $schema and base URI, work based on the schema document scope, not how you got to a schema object.

[handrews]

I think with the vocabulary stuff in draft-08 we'll be clarifying the schema document / file-scope stuff a lot, including how $schema and the closely related proposed $vocabulary work.

[MikeRalphson]

Thanks @handrews

[darrelmiller]

UPDATED Proposal based on feedback from TSC meeting of April 16th.

This proposal is to enable to use of schemas other than the native OAS Schema object for describing request and response payloads and complex parameter/header structures.

openapi: 3.0.2
info:
  title: A sample using real JSON Schema and xsd
  version: 1.0.0
paths:
  /cat:
    get:
      responses:
        '200':
          description: Ok
          content:
            application/json:
              schema:
                type: object
              x-oas-draft-alternateSchemas:
                - type: json-schema
                  externalValue: ./cat-schema-draft-08.json
                - type: json-schema
                  externalValue: ./cat-schema-draft-05.json
                - type: json-content-rules
                  externalValue: ./cat-JCRschema.json
            application/xml:
              x-oas-draft-alternateSchemas:
                - type: xml-schema
                  externalValue: ./cat-schema.xsd
                - type: relax-ng
                  externalValue: ./cat-schema.ng
                - type: schematron
                  externalValue: ./cat-schema.st

• The property is called alternate-schemas because they can be used in instead of, or in addition to the OAS Schema object. If both OAS Schema and alternate schemas are present then both the OAS schema and one of the alternate schemas SHOULD be respected.
• The alternate-schemas property is a list of schemas. They SHOULD be processed in the order included in the list. Tooling is expected to walk the list until it finds an alternate schema that it knows how to process. Multiple alternate-schemas are supported to enable a single specification be used to drive tools with different schema support capababilities.
• The type field is required and must match one of the values identified in an alternate schema registry which will be created as part of this proposal. The alternate schema registry will provide a link to a specification for the schema file.
• It is possible for the same schema type to appear multiple times in the list of supported schemas. This is to provide support for different versions of the same schema type.
• It is recommended that tools provide users with warnings when they are unable to find an alternate schema that they support. Tools should not fail to process a description, unless the schema is essential to the operation. Ideally they should continue as if the alternate-schema is not present in a form of graceful degradation.
• The externalValue property is required and must be a relative or absolute URL.
• The alternate-schema property can be used anywhere the schema property can be used.
• The use of an array of alternate schemas provides the ability for x- extensions to be included in the alternate schema information.

[cmheazel]

In this example shouldn't the externalValue property reference the schema for the returned content? For example, if I want to return GeoJSON (an IETF standard) then the externalValue could resolve to https://github.com/fge/sample-json-schemas/blob/master/geojson/geojson.json. The client would then validate the returned content against the GeoJSON schema.

[darrelmiller]

@cmheazel Yes you are correct. I see how my example file names are confusing. I'll fix that.

[cmheazel]

@darrelmiller here are some candidates (in addition to GeoJSON)

Geography Markup Language (GML) - MIME type application/gml+xml
Version 3.2.1 = http://schemas.opengis.net/gml/3.2.1/gml.xsd
Version 3.1.1 = http://schemas.opengis.net/gml/3.1.1/base/gml.xsd
Version 3.0.0 = http://schemas.opengis.net/gml/3.0.0/base/gml.xsd

And an in-development JSON encoding of the GML data model:
http://example.com/GML-SF0-JSON.json

[cmheazel]

Another question - XML validation can be a two step process; XML schema validation followed by XML content validation (Schematron). Would it be legal to have two externalValue properties? Validation would be the AND of the two.

[cmheazel]

I have some folks who want to use this feature in a RESTful Web Services standard. When can we lock this feature down and establish a timeline for adoption?

[darrelmiller]

This a proposal to add a new field called alternativeSchemas to the Media Type Object. We are considering the possibility of also allowing it to be added to the Parameter Object for simple parameter definitions. However, we will wait for implementer feedback on that issue.

Media Type Object

Fixed Fields

Field Name | Type | Description
---|:---:|---
alternativeSchemas | [alternative Schema Object] | List of alternative schemas. These schemas can be used in addition to, or as an alternative to any existing schema property. If both OAS Schema and alternative schemas are present then both the OAS schema and one of the alternative schemas SHOULD be respected. Alternative schemas MUST be processed in order. It is sufficient for tooling to process just the first alternative schema they are capable of processing in order to consider the content as valid. If tooling cannot process any alternative schemas, then they MAY issue a warning, but MUST not report the OpenAPI description as invalid.

Alternative Schema Object

This object makes it possible to reference an external file that contains a schema that does not follow the OAS specification.

Fixed Fields

Field Name | Type | Description
---|:---:|---
type | string | REQUIRED. The value MUST match one of the values identified in the alternative Schema Registry name of the tag.
externalValue | url | REQUIRED. This is a absolute or relative reference to an external file containing a schema of a known type.

This object MAY be extended with Specification Extensions.

Alternative Schema Registry

The Alternative Schema Registry is located at https://spec.openapis.org/registries/alternative-schema. Inital contents of the registry include:

Name | Link | Description
---|:---:|---
jsonSchema | |
xmlSchema | |

[tedepstein]

Question: have we considered allowing scenarios where schema is used as an "information-only" approximation of the "real" schema, specified under alternateSchemas?

This could be useful to provide documentation in formats or interactive UIs that are not able to process the authoritative schema in its native format, but can still present standard OAS Schema Object as an abstraction, or high-level guideline to the message structure.

[tedepstein]

Also, can you include an OAS Schema Object (i.e. a "standard" OpenAPI schema) in alternateSchemas list, as a way to specify its order of preference with respect to other alternateSchemas?

If you don't do this, but you do specify both schema and alternativeSchemas, is the standard schema entry assumed to go at the _end_ of the list, as the _least preferred_ option?

[handrews]

@darrelmiller

We are considering the possibility of also allowing it to be added to the Parameter Object for simple parameter definitions. However, we will wait for implementer feedback on that issue.

So this means it cannot be used anywhere outside of the Media Object? That feels more complicated to me. In particular, it makes the case of sharing a schema between the Media Object and Parameter Object (or other places where a schema can occur) impossible with alternate schemas, so now instead of being able to use fewer schemas you need to use even more.

[darrelmiller]

@tedepstein @handrews Following a conversation with @webron I am considering an alternative place to put the alternative schema field. :-)

Specifically because of the fact that you can $ref native schema objects, it might make sense to put the alternativeSchemas property in the native Schema Object. This gives a few benefits:

• easier to add alternativeSchemas that currently exist in a document and are re-used in multiple places.
• tooling implementers only have to implement the property in one object
• we are able to use alternativeSchemas anywhere the schemas are currently used.

This would make it look like this:

openapi: 3.0.2
info:
  title: A sample using real JSON Schema and xsd
  version: 1.0.0
paths:
  /cat:
    get:
      responses:
        '200':
          description: Ok
          content:
            application/json:
              schema:
                type: object
                x-oas-draft-alternativeSchemas:
                  - type: json-schema
                    externalValue: ./cat-schema-draft-08.json
                  - type: json-schema
                    externalValue: ./cat-schema-draft-05.json
                  - type: json-content-rules
                    externalValue: ./cat-JCRschema.json
            application/xml:
              schema:
                x-oas-draft-alternativeSchemas:
                  - type: xml-schema
                    externalValue: ./cat-schema.xsd
                  - type: relax-ng
                    externalValue: ./cat-schema.ng
                  - type: schematron
                    externalValue: ./cat-schema.st

[handrews]

@darrelmiller @webron @tedepstein I like this new idea of putting it in the Schema Object! Much more consistent. I think it's important that schemas (in the general sense) have the same capabilities anywhere they can be used.

From a JSON Schema perspective of processing the OAS-native schemas, x-oas-draft-alternativeSchemas can be treated kind of like an opportunistic allOf.

I was going to ask about externalValue vs $ref, but I guess externalValue makes more sense for non-JSON Schema alternatives. I don't feel strongly about it- if I want to $ref something in some way that doesn't work with externalValue I can make my external document just {"$ref": "whatever"}

Is the type: object in the application/json example required for some reason or is it just there to provide a minimal indicator to tools that don't understand x-oas-draft-alternativeSchemas?

[ralfhandl]

@darrelmiller @handrews I would have assumed that I can use JSON Pointer fragments in externalValue with type: json-schema to avoid having "wrapper files" for addressing a certain part of a "schema collection" within a single file.

[darrelmiller]

@handrews the type:object was just my minimal OAS Schema to indicate that OAS Schemas can be used in combination with alternativeSchemas.

@ralfhandl That's a good question. URL fragments are a natural feature of a URL, so it seems reasonable to enable support for JSON Pointers in URL fragments. We need to decide if we want to add that burden onto tool devs. It does seem like a reasonable requirement but we definitely should call it out if we do decide to support it.

[handrews]

@ralfhandl @darrelmiller URI fragment syntax/support is determined by the media type of the target resource, not the source, so any application/schema+json document should be addressable with either JSON Pointer fragments or plain name fragments (created by "$id": "#this-is-a-plain-name").

Not supporting it would be failing to support the media type properly.

@ralfhandl never mind my $ref example, I can't even remember exactly what I thought externalValue couldn't do. Although if it forbids part of the URI syntax (fragments) that would be a problem and would require weird indirection such as an extra schema document that is just a $ref.

[chriskapp]

Hi, could we not also simply add the $schema keyword to the Schema Object to indicate the JSONSchema version of the provided schema i.e.:

application/json:
  schema:
    $schema: "http://json-schema.org/draft-07/schema#"
    type: object

If no $schema is available we would assume the OAI JSON Schema. For alternate schemas XSD etc. we could still use the x-oas-draft-alternativeSchemas keyword but I think for tool developers (like me) this would make things much easier.

[handrews]

@chriskapp how would handling it inline be easier? Mostly people seem to want to be able to just hand the external file off to whatever 3rd-party schema tooling is being used. Are you planning to directly implement standard JSON Schema support within your OpenAPI implementation?

[cmheazel]

Can we please stop changing this? I'm trying to implement.

[chriskapp]

@handrews yes this was my idea/hope to use a normal JSONSchema parser inside OpenAPI. If we take a look at the solution:

application/json:
  schema: <-- OpenAPI JSONSchema standard
    type: object
    x-oas-draft-alternativeSchemas:
      - type: json-schema
        externalValue: <-- JSONSchema standard draft-04, draft-08, etc.

a parser would then basically look for both schemas and then its maybe not clear which schema to use. Maybe the logic is to use the OAI JSONSchema for code generation and the normal JSONSchema only for validation? On the other hand a generator maybe uses the same schema multiple times, maybe also with the extra burden to generate a separate file. I think this makes the implementation and tooling more difficult since OAI JSONSchema and JSONSchema could be the same with some extra effort.

@cmheazel I dont want to generate work here but Iam under the impression that this topic is still under discussion and I feel like its important to give feedback from a tool developer perspective.

[MikeRalphson]

@chriskapp perhaps this wording

If both OAS Schema and alternative schemas are present then both the OAS schema and one of the alternative schemas SHOULD be respected. Alternative schemas MUST be processed in order. It is sufficient for tooling to process just the first alternative schema they are capable of processing in order to consider the content as valid.

Needs to be clarified to indicate that "capable of processing" is specific to the context for which the tool is using the schema/alternativeSchema (i.e. validation, documentation generation, code generation etc).

Thank you for the link to JSON Schema Code: JSON Schema rules to simplify code generation I hadn't previously come across this. Perhaps it may inform the proposed JSON Schema code-generation vocabulary?

[cmheazel]

@chriskapp I was under the impression we had settled this at the last TSC meeting. Re-reading the minutes it appears that I was mistaken.
Personally, I found the version we discussed Monday to be simple, elegant, and easy to implement.

[chriskapp]

@MikeRalphson yes this spec was a first attempt to build such a vocabulary for the JSON Schema spec to combine OAI JSON schema and JSON schema. There is also a PR at the JSON schema vocabulary repository regarding this. Iam currently not sure about the state of the vocabulary separation inside the JSON Schema spec (@handrews can probably say more about this) but maybe its worth to take those ideas into consideration.

[handrews]

@chriskapp my apologies for not recognizing you- you changed your github id didn't you? :-)

All: @chriskapp wrote that spec (and the related PR) incorporating advice from me on stating things in terms of restricted interpretations. However, the difficulties associated with that approach (you can see me comment on some in the PR) were a major motivation for the formalized annotation collection and extensible vocabularies approach planned for draft-08.

Rather than restricting keywords (which is not visually apparent when looking at the keywords in the schema), I expect things like code generation to add annotations that disambiguate keywords that are difficult to handle. There is a bit more about this in the PR, although nothing like a new proposal. I don't want to derail this issue, but this shift in approach to generative use cases for JSON Schema (code generation, UI generation, documentation generation) is arguably the most important thing in draft-08. And very relevant to OpenAPI.

[darrelmiller]

@cmheazel I thought we had a reasonable approach on Monday, but then it was brought to my attention that if we have the alternativeSchema as a peer to schema that is going to be a real pain for schemas that are referenced. This was big enough of an issue for me to change the status back to draft:proposal from draft:pilot. My apologies for not making that clear.

[darrelmiller]

I want to update this proposal with the new approach using alternativeSchemas as a property of the schema object. However, this opens a whole new set of questions. The Schema object is defined recursively and therefore we need to ask if it makes sense to have representations where portions are defined by the native Schema Object and other parts by an alternative schema. Also, if the Schema object has an oneOf and allOf, do we really need alternativeSchemas to be a list.

Here are some spitballed ideals...

Simple case

type: object
alternative-schema:
  type:  jsonSchema 
  externalValue: ./myschema.json

Nested alternative schemas

type: array
items:
  alternative-schema:
    type:  jsonSchema 
    externalValue: ./myschema.json

Support for tools with different capabilities

type: object
oneOf:
  - alternative-schema:
      type:  jsonSchema 
      externalValue: ./myschema-draft-06.json
  - alternative-schema:
      type:  jsonSchema 
      externalValue: ./myschema-draft-07.json

Layered validation

type: object
allOf:
  - alternative-schema:
      type:  xmlSchema 
      externalValue: ./myschema.xsd
  - alternative-schema:
      type:  schematron 
      externalValue: ./myschematronvalidation.st

[darrelmiller]

@chriskapp Based on various feedback I'm getting the impression we are going to have to have a way to further qualify the "version" of the type of alternate schema. My immediate reaction is that I'm not a fan of the $schema notion just because it is feels specific to JSON Schema and we need this ability for other schema types also. But I agree we need to think more about this.

[cmheazel]

@darrelmiller I thought we had a reasonable approach as well. I've seen these efforts turn into a debating club. Round and round without ever reaching a conclusion. I was afraid that was happening here. That being said, the discussion has moved forward addressing valid issues. So I'll be patient and behave myself.

[tedepstein]

The property is called alternative-schema because it can be used in instead of, or in addition to the OAS Schema object. If both are present then both schemas must be respected.

When we say both schemas must be respected, that means the content must conform to both schemas in order to be valid. But it doesn't necessarily mean that implementations are required to validate against both schemas, or otherwise "use" both schemas. Is that right?

If we move x-oas-draft-alternative-schemas into the Schema Object, then I think this also means:

1. If the Schema Object _does not_ specify any constraining properties (type, properties, maxLength, etc.), then the content does not need to be JSON. It only needs to conform to the specified alternative schemas.
2. If the Schema Object _does_ specify constraining properties, then the content must be JSON, and must conform to the native Schema Object constraints, as well as any alternative schemas.
3. (Maybe) if the Schema Object specifies non-constraining properties like title, definition, and documentation those become part of the schema object definition, but do not require the content to be valid JSON.

Thoughts about that last point? I think it's valuable to assign non-constraining metadata to the Schema Object without implying that the content must be valid JSON.

Also, I think we may still need to clarify the relationship of the native OAS Schema Object vs. alternative schemas:

• The alternative schemas form a sequence, in order of preference from the designer's point of view. I assume we will give guidance that implementations SHOULD honor this order, by using the most preferred schema that the implementation is capable of using for its purpose (validation, documentation, etc.).
• In the case where a native Schema Object and an alternative schema are both applicable, are implementations required (MUST) or encouraged (SHOULD) to use both of these for validation purposes? (I would vote no.)
• In that case, should the matching alternative schemas take priority over the native schema objects, so the native Schema Objects are least preferred? (This seems like a reasonable default behavior.)
• Will native OpenAPI 3.x Schema Objects be one of the entries in our schema types registry, so the API can specify this in the alternative schemas list, and thus assign a position in the order of preference? (This seems like it would be useful.)

[tedepstein]

Will there be a way to designate the native Schema Objects as "information-only", so they can be used in documentation as a general guide to the structure of the message? I think this is important for documentation formats in particular. It will be difficult (or at least will take some time) for documentation presentation formats to support all of the most common schema types, and it is always possible to introduce new schema types that aren't universally supported by mainstream documentation formats. Users will be discouraged from using these formats if they cannot also use common documentation formats with them.

@chriskapp wrote:

Maybe the logic is to use the OAI JSONSchema for code generation and the normal JSONSchema only for validation?

This suggests that we might need a usage keyword, whose value is an array of SchemaUsageType values, where SchemaUsageType is an enum of common schema usage contexts: documentation, validation, code generation, and maybe a few others.

The default would be all usages, so the presence of a usage value has the effect of limiting the usage. This also changes the schema selection algorithm: tools should use the first applicable schema that the tool is capable of processing, where "applicable" means that the schema matches the media type of the content, _and_ matches the usage context.

          content:
            application/json:
              schema:
                type: object
                x-oas-draft-alternativeSchemas:
                  - type: json-schema
                    externalValue: ./cat-schema-draft-08.json
                    usage: 
                    - documentation
                  - type: json-schema
                    externalValue: ./cat-schema-draft-05.json
                  - type: json-content-rules
                    externalValue: ./cat-JCRschema.json
            application/xml:
              schema:
                x-oas-draft-alternativeSchemas:
                  - type: xml-schema
                    externalValue: ./cat-schema.xsd
                    usage: 
                    - documentation
                    - validation
                  - type: relax-ng
                    externalValue: ./cat-schema.ng
                  - type: schematron
                    externalValue: ./cat-schema.st

@MikeRalphson wrote:

Needs to be clarified to indicate that "capable of processing" is specific to the context for which the tool is using the schema/alternativeSchema (i.e. validation, documentation generation, code generation etc).

Yes, similar idea, factored a bit differently. ;-) But I'm suggesting we make the intended usage contexts explicit in the API spec.

@handrews wrote:

Rather than restricting keywords (which is not visually apparent when looking at the keywords in the schema), I expect things like code generation to add annotations that disambiguate keywords that are difficult to handle. There is a bit more about this in the PR, although nothing like a new proposal. I don't want to derail this issue, but this shift in approach to generative use cases for JSON Schema (code generation, UI generation, documentation generation) is arguably the most important thing in draft-08. And very relevant to OpenAPI.

This sounds kind of similar to what I'm suggesting, but I really cannot tell without more background on the PR you're describing. Should I read up on some of this to get a different perspective on this idea of usage context, and how we might support it in OAS alternative schemas?

[tedepstein]

@cmheazel wrote:

Another question - XML validation can be a two step process; XML schema validation followed by XML content validation (Schematron). Would it be legal to have two externalValue properties? Validation would be the AND of the two.

I know we talked about this on the last meeting. Where did that land?

[tedepstein]

@handrews wrote:

I was going to ask about externalValue vs $ref, but I guess externalValue makes more sense for non-JSON Schema alternatives. I don't feel strongly about it- if I want to $ref something in some way that doesn't work with externalValue I can make my external document just {"$ref": "whatever"}

The $ref would imply that it's a JSON Reference, which is not _necessarily_ what we want in all cases, but it does give users and tool vendors clear guidance as to how to handle the fragment. In the absence of that, how will we manage referencing of schemas somewhere within a resource? Will we apply some standard across all of the schema types? Or will each schema type in our registry specify hash fragment interpretation (e.g. XPath for XML Schema)...?

URL fragments are a natural feature of a URL, so it seems reasonable to enable support for JSON Pointers in URL fragments. We need to decide if we want to add that burden onto tool devs. It does seem like a reasonable requirement but we definitely should call it out if we do decide to support it.

JSON Pointer fragments are already supported, and have great practical value for packaging related definitions into a single resource, and referencing them there. My concern is that JSON pointer is (AFAIK) useless for schema formats that are not JSON.

So I'm thinking that standardizing the fragment semantics as part of the schema type registry entry might be a pretty sensible way to deal with this.

[handrews]

@tedepstein

So I'm thinking that standardizing the fragment semantics as part of the schema type registry entry might be a pretty sensible way to deal with this.

Per RFC 3986 §3.5:

The semantics of a fragment identifier are defined by the set of
representations that might result from a retrieval action on the
primary resource. The fragment's format and resolution is therefore
dependent on the media type [RFC2046] of a potentially retrieved
representation, even though such a retrieval is only performed if the
URI is dereferenced. ... Fragment identifier semantics are independent of the
URI scheme and thus cannot be redefined by scheme specifications.

OAS cannot define fragment syntax and semantics, in a registry or otherwise. Only the media type of the target representation can define those semantics. JSON Schema defines semantics for JSON Pointer and plain name fragment syntax. XML defines fragment based on the XPointer framework. These are not re-definable.

[handrews]

@tedepstein

If the Schema Object does specify constraining properties, then the content must be JSON, and must conform to the native Schema Object constraints, as well as any alternative schemas.

JSON Schema does not require the content to be JSON, only that the content can be mapped into the data model, which is derived from JSON. Some media types, such as YAML as long as it doesn't use the more exotic features, are trivial to map, while others (XML) do not have an obvious mapping.

(Maybe) if the Schema Object specifies non-constraining properties like title, definition, and documentation those become part of the schema object definition, but do not require the content to be valid JSON.

This gets into the difference between assertions and annotations. How annotations get used is application-defined, so OAS can say anything it wants about that.

[handrews]

@tedepstein I am definitely against a "usage" concept. With multi-vocabulary support in draft-08, the vocabulary in use should indicate its purpose. Of course applications can attempt to do whatever they want with a schema (such as generate code from the validation vocabulary alone, even though it's not well-suited for such things), but the way you'd tell if a schema is intended for code generation is to check whether it uses the code generation vocabulary.

[darrelmiller]

@tedepstein Regarding this:

When we say both schemas must be respected,
Perhaps a better word would have been "processed" instead of "respected". If the tool doesn't understand an alternativeSchema, it cannot be expected to guarantee that it validates.

The idea of adding a usage property concerns me significantly. The application of schemas are currently vauge and I actually think that is a feature. I think any attempt to nail down an enumeration of usages would be both insanely challenging and ultimately restrictive.

The alternativeSchema feature is an escape hatch for people who know what they are doing, understand the limitations of OAS Schema and are aware of the limited tooling support for alternative schemas.

OAS schema objects are already used for non-JSON payloads, e.g. form encoding.

I believe if OAS schema constraints exist, their impact will be unaffected by the addition of the new alternativeSchema property. alternativeSchema is always in addtion to the OAS Schema object. This should never cause a problem because an empty OAS Schema Object allows everything.

If you look at the example snippets I gave in my last comment, the one labelled Layered Validation shows how allOf could be used to address the scenario @cmheazel brought up with Xml Schema and Schematron.

[darrelmiller]

So far, I like the idea of adding alternativeSchema to OAS Schema because:

• It makes $refs just work
• It makes alternativeSchema an object instead of a list, making the simple case even simpler
• It enables allOf and anyOf scenarios covering both reasons for having multiple alternative schemas
• It enables single representations to be validated by multiple schema languages. e.g. folks who embed encoded XML in JSON properties could actually validate that XML.

The one thing I'm not sure of is whether JSON Schema's rules for processing oneOf match what we were propsing. i.e. Walk the array in order and stop as soon as you find one that works.

I do think we should be explicit in indicating that externalValue can have a fragment identifier and therefore can point to a portion of a target document.

The only remaning issue I see is that not having a schema version property is going to force tools to load a schema to learn whether they support that version. That may be prohibitively expensive. So we probably need a generic way of identifying a version of a schema.

[tedepstein]

The one thing I'm not sure of is whether JSON Schema's rules for processing oneOf match what we were propsing. i.e. Walk the array in order and stop as soon as you find one that works.

It sounds like anyOf would be a better fit for this.

[philsturgeon]

I'm worried that we're overthinking some of this, and this thread feels like we're jumping the shark. 🏄🦈

I really liked the initial proposal, as defined here my Darrel in this comment.

[darrelmiller]

@philsturgeon You don't see the $ref issue of having to keep the $ref'd schema in sync with alternate schema externalValue to be problematic? Especially if the $ref'd OAS schema is actually in another file.

[tedepstein]

I’m really liking the level of integration afforded by making alternative schemas a property of the Schema Object, especially the possibilities of using Boolean assertions for composition: allOf, anyOf, oneOf, etc.

I posed bunch of questions and concerns in previous comments, before I had fully taken in this new idea. Now, I actually think I can go back and answer many of my own questions, using the new structure.

[handrews]

@darrelmiller oneOf cannot be evaluated in the way that you mention, as you need to verify that exactly one alternative works. anyOf can be short-circuited for _assertion_ processing, but not for _annotation_ processing. You know that the assertion result is true as soon as one subschema validates, but for applications that care about annotations (and code generation will be one such application), you need to examine all subschemas as all annotations from valid subschemas must be collected.

If you want a true "stop at first usable schema" approach then that needs to be an OAS extension keyword. Redefining the behavior of oneOf/anyOf would be very confusing.

You don't see the $ref issue of having to keep the $ref'd schema in sync with alternate schema externalValue to be problematic? Especially if the $ref'd OAS schema is actually in another file.

I'm confused on this- aren't all of the alternate schemas in external files? Why would the OAS schema be any harder to keep in sync? I may have missed something here...

[handrews]

The $ref would imply that it's a JSON Reference, which is not necessarily what we want in all cases

In draft-08 we are redefining $ref from "logically replace this with the target" to "this has the same results as evaluating the target", in part to ensure that you can $ref between schemas that use different processing rules. So OAS could adopt that definition for the OAS-specific schema when $ref is used with x-oas-draft-alternateSchema (to avoid breaking compatibility elsewhere).

[tedepstein]

@handrews wrote:

OAS cannot define fragment syntax and semantics, in a registry or otherwise. Only the media type of the target representation can define those semantics. JSON Schema defines semantics for JSON Pointer and plain name fragment syntax. XML defines fragment based on the XPointer framework. These are not re-definable.

OK, makes perfect sense that it's defined in the media type. As long as fragment resolution is defined somewhere, I'm happy.

[tedepstein]

Closing the loop on some of my earlier comments, to the extent that I think these are addressed by moving alternative schemas into Schema Object...

...can you include an OAS Schema Object (i.e. a "standard" OpenAPI schema) in alternateSchemas list, as a way to specify its order of preference with respect to other alternateSchemas?

Yes, using anyOf (or a new boolean logic assertion, TBD...)

anyOf:
  - alternative-schema:
      type:  jsonSchema 
      externalValue: ./myschema-draft-07.json
  - alternative-schema:
      type:  jsonSchema 
      externalValue: ./myschema-draft-06.json
  - type: object
      properties: 
        foo:
          type: string
     ...

If you don't do this, but you do specify both schema and alternativeSchemas, is the standard schema entry assumed to go at the _end_ of the list, as the _least preferred_ option?

IIUC, including an alternative schema in a Schema Object that also has assertions is equivalent to combining the two with allOf. These two Schema Objects should be equivalent:

Explicit allOf:

allOf:
  - type: object
    properties: 
      foo:
        type: string
     ...
  - alternative-schema:
      type:  jsonSchema 
      externalValue: ./myschema-draft-07.json

Implicit allOf:

  type: object
  properties: 
    foo:
      type: string
     ...
  alternative-schema:
      type:  jsonSchema 
      externalValue: ./myschema-draft-07.json

This was true before the refactoring of alternative schemas into Schema Object. The semantics of a Schema Object with assertions+alternative schema are the same as the previously specified semantics of using schema+x-oas-draft-alternativeSchemas.

But I think it's clearer now, for reasons I'll explain in the next point...

The property is called alternative-schema because it can be used in instead of, or in addition to the OAS Schema object. If both are present then both schemas must be respected.

When we say both schemas must be respected, that means the content must conform to both schemas in order to be valid. But it doesn't necessarily mean that implementations are required to validate against both schemas, or otherwise "use" both schemas. Is that right?

That's from the earlier design.

But now we have (or will have) a way to explicitly place a native OAS schema in a list, in preferential order relative to other schemas. The implicit and explicit allOf semantics are clearer now (to me, anyway) because they don't need to address the order-of-preference use case. That use case is handily covered by anyOf.

So it's clear that validators and other processors _must always_ process all of the native OAS schema assertions, annotations, and/or alternative schema provided in an allOf scope, whether explicit or implicit.

[tedepstein]

@handrews,

oneOf cannot be evaluated in the way that you mention, as you need to verify that exactly one alternative works. anyOf can be short-circuited for assertion processing, but not for annotation processing. You know that the assertion result is true as soon as one subschema validates, but for applications that care about annotations (and code generation will be one such application), you need to examine all subschemas as all annotations from valid subschemas must be collected.

OK, I see that requirement specified here, and I understand the need for it.

Also, anyOf doesn't imply an order of preference, so processors _could_ evaluate the subschemas in any order, and short-circuit validation after evaluating a subschema that is _not_ the most preferred.

If you want a true "stop at first usable schema" approach then that needs to be an OAS extension keyword. Redefining the behavior of oneOf/anyOf would be very confusing.

I think this is important, both to allow more efficient validation, and to avoid the potential complexities of handling conflicting annotations from multiple subschemas.

But these concerns don't seem unique to OpenAPI. If we introduced a new firstOf or firstOneOf that did exactly what we're describing here, would JSON Schema consider adding it to a future draft?

[tedepstein]

Regarding usage: I'll withdraw this proposal.

I'm still not entirely comfortable that we have fully defined how native JSON Schema/Schema Object assertions and annotations are supposed to be applied to content that is not JSON, and does not have a well-defined, authoritative mapping to JSON (like YAML).

I'll try to describe those concerns in a later post or on Monday's call. For now, I just want to pull usage off the table, so @darrelmiller and @handrews can sleep better. ;-)

[darrelmiller]

UPDATED

This a proposal to add a new field called alternativeSchema to the OAS Schema Object. While still in draft, the field name will be prefixed with x-oas-draft-.

Schema Object

Fixed Fields

Field Name | Type | Description
---|:---:|---
x-oas-draft-alternativeSchema | alternative Schema Object | An external schema that participates in the validation of content along with other schema keywords.

Alternative Schema Object

This object makes it possible to reference an external file that contains a schema that does not follow the OAS specification. If tooling does not support the type, tooling MUST consider the content valid but SHOULD provide a warning that the alternative schema was not processed.

Fixed Fields

Field Name | Type | Description
---|:---:|---
type | string | REQUIRED. The value MUST match one of the values identified in the alternative Schema Registry.
location| url | REQUIRED. This is a absolute or relative reference to an external resource containing a schema of a known type. This reference may contain a fragment identifier to reference only a subset of an external document.

This object MAY be extended with Specification Extensions.

Examples

Minimalist usage of alternative schema:

schema:
  x-oas-draft-alternativeSchema:
    type: jsonSchema
    location: ./real-jsonschema.json

Combination of OAS schema and alternative:

schema:
  type: object
  nullable: true
  x-oas-draft-alternativeSchema:
    type: jsonSchema
    location: ./real-jsonschema.json

Multiple different versions of alternative schema:

schema:
  anyOf:
  - x-oas-draft-alternativeSchema:
      type: jsonSchema
      location: ./real-jsonschema-08.json
  - x-oas-draft-alternativeSchema:
      type: jsonSchema
      location: ./real-jsonschema-07.json

Combined alternative schemas:

schema:
  allOf:
  - x-oas-draft-alternativeSchema:
      type: xmlSchema
      location: ./xmlSchema.xsd
  - x-oas-draft-alternativeSchema:
      type: schematron
      location: ./schema.sch

Mixed OAS schema and alternative schema:

schema:
  type: array
  items:
    x-oas-draft-alternativeSchema:
        type: jsonSchema
        location: ./real-jsonschema.json

Alternative Schema Registry

* Note this is a placeholder registry. Don't take the values seriously. *
The Alternative Schema Registry is located at https://spec.openapis.org/registries/alternative-schema. Inital contents of the registry include:

Name | Link | Description
---|:---:|---
jsonSchema | |
xsdSchema | |

[earth2marsh]

Minor nit: does "Value" add anything in the name externalValue? Every name has a value, ideally, after all. :)

Suggest externalSchema to better align with externalDocs. WDYT?

[darrelmiller]

@earth2marsh externalValue was chosen to match the way example object works. ExternalDocs is the containing object that has a url property. We could take that approach, but I feel like externalValue is more explicit.

[tedepstein]

@darrelmiller, in the example labeled "Combined alternative schemas", shouldn't it be allOf rather than anyOf?

I'm assuming this is supposed to work the same way as the previous "Layered Validation" example. The two schemas are complementary, covering different aspects of validation, and are intended to be used in combination.

I'm also assuming that the following rule would apply in allOf scenarios like this:

It is recommended that tools provide users with warnings when they are unable to find an alternate schema that they support. Tools should not fail to process a description, unless the schema is essential to the operation. Ideally they should continue as if the alternate-schema is not present in a form of graceful degradation.

So if the processor only supports some alternative schemas in an allOf group, it should process the ones it can, and warn on the others.

[earth2marsh]

@darrelmiller externalValue in examples complements the in-line value, plus it's actually an example value. In the case of alternativeSchema (an aside, wondering if altSchema would be worth considering?), the item being referenced, while it certainly has a value (as should any reference) it really is an external schema, hence my pref for externalSchema as the name.

[handrews]

and to avoid the potential complexities of handling conflicting annotations from multiple subschemas.

We are further formalizing how annotations are collected in draft-08, building on the work that is already present in draft-07. There are no problems with "conflicting" annotations, the collection process provides applications (in this case, OAS implementations) with enough information to disambiguate the values (or just keep them all, or whatever).

But these concerns don't seem unique to OpenAPI. If we introduced a new firstOf or firstOneOf that did exactly what we're describing here, would JSON Schema consider adding it to a future draft?

Unlikely. I'm not aware of this _ever_ coming up before, and I have read a huge number of JSON Schema issue and feature requests. TBH I'm not even sold on the idea that OAS needs it. If you want to do validation only and not annotations, the spec already allows for short-circuit validation in such cases. If you want to do annotations, there is no conceivable way to short-circuit anyOf, contains, etc.

You can still short-circuit allOf on the first failure (as all annotations are dropped), and you can still short-circuit oneOf on the _second_ successful validation (which fails overall, again meaning that all annotations are dropped). The annotation collection/dropping conditions will be more explicit in draft-08.

[tedepstein]

TBH I'm not even sold on the idea that OAS needs it. If you want to do validation only and not annotations, the spec already allows for short-circuit validation in such cases. If you want to do annotations, there is no conceivable way to short-circuit anyOf, contains, etc.

That's true for the intended anyOf use case. The logic says, for each subschema, if the instance validates against that schema, then include its annotations.

In our firstOf case, we assume that each of the schemas in the list is sufficient; so once the processor finds a schema it can understand, it doesn't need to visit other schemas in the list.

Clearly our use case is different. JSON Schema's boolean assertions (at least through draft-07) naturally assume the processor understands the entire schema language, so the conditional logic is all about whether the instance validates. In our case, we're asking a different question - whether the subschema is usable by the processor. If so, we use that schema, and no others. I don't think it's our intent to proceed to the next schema, if we've already found a schema we can use, and the instance has already failed validation against that schema.

IMO, this justifies introducing a firstOf assertion, rather than trying to repurpose anyOf. But I can understand that this might not be of interest outside of OpenAPI.

[handrews]

@tedepstein the use case is so specific that I wouldn't even call it firstOf, as that sounds like a thing that people could use in general or could be in the regular schema vocabulary. I'd call it schemaList or something else very tied to how it is being used and why it behaves that specific way.

[zolyfarkas]

Can we also add avro(https://avro.apache.org) as alternative schema's, and also a way to provide the schema inline (since avro schema's are json)?

Here is some detail on my use case:

We currently use avro schemas to declare the payload of our REST services. Avro schemas are JSON and as such we currently inline them inside the swagger JSON via custom x-avro-schema field. Every avro REST endpoint we have, depending on the Accept provided by the client (application/json or application/octet-stream) will return the content either as JSON or as AVRO binary, this is very important where the content is large (array) (binary payload can be an order of magnitude smaller than the json payload). Although you can have a limited conversion of a avro schema into a swagger model, the 2 formats do not have feature parity and probably never will, it would be really useful to have a more "official" way of declaring different schema types...

Here is a bit more detail on the use case:

• schema's are developed in schema projects (maven), and can depend on each other for reusability. The maven repository is used to publish/distribute the schema's, language bindings, documentation and/or gh-pages.

• A example REST service built with avro schemas.

Come to think of it, it might make sense to package and distribute open api specs with maven/maven repos as well... right now everybody seems to copy/paste these swagger definitions (like here). Publishing them to a maven repo would make things better...

[adjenks]

This looks great. Decoupling the schema specifications from one another seems like a wise choice. I would like to be able to inline the definition though. Otherwise I'd need to have a pile of small external schemas and that's not particularly convenient.

[darrelmiller]

@adjenks The problem with allowing inlining is that it would only work for JSON and compatible YAML formats. You don't necessarily have to have lots of small external files. You could use a single external schema file and use a JSON Reference fragment identifier to point into the external file.

[fmvilas]

If the external file is not JSON (e.g., Protobuf or XSD) you'll not be able to use JSON Reference fragment identifiers. Or you will if you put the values in strings but in this case why not allowing inlining as a string in the original OpenAPI document?

[philsturgeon]

I think the advice was meant to be: if you want to inline then it has to be JSON or similar. If that’s the case you can have one file and use fragments.

If that’s NOT the case you’re not going to be able to use fragments anyway, so better use solo external files or whatever.

--
Phil Sturgeon
@philsturgeon

On Mar 25, 2019, at 08:16, Fran Méndez [e-mail] wrote:

If the external file is not JSON (e.g., Protobuf or XSD) you'll not be able to use JSON Reference fragment identifiers. Or you will if you put the values in strings but in this case why not allowing inlining as a string in the original OpenAPI document?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[tedepstein]

I also think it's really important to allow alternative schemas to live inside the primary OpenAPI document if that's what makes sense to the API designer. It will certainly be a lot more convenient and a lot more readable for certain users, using certain schema formats in certain usage patterns.

Like other reusable components (where we use $ref), and like example values (where we use value and externalValue), the API designer should be able to decide whether to keep the value inline or refer to an external resource.

In cases where the schema format is not YAML or JSON, they can be embedded as string values. This doesn't put any special burden on tools; extracting and processing a string property value is just as easy, maybe easier, than resolving a URL.

I think this is similar to the current example property and the examples --> [Example Object] construct, used in Parameter and Media Type objects. These are also designed to support JSON, YAML, and other formats. So the example property and the Example Object.value property are defined as type Any.

Example Object allows the URL-typed externalValue property as an alternative to value. I assume OpenAPI chose to separate these two cases instead of using $ref, because we generally use $ref only when it refers to one of our reusable objects in the Components Object. Components Object has a map of Example Objects. Adding a separate map of example values could be confusing, or just overkill, so we didn't do that, and didn't use $ref in that case. Is this more or less correct, historically speaking...?

It looks to me like the same reasoning applies with alternativeSchema.

[tedepstein]

BTW, the parallel with Example Object came up earlier in this discussion. @darrelmiller said we're using externalValue to follow the precedent of Example Object, while @earth2marsh argued that externalSchema would be clearer.

FWIW, I think value and externalValue make sense where you have a Foo Object as a narrowly scoped, thin wrapper around a Foo. You have a few properties that are clearly metadata about the Foo, and you have value or externalValue as the actual Foo entity. That naming is intuitive (to me) as long as Foo Object remains as a thin wrapper. It could get less intuitive if Foo Object later gets augmented with more substantial properties, to the point where it's less clear what value is referring to.

Bottom line, I don't have a strong opinion about that. My argument in support of inline value as an alternative to externalValue applies independently of the naming question. I'll be just as happy if it's schema and externalSchema. Or schemaValue and externalSchemaValue.

[fmvilas]

Food for thought:

1. What would be the downside of inlining something that's not JSON/YAML?

2. Are we really taking into account the user experience? It seems overkill to me that we have to use more than one file even if it's a simple/small API. All just because I use, say, protobuf.

[aowss]

I thought this proposal was for a pluggable type system, i.e. enabling the use of JSON Schema or any other schema language in place of the OpenAPI built-in schema.
I think this is a great idea.

It seems combining different type systems is now part of this proposal.

• This seems to complicate the syntax with the introduction of schema constructs, e.g. anyOf, within the schema element.
• This assumes that these schema languages can be combined using these constructs. This might seem true at first glance but I am not sure it is as obvious as it seems, e.g. the use of default in these different languages.
• The syntax for combining schema is different if one of the schema is an OAS schema
• I am not clear about the real use case for that.

[earth2marsh]

Surely this is a problem that could be solved with RFC1341? Or better yet, CDATA? ;)

In all seriousness though, it already feels weird to me to express in YAML a syntax for describing JSON, but at least it's consistent with the YAML/JSON expression of the spec itself. In-lining other validation languages seems potentially perilous... I even balk at the idea of supporting multiple versions of JSON Schema in the spec just because there's benefit in having one narrow happy path.

@tedepstein for me the "value" is what you get when you dereference a thing, which is why I prefer externalSchema to externalSchemaValue. In this case, "external" connotes that the value lies elsewhere.

OTOH, +1 to @fmvilas 's point about considering the user experience carefully. No question that having XSD inlined simplifies the initial spec authoring task when trying to describe non-JSON formats. But it also bloats spec files over time as schemas and examples grow and grow. Having good multi-file support can help separate that to keep the core spec tighter, but it somewhat depends on having good tooling to support it.

One parting thought, "Why WADL when you can Swagger" goes all the way back to the naming origin where we laughed about how no one wanted to describe JSON APIs in XML, and yet here we are considering the reverse. At least history has a sense of humor?

[cmheazel]

@earth2marsh We are not using XML to define a JSON API. In addition to JSON, an API may deliver resources in XML, Protobuf, JPEG, etc. The use case behind alternative schema was delivery of a resource which is defined by an XML schema AND a set of Schematron rules. The OAS schema construct allows us to build a logical expression capturing the AND / OR relationship when multiple alternatative schema are applicable. That expression describes the resource, not the API. And logically we should use the proper descriptive language for the type of resource we are describing.

[handrews]

@fmvilas

If the external file is not JSON (e.g., Protobuf or XSD) you'll not be able to use JSON Reference fragment identifiers.

That's not quite true. $ref values are just URIs, and in URIs, the fragment syntax is determined by the media type of the target resource. So assuming XSD has a fragment syntax (I'm guessing it inherits XML's, but I'm too lazy to look right now), then $ref-ing an XSD with a fragment to reference some portion of it is valid. At least assuming it is ever valid to reference part of an XSD document (I know basically nothing about XSD).

Protobuf is more complicated b/c it does not have a media type and therefore cannot properly define a fragment syntax. Same with YAML, although one could make a case for JSON Pointer I suppose since using YAML as a media type involves making up an extension media type anyway- that's what OAS does and it works just fine. But those are liimtations of the formats, not limitations of $ref.

@tedepstein

In cases where the schema format is not YAML or JSON, they can be embedded as string values.

Modern JSON Schema accounts for this with contentMediaType and contentEncoding, so if OAS wants to support this we can still write a meta-schema for it if we're willing to move to a recent draft 😛

@earth2marsh

it already feels weird to me to express in YAML a syntax for describing JSON, but at least it's consistent with the YAML/JSON expression of the spec itself.

JSON Schema proper handles this by defining things over a data model which is derived from JSON rather than directly over JSON text. This allows for working with, say, YAML or CBOR as long as the mapping into the data model is clear. Working with XML would be challenging as the data model (elements, tags, contents, CDATA) is more complex.

[darrelmiller]

1) We spent a non-trivial amount of time deciding that we want to support multiple schema types for a number of reasons and it turned out that using the existing schema syntax was the cleanest way to do that.

2) If someone is considering using alternative schemas, they are already a long way past the "simple case". This is an advanced feature.

3) While inlining foreign schemas into seems like low hanging fruit, I don't think it necessarily is. It introduces inconsistencies in handling different types of schemas. It introduces escaping issues. It raises the questions of how future versions of JSON schema should be rendered in YAMLdocuments. We already have this issue with examples and the answers are not clear.

4) I would be willing to discuss how we might support inlined schemas in the future if deployment experience shows it is essential. We can ensure this would not be a breaking change.

[philsturgeon]

I am very confused about what use cases people are proposing. Lets ask a simple question: why on earth would somebody want to have their protobuf definitions inlined into OpenAPI? That would make them _useless_ as they would have to split them out in order to use them for their actual code.

No. These protobufs already exist as .proto files, so referencing them from OpenAPI seems fine.

Smashing them in as strings or CDATA or something is all technically possibly (other than potential escaping troubles as @darrelmiller mentions), but I'm really scratching my head as to why you would want to do that.

[tedepstein]

@philsturgeon,

I am very confused about what use cases people are proposing. Lets ask a simple question: why on earth would somebody want to have their protobuf definitions inlined into OpenAPI? That would make them useless as they would have to split them out in order to use them for their actual code.

That depends entirely on the user and the toolchain. The splitting could be done by a code generator or some other downstream processor. And the potential benefit of having them in the OpenAPI file is readability and ease of maintenance. The exact same reason why most OpenAPI documents have their schemas inline, or in components/schemas, unless and until there's a good reason to move them to a separate file.

Tools might build in validation and other kinds of editing support for alternative schema languages. But some users could find it best to maintain these schemas in the source OpenAPI spec even without some of these editing and codegen features.

Smashing them in as strings or CDATA or something is all technically possibly (other than potential escaping troubles as @darrelmiller mentions), but I'm really scratching my head as to why you would want to do that.

The idea of "strings or CDATA or something" might make me nervous, except for the fact that we've been maintaining an OpenAPI editor for some years now, and have never really had a problem with strings. Between quoted and block styles, with folded and literal variants, YAML seems to have a string syntax for every occasion, and the parsing of string values seems very robust in the stack we're using (SnakeYAML + YEdit, with our own processing mostly written with Jackson, IIRC).

I would like to understand better what kinds of encoding issues we've seen with example properties and Example Objects.

The "can" vs. "should" question applies on another level as well. Just because the OpenAPI specification _can_ decide for the entire ecosystem whether inline alternative schemas are a good idea or a bad one, doesn't mean it _should_.

If there are legitimate difficulties in supporting it, that might be a good reason to disallow this. But the judgment as to whether it's desirable and advisable really should be made by OpenAPI users and tool providers. I don't think we should be taking such a narrow view of what OpenAPI tools can do, and what usage patterns API designers might invent and embrace.

Seriously consider, for a moment, the possibility that something other than JSON Schema might emerge as a wildly popular schema language for OpenAPI, maybe for JSON message payloads, maybe for some other wire format, or both. And consider that this kind of unexpected evolution could be a great thing for OpenAPI and its user community.

Whether or not we think it's likely to happen, we should be laying the groundwork to allow for this kind of open evolution. If we impose a penalty on alternative schemas, taking them out of the flow, and making documents that use them harder to read and harder to maintain, we're stifling growth.

[darrelmiller]

Embedded foreign schemas into an OpenAPI document requires an OpenAPI parser that can extract the foreign schema and then hand them off to an external tool or plugin for validation. Having the foreign schemas in external files means that you can validate/process the foreign schemas with native schema tooling.

Embedded schemas that reference each other for the purpose of reuse would require a referencing system in the foreign schema that can somehow reference another embedded schema in a file format it has zero knowledge about. Examples tend not to have this requirement. Schemas reference each other all the time. How would an embedded XSD schema in a string create an xsd:import that references another XSD that is embedded into the same OpenAPI document? My brain hurts thinking about the implications of using JSON References as fragment identifiers inside an XSD.

I have no fundamental objection to embedding alternate schemas but I don't think we should delay releasing external alternate schemas while we figure out these details.

[tedepstein]

I have no fundamental objection to embedding alternate schemas but I don't think we should delay releasing external alternate schemas while we figure out these details.

I agree, but do we even need to work out those details now?

Maybe the most we would need to figure out is a small set of issues about the OpenAPI media type (#110). Questions that come to mind:

• Is there a media type for OpenAPI documents?
• What fragment syntax(es) are supported by the OpenAPI document? (Presumably JSON Pointer would be one of these, maybe the only one.)

Given the answers to those questions:

• If a given schema type has a referencing scheme that can't work with OpenAPI's URLs and media type, then that schema type simply won't be able to refer to other schemas embedded in the OpenAPI document.
• If that's a big enough problem, because that schema type is important, and the folks using it really want to embed their schemas in OpenAPI documents, it can be addressed by updating the alternative schema language itself, and/or by doing something fancier with the OpenAPI media type in a later release.

So I'm saying that whole set of problems can be deferred until there's a real demand to address it. Meanwhile, other alternative schema formats that are more amenable to OpenAPI's fragment syntax can benefit from inline schema support.

[philsturgeon]

So I'm saying that whole set of problems can be deferred until there's a real demand to address it.

Fully support kicking this can down the road. Hopefully it'll be kicked into infinity.

[tedepstein]

Could we please try to separate the technical challenges from the aesthetics of this idea?

@darrelmiller, on the technical concerns:

While inlining foreign schemas into seems like low hanging fruit, I don't think it necessarily is. It introduces inconsistencies in handling different types of schemas. It introduces escaping issues. It raises the questions of how future versions of JSON schema should be rendered in YAMLdocuments. We already have this issue with examples and the answers are not clear.

Questions:

• Would you be able to point to some open issues and/or other resources so I can better understand where we've hit problems with inline examples?
• Are the encoding/escaping problems not avoided by using literal-style multi-line strings in YAML? (i.e. value beginning with a | pipe character.)
• Are the encoding/escaping problems likely to happen with text-based schema lanuages, or are they mainly a concern with non-text media types, like we'd expect to find in example values?
• What are the questions about future versions of JSON Schema in YAML documents?

I can't say whether I think these are good reasons to defer support for inline alternative schemas, because I just don't have the background on those concerns yet.

But what I'm hearing is a kind of distaste for the idea. That we don't want to do it because it's messy, somehow force-fitting one language into a string value so we can shoehorn it into another language.

So, here are a some examples of embedded languages in OpenAPI today:

Markdown Embedded in OpenAPI
HTML Embedded in Markdown
CSS Embedded in HTML

paths:
  /products:
    get:
      summary: Product Types
      description: |
        The Products endpoint returns information about the _Beamup_
        products offered at a given location. The response includes:
        * the display name 
        * the SKU
        * current pricing
        * product image

        This diagram shows the full data structure:
        <img src="https://api.beamup.io/diagram1.png" style="width: 78px;" />

CSV Embedded in OpenAPI

      responses:  
        200:
          description: A price table in CSV format.
          content:
            text/csv:
              example: | 
                Product,PriceTier,MinimumQuantity,Price
                P1234,Retail,1,43.95
                P1234,Wholesale,100,33.24
                P1234,Wholesale,1000,29.48
                P9877,Retail,1,8.34
                P9877,Wholesale,100,7.02
                P9877,Wholesale,500,6.81
                P9877,Wholesale,1000,6.6

Regular Expressions Embedded in a Schema

    Contact:
      type: object
      properties:
        firstName:
          type: string
        lastName:
          type: string
        homePhone: 
          type: string
          pattern: |-
            ((\(\d{3}\) ?)|(\d{3}-))?\d{3}-\d{4}
        customerID:
          type: string
          pattern: |-
            ^\d{3}-\d{2}-\d{4}$

... and here's what it _might_ look like to embed protobuf schemas:

Protobuf Embedded in OpenAPI

components:
  schemas:          
    Person:
      type: object
      x-oas-draft-alternativeSchemas:
        - type: protobuf
          value: |           
            syntax = "proto2";
            package contacts;

            message Person {
              required string name = 1;
              required int32 id = 2;
              optional string email = 3;

              enum PhoneType {
                MOBILE = 0;
                HOME = 1;
                WORK = 2;
              }

              message PhoneNumber {
                required string number = 1;
                optional PhoneType type = 2 [default = HOME];
              }

              repeated PhoneNumber phones = 4;
            }

This doesn't feel so onerous to me. We've all gotten so used to the current affordances for language embedding (Markdown, HTML, example data, regex, etc.) that we don't even think of OpenAPI as a polyglot language. But it is, and it works just fine.

And BTW, "just fine" doesn't mean there aren't rough edges here and there. Try explaining to a new OpenAPI user why her line breaks aren't being preserved, when they _might_ be removed by a folded YAML string, or it _might_ be the Commonmark processor downstream. It ain't perfect. But it's still really good. I don't hear anyone saying we should externalize all the description property values because Markdown is a different language.

Besides, whatever our gut reaction to these things, current and proposed, what I really want to say is that we shouldn't allow our decisions to be so heavily influenced by gut reactions.

We have really smart people here with strong opinions, and I don't want to dismiss anyone. If the thought of embedding foreign schema languages makes you want to barf, that's a fine starting point for a discussion. But it shouldn't end there, because these decisions have important implications in what users and tool providers can and can't do.

Generally, OpenAPI property values that can be externalized can also be embedded inline. That's true for documentation (externalDocs), schemas, examples, and probably others. License Object is one exception; it can only take a URL, not the text content of the license.

Disallowing inline alternative schemas for substantive technical reasons might be necessary. But disallowing them because some people think it's yucky doesn't seem like the right thing to do.

[adjenks]

Just allow inlined schemas as string types, then you can parse it as whatever you want as long as you know what format it's in.

x-oas-draft-alternate-schema:
    type: xml-schema
    schema: "<hello-world/>"

[philsturgeon]

"Just"? Strings are more complex than you give credit.

Please nobody has actually requested this feature and we're all just typing over and over about supporting it whilst saying we dont need it. Most of us are developers and as such we like to try and solve problems, but we are spending time on a problem nobody has requested. If we had a project or a product manager in here they'd both be telling us to get on with it.

External is fine.

[tedepstein]

@philsturgeon, we have customers who want this for JSON Schema, possibly other languages. I also have plans to support an alternative schema language. In both cases, we need inline/embedded alternative schemas in order to provide a reasonably good UX for editing.

It sounds like @fmvilas and @adjenks are both saying they would want and expect this.

The error messages in your screenshots would be addressed by using literal multi-line strings, i.e. adding a pipe character after the schema: key, and putting the schema content on separate lines, indented:

    x-oas-draft-alternate-schema:
      type: xml-schema
      schema: |
        <hello-world some-attr="Hi" />
        <hello-world some-attr="isn't this great" />

Folded strings (> instead of |) work for this purpose too, and most OpenAPI specs that have substantial markdown documentation are already using folded or literal multi-line strings for description properties. So it's not like we'd be requiring users to do something exotic.

[handrews]

@tedepstein do you not support multi-file specs at all?

[tedepstein]

@handrews , I'm fully in favor of OpenAPI _allowing_ multi-file specs, and our tools provide robust multi-file support. It's great to have that option.

But I am not in favor of _forcing_ multi-file specs where they're not needed.

If an API designer wants to use full JSON Schema or some other schema language, barring any specific technical reason that prevents it, it should be their choice to keep the schemas inline, or in external files.

BTW, I'm also surprised that we're spending so much time on this, because I thought this would be a no-brainer. We allow inline values for everything else (except license), so of course, we'll allow them here. If they don't work for you, or for your choice of schema language, feel free to use the external___ property or $ref to keep them external.

I think we're spending a lot of time on this because so much of the discussion has been around the value judgments of whether a particular usage is good or bad, pretty or ugly, as opposed to whether there's actually any harm in allowing it, and allowing users and tool providers to make their own judgments and do what they want.

I just don't understand the visceral reaction that we're opening some kind of Pandora's box with inline schemas. Like it's so dangerous that we have to break from our usual pattern of allowing inline values, raise the spectre of runaway complexity, etc., when all we're proposing is to follow the pattern we've established with description, examples, and all of the $ref-enabled properties: allow them inline, or reference them externally.

As I see it, this isn't about "adding a feature." It's about removing an intentional restriction on the feature. And I'm not seeing a rational basis for most of this controversy.

[philsturgeon]

Just a quick reminder to folks in the thread that JSON Schema is not currently supported inline, OpenAPI Schema Objects (which are loosely based on JSON Schema) are supported inline. If you want to use JSON Schema you _have to_ link to an external file through alternatioveSchema.

We are not proposing we allow JSON Schema proper by shoving it into a YAML multi-line string, so I do not know why we would support other alternative schemas by shoving them into a YAML multi-line string.

Extra reminder: JSON does not support fancy > syntax for multi-line strings, so the examples I posted above would continue to fail for JSON users.

[tedepstein]

@philsturgeon ,

We are not proposing we allow JSON Schema proper by shoving it into a YAML multi-line string, so I do not know why we would support other alternative schemas by shoving them into a YAML multi-line string.

Objection to the word "shoving." We're not proposing to shove anything, any more than we're currently shoving multi-line markdown into description properties. See my comment here.

Extra reminder: JSON does not support fancy > syntax for multi-line strings, so the examples I posted above would continue to fail for JSON users.

Misleading. JSON can represent multi-line strings with character escaping, just as it currently does with embedded markdown, regular expressions, request and response examples. Embedded schemas would introduce absolutely no new JSON compatibility issues.

[handrews]

@tedepstein the thing I'm focused on here is that part of this proposal has a solid consensus behind it, while the inline part clearly does not (never mind why or whether it is reasonable, there is clearly no consensus and it will take effort to work through this).

I am in favor of decisively including the part that we have consensus on and continuing to discuss inlining separately. If inlining makes it into 3.1, great!

But please let's move ahead with the part we agree on. It does not make inlining harder to add if we decide to do it in 3.1.

I have a number of concerns about inlining, but I would prefer to not continue to discuss the topic here. I will chime in if we split it off.

[tedepstein]

@handrews , I would like to give others a little time to respond.

IMO, this debate has gotten dysfunctional. For my part, I have tried to keep it clear and focused, but there is a lot of smoke here. I am not at all happy to have this issue decided, or deferred, because the discussion degenerated into pressure tactics, memes, and misinformation. For the record, this sucks.

I will feel _so_ much better letting this go, or taking it into a new issue, if your concerns about inlining, or @darrelmiller's, or anyone else's, turn out to be significant issues that we had not considered. So whether we end up discussing it here or elsewhere, if you get a few minutes to summarize those concerns, I'd like to hear them.

[handrews]

@tedepstein I think I'll also wait on others before going further into the substance of this issue.

Regarding the process, I am sympathetic to your frustration. Part of my suggestion for splitting this out into a separate topic is that, having dealt with more than a few thread of this nature in the JSON Schema project, splitting has been one of the most effective ways to de-escalate arguments and make progress where we were able to do so.

In this particular case, I am very concerned that alternativeSchema will miss the 3.1 release because of a late-added feature request that is not at all essential to alternativeSchema's usefulness. It may be essential for your tooling, but there are many people who are waiting on this feature who do not need inline schemas.

If alternativeSchema gets punted out of 3.1 over the question of inline schemas, I'm going to be more than a little upset over that 😢

So I don't really want to get into the merits. I want to ensure that the core of the proposal moves forward, because it's extremely important to me that alternativeSchema be in 3.1. Obviously as someone driving the JSON Schema project, the single most important thing for me in OpenAPI is to be able to use actual JSON Schemas.

[aowss]

I completely agree with @handrews' comment : inlining can be considered as a separate request.
Being able to use other schema languages is essential for some of us.

I am not a big fan of the syntax but that's a separate issue and I seem to be the only one with this opinion so feel free to ignore that.

[adjenks]

Not a problem with me, fork the discussion. I'm not one to call the shots on what your deadlines are and what has the greatest priority. I'm just here to tell you that in the context of my project code it makes no sense to externalize the schemas, so I would like to be able to inline it eventually. In the meantime I can make workarounds and things. Feel free to prioritize however you deem fit, and fork the discussion. I don't want to stall things, just want to inform you of my needs and I thought this was the place to do that. I really like this project, so thank you for your hard work.

[aowss]

Thanks for coming back to me.

I am very aware of anyOf in JSON Schema.

1) My concern was the addition of schema-like keywords in a property that is supposed to just import / include schemas. I feel that we are mixing concerns by adding this ability in this property. It’s probably the job of the imported schema to do that.
=> IMHO this property should only be used to include an external schema. Nothing more, nothing less

2) The concept of anyOf is very JSON Schema-specific. What does it mean to do that when the schemas are not JSON Schema ?

3) Did anyone express the need to include different schemas that are using the different schema languages ?
=> I am afraid that, as is the case with inlining non-JSON schemas in this property, we are trying to solve something bigger than the initial ask, i.e. include an external schema. In this specific case, I am not even sure that there is a real use case for that ( I might be wrong about this ).

In conclusion, I think that we should focus this feature on the ability to include an external schema and deal with other asks as separate features.
The inclusion of external schemas is a priority for a lot of people. A lot of us want their schema system to be separate from their API description. This allows us to use the latest version of JSON Schema for example even though OpenAPI supports a different version.

Thanks again for coming back to me on this,
Aowss

On Apr 3, 2019, at 12:32 PM, Henry Andrews [e-mail] wrote:

@aowss https://github.com/aowss I can perhaps address some of your syntax concerns:

This seems to complicate the syntax with the introduction of schema constructs, e.g. anyOf, within the schema element.
This assumes that these schema languages can be combined using these constructs. This might seem true at first glance but I am not sure it is as obvious as it seems, e.g. the use of default in these different languages.
The semantics of JSON Schema's anyOf (and many other keywords) have been clarified in recent drafts. The keyword takes an array of subschemas, evaluates them according to their own rules, and logically ORs the validation results.

In the context of JSON Schema proper, the subschemas are expected to also be JSON Schemas, but really the evaluation model lets them be opaque. So the way anyOf is being used here is entirely consistent with JSON Schema.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/OAI/OpenAPI-Specification/issues/1532#issuecomment-479564033, or mute the thread https://github.com/notifications/unsubscribe-auth/AGAZHpESBKXTyeBTojFPvu4MkuByE0BQks5vdNeAgaJpZM4TL4FX.

[cmheazel]

@aowss I work with systems that return XML. The response undergoes syntax validation through an XML Schema document and valid value validation through a set of Schematron rules. Due to the complexity of Schematron, these rules are packaged in multiple Schematron files. So the answer to question #3 is yes.

[Relequestual]

Is this resolved by https://github.com/OAI/OpenAPI-Specification/pull/1270 ?

[MikeRalphson]

@Relequestual no, #1270 was for a schema instance for partially validating OAS 3.0.x documents. This issue concerns the tentative process (using x-oas-draft- extensions so it would not have to wait for OAS 3.1 for implementations) of support for non-OAS 3.0 schema languages in requests and responses.

[Relequestual]

Ach, sorry. I should learn to read properly!

[BlueCoder77]

I'm confused as to why this was not implemented. I really need the ability to reference an XML Schema as the schema for my request and response. Nothing fancy, just an external reference to a single XML Schema document.

Can we please get this implemented? Pretty please?

[philsturgeon]

@BlueCoder77 sorry. It's not really a matter of how nice you ask, it's that the proposal in its current shape was no good (#1943). We mostly decided that 3.1 wasn't the right time for alternative schemas, and maybe v4.0 would be instead. 3.1 is getting proper JSON Schema support (#1977), which is what the majority of folks are doing¡ with OpenAPI anyhow.

XML Schema support is important, but it's enough hard work trying to get this smaller thing done. 😅

Maybe you could check out #1943 and help get the feedback implemented? Might be doable for 3.2 with a bit of effort. Who knows.

[BlueCoder77]

@philsturgeon - Thanks for the help.

[darrelmiller]

@BlueCoder77 @philsturgeon

Allow me to add a slightly different perspective on this story. Phil isn't wrong, he just has a perspective :-)

The Alternative Schema proposal is in a state that we considered incorporating into the specification. However, when doing a "pilot" proposal we require multiple actual implementations of the feature before we incorporate into the specification. This is a validation process to ensure we don't add big features to the spec that are problematic for tooling.

Phil and the Stoplight crew attempted to implement the current proposal (Yay!) and had a bunch of feedback that they felt the approach was over-complicated and suggested ways it could be simplified. This is how the process is supposed to work.

In theory, multiple other implementers could come along and implement the proposal as is and say THIS IS AWESOME, SHIP IT. And in theory we could allow that feedback to overrule the Stoplight opinion. Unlikely, but theoretically possible.

There wasn't really a decision to say 3.1 is not the right time for alternative schema. We are in a place now, were we either need to re-work the proposal to make Phil et al. happy, or wait for feedback from other implementers who have actually tried to implement it. We missed the 3.1 shipping window for this proposal.

I don't think we need to wait until 4.0 for this. We do need more participation from people who actually build tooling.

[BlueCoder77]

@darrelmiller - Okay, so what do y'all consider an implementer? I really need this feature available for an OpenAPI I am writing for a client. Can I implement this and give me feedback?

[darrelmiller]

@BlueCoder77 An "implementer" is not someone who is creating OpenAPI descriptions. An "implementer" is someone who is writing tools to process an OpenAPI description. Processing could be for validation, editing, client code generation, mock servers, testing, or documentation generation.

If you are writing tools for your client to process OpenAPI descriptions that have included alternative schemas then your feedback would be most appreciated.

[BlueCoder77]

@darrelmiller - Okay, I understand. No, I am not an implementer. Are y'all aware of any current editor implementers that support importing an XML schema to a component model?

[philsturgeon]

I am not.

[matjung]

Regarding alternative schemas.
I was looking for a way to model CSV within OpenAPI and did not find out how to do it yet.
Is it possible to make references to another organizations schemata/specifications?
My thinking is to use DFDL (https://github.com/DFDLSchemas/CSV)
With DFDL one can use XML as model for CSV.
Or is there already an Open API method available for handling CSV in the response.

[Fak3]

I would also love to see a way to link openapi schema with a csvw or frictionless data csv schemas

[dret]

can somebody please summarize where this proposal stands in terms of OpenAPI 3.1? https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.1.0.md#schemaObject looks as if JSON schema is still hard-coded into OpenAPI? specifically, i am looking for a way how https://tools.ietf.org/html/rfc8927 could be used with OpenAPI. looking at this issue, it seems that this issue is the right place to look at, but looking at the current 3.1 RC it seems that this proposal didn't make it into 3.1?

[philsturgeon]

That’s right. The tldr was that instead of getting multiple schemas supported we focused on supporting the one same single type of schema but removing all the problematic discrepancies.

I think the thinking was 3.1: JSON Schema Properly, 4.0: multiple schemas.

[dret]

On 2020-12-21 11:54, Phil Sturgeon wrote:

That’s right. The tldr was that instead of getting multiple schemas
supported we focused on supporting the one same single type of schema
but removing all the problematic discrepancies.

I think the thinking was 3.1: JSON Schema Properly, 4.0: multiple schemas.

thanks for the clarification. looking forward to OpenAPI becoming more
of its "own thing" without hardcoding the choice of a schema language.

[philsturgeon]

OpenAPI is absolutely it's own thing, its an API Description Format that covers the Service Model: Headers, Paths, Parameters, everything that is API specific. v3.1 just means it now uses an existing standard for describing the "Data Model", which is a huge improvement for 3.1 as it means tooling can converge instead of people copying and pasting tools to add exceptions for nullable and other discrepancies.

I remember you being pretty excited about the discrepancy issue being resolved and wanted it to happen sooner. Well, we got it done for 3.1, and this all takes a lot more time than anyone would think! 😅

I would like to figure out how we can make OpenAPI support XML Schema, Protobuf, and GraphQL Types, as they're things we want at Stoplight anyway. If anyone would be interested in making a working group to go make a plan for this, lmk, and I can set up some calls and shuffle it along. In the past it was just lots of people saying "why isn't this done yet" then @darrelmiller trying to find time to satisfy those demands, which probably isn't how we should do things going forward.

[dret]

On 2020-12-23 06:41, Phil Sturgeon wrote:

OpenAPI is absolutely it's own thing, its an API Description Format that
covers the Service Model: Headers, Paths, Parameters, everything that is
API specific. v3.1 just means it now uses an existing standard for
describing the "Data Model", which is a huge improvement for 3.1 as it
means tooling can converge instead of people copying and pasting tools
to add exceptions for nullable and other discrepancies.

sure, no doubt about that. my comment was just about OpenAPI currently
subsuming JSON schema, and it doesn't have to be this way.

I remember you being pretty excited about the discrepancy issue being
resolved and wanted it to happen sooner. Well, we got it done for 3.1,
and this all takes a lot more time than anyone would think! 😅

i still think that it's great to see JSON schema versioning having one
less complication!

I would like to figure out how we can make OpenAPI support XML Schema,
Protobuf, and GraphQL Types, as they're things we want at Stoplight
anyway. If anyone would be interested in making a working group to go
make a plan for this, lmk, and I can set up some calls and shuffle it
along. In the past it was just lots of people saying "why isn't this
done yet" then @darrelmiller https://github.com/darrelmiller trying to
find time to satisfy those demands, which probably isn't how we should
do things going forward.

i apart from some comments here and there i haven't really participated
much in OpenAPI work. this seems like a worthy (and potentially tricky)
thing to do, and i'd be more than happy to help.