Openapi-specification: Extra JSON Reference properties ignored - reduces reusability of references

[EvilJimJafar]

Hi everyone.

I feel that the ignoring of extra properties when using JSON References goes against the goal of reusability.

Take this example from a fictional Swagger contract:

{
    addresses: [
        "homeAddress": {
            "description": "The premises at which electricity is provided.",
            "$ref": "#/definitions/Address"
        },
        "invoiceAddress": {
            "description": "The billing address - must be where the customer's credit card is registered.",
            "$ref": "#/definitions/Address"
        }
    ]
}

The "description" properties help to distinguish between the two uses of the "Address" Reference. Of course you could argue that better naming of the "homeAddress" and "invoiceAddress" properties could achieve this, but we are not always in control of property names in our data models and sometimes you need a more verbose description.

The Swagger editor (http://editor.swagger.io/#/) is an example of an application that could be improved by removing this limitation.

Is this something that has been raised previously? Please see here for a discussion on the Swagger Google Group: https://groups.google.com/forum/#!topic/swagger-swaggersocket/ewgimdO2cOI

[webron]

Currently, we use JSON References as-is. While there's a huge advantage to that tooling-wise, it does hinder reusability in some cases, and the issue has been raised (unofficially) a few times. Overriding the description is one case, but there are also cases of wanting to override whether fields are required and so on. Not sure if we should change it, given the cost, but we should consider it.

[qcho]

I think this is a very good Proposal.
In my example I want to set:

      parameters:
        - $ref: '#/parameters/fields'
          default: 'local_default'

[whitlockjc]

I would be willing to update json-refs to have a flag that allowed you to handle this situation. Based on OpenAPI feature requests, I can see a few different things we'd need:

• Merging keys, like the example above
• Overriding/Replacing keys, possibly like the example below

The JSON Reference spec clearly says these should be ignored. But...there's nothing stopping json-refs from allowing extra functionality on top in an opt-in manner.

[whitlockjc]

Realize, this has nothing specific to do with OpenAPI other than a number of the Node.js tools around OpenAPI use json-refs. A decision on either side does not have to reflect or impact the ability/decision of the other. I hope that is clear.

[fehguy]

I would love this feature, but I strongly advise we don't do it. If we did, tooling support would be inconsistent because the spec says it's a no-no.

There is a construct in JSON schema draft 5 but I don't think we're considering updating to that (@webron?)

[whitlockjc]

Good point. I was just pointing out that if the decision was to do this, I have no objection for my tooling. As for JSON Schema Draft 5, unless it alters the JSON References specification I don't see us being able to support this if we do things the JSON References way. I wonder if we should reach out to the JSON References authors?

[drewish]

I'd also love to be able to override items from a $ref. I'd actually been using them before but now that the swagger editor is flagging them as warnings I'd been reconsidering it.

[whitlockjc]

The warnings are accurate so don't attempt to merge/override properties in a referenced object as it won't work. I would love to see this addressed as well.

[webron]

Parent meta issue: #579

[webron]

I haven't checked draft 5, so I don't know if it has effects on JSON References as well (possibly with a new draft to that). I don't think we'll be moving to draft 5 as it's not official (yeah, an official draft).

Breaking away from specs is a challenge. Unlike some expansions to JSON Schema which are actually allowed by JSON Schema itself, we're suggesting here something that goes against the spec of JSON References.

That said, within the spec, the inability to add/override JSON References makes them unusable in some cases and forces people to not DRY or make incomplete documentation.

The real challenge here is not to say we allow it, but try to think of the edge cases that could help users abuse this feature in a way, and causing more problems for the tooling. I'd like to see this change happening, but we need to be careful here.

[fehguy]

There is a merge construct which addresses this issue. But I think moving to draft 5 would be a big mistake right now.

[jmatcuk]

I have a team trying to use the swagger editor hosted on swaggers site to edit our API's but with the recent enhancement to the editor to use this update, our descriptions were showing as warnings. We have a very similar issue as @JimSangwine which I can post our actual example but for now we've just cloned an older tag of the swagger editor and are running it locally.

So the reason that I posted is to support this change but to ask for clarification, is @fehguy suggesting that we do not support this because the draft version of the JSON Schema does not support it and the swagger editor was upgraded to use the new draft version?

If i'm following correctly, it's a draft and a reason why this argument should be made to change the specification.

[IvanGoncharov]

Actually it supported both in Swagger and Draft-4, you just need to use allOf keyword.
So instead of:

description: 'The premises at which electricity is provided.'
$ref": '#/definitions/Address'

You should write:

description: 'The premises at which electricity is provided.'
allOf:
  - $ref: '#/definitions/Address'

It standard usage of allOf nothing fancy here, and I think it fully solve this problem.

[tadhgpearson]

Thanks @JimSangwine - I also face this issue, indeed I opened this as a question on the JSON Schema Group
https://groups.google.com/forum/#!topic/json-schema/YS1dbWXk_4s

@IvanGoncharov - I really like this suggestion. It seems syntactically valid, but functionally unclear.
For example - if both elements happen to have a description, which one wins?
Furthermore, fields which include this type of reference are dropped from the output of the code generator, so for me, it's not an acceptable 'interim fix' :(

[ralfhandl]

Found out accidentally today that Swagger UI supports and visualizes description within $ref object as in the initial example, whereas Swagger Editor issues a warning

[webron]

Tackling PR: #741

[ghost]

@IvanGoncharov Regarding the suggestion to use allOf for the purpose, I am not sure if this is the intended use of allOf as has been suggested. allOf intended use does not seem to suggest that it can be used for overriding property values or providing additional "description" property.

[tadhgpearson]

Maybe this can be best resolved by the proposed new $use extension of JSON Schema - see https://github.com/json-schema-org/json-schema-spec/issues/98

[advance512]

allOf is pretty much unsupported in most tools.
e.g.
https://github.com/swagger-api/swagger-editor/issues/476

[nickreevesuk]

The overarching goal of OpenAPI is to meet the needs around meta data definition of APIs. If a technical choice (e.g. to model references as JSON References) conflicts with those needs it is not the original need that it is at fault but the technical choice, because it is not fit for purpose.

There are many genuine uses for wanting to add meta data at the point of reference. If JSON References cannot meet that need, then Open API should select a different syntactic construct as an alternative to meet that need. This might be the new $use extension of JSON schema as above, or an alternative specific to Open API. For backwards compatibility this would probably be an alternative option rather than a replacement.

Failure to address reasonable community needs will simply cause the community to move elsewhere to projects that can meet their needs.

[darrelmiller]

@nickreevesuk If you take the time to read the history of this issue, you will see that we all would like this feature to exist. $ref works the way it does in OpenAPI because that's the way it is defined in JSON Schema. Despite this shortcoming of the $ref feature, community consensus is that people want OpenAPI to use JSON Schema more consistently with its spec, not less.

There have been efforts to address this issue in JSON-Schema in the past, but those efforts stalled. Now that the folks working on that spec have got some momentum again, it is possible that we may see this issue resolved in the near future.

As I noticed you are new here on GitHub, perhaps you are not aware this is a volunteer run project and the best way to make things happen is to contribute. Those who are involved tend not to react positively to threats suggesting that if you don't implement this feature, all your users are going to leave.

Now, I am going to go back to trying to analyze the differences between the draft-4 JSON Schema spec and the latest draft so that we can adopt the new JSON-Schema spec in OpenAPI V3 without introducing breaking changes for our users.

[nickreevesuk]

@darrelmiller no threat intended, that's just simply the way mindshare of projects works. I was just trying to move the discussion back to the requirements.

[emil-prager]

@darrelmiller Thank you for your patience and for the intention to (re)check whether the last draft of the JSON spec might add more flexibility in this regard!
However, regarding your remark that:

the best way to make things happen is to contribute

I'd say that actually there is at least such an initiative from the community --- see this comment, for example.

[handrews]

@darrelmiller do note that Draft 06 is almost ready, with most of the remaining PRs as minor bugfix sorts of things: https://github.com/json-schema-org/json-schema-spec/milestone/2

I strongly recommend looking at this rather than Draft 05 as there are significant improvements.

[webron]

We're not going to include changes from draft 05 or 06 at this stage. The reason we're considering moving to 05 is that it doesn't change the functionality of the things that are already supported by the OAS. It looks like 06 adds functionality, and it's too late for us to evaluate whether we can support it or not. To add to that, since tooling takes time to catch up, I don't think it's wise for us to play the catch-up game in this case.

There is one major issue with draft 05 that has to be resolved before we can switch to it, and that's the drop of the integer type. If that's added back to draft 05, we're likely to move to it. If not, for now, we'll stick with draft 04 (even if expired).

[handrews]

@webron it was apparently an unintentional deletion on the draft editor's part and has been added back in Draft 06. You can't actually change a draft once published so the actual draft 05 document won't reflect this, but it's safe to treat draft 05 as if integer was still supported. There is actually one part in draft 05 that mentions 7 types as if integer was still present, so it wasn't even entirely removed.

The only other thing I'd note about going from draft 05 to draft 06 is that draft 05 added a "uriref" format which became "uri-ref" in draft 06 (because "date-time" and making all compound formats use hyphens).

Otherwise, draft 06 validation is compatible with draft 05 so you'll still be on a smooth path. The big changes (in both) were in hyper-schema which does not impact you.

[webron]

Yeah, I think it was @Relequestual who mentioned that it was an unintentional drop. However, it don't know if it's safe to treat it that way. If you can't change it, then I would expect you to release draft 06 with the fix and push the rest of the changes as draft 07. I realize this may not work with your plan.

To avoid confusion, it might be safer for us to stay with draft 04 for now (assuming you can't make the suggested change) and look back on updating the support in the next version of our spec.

[nickreevesuk]

Having looked at the root cause of this I can suggest one other possible approach in addition to those I mentioned above.

The underlying reason why JSON schema specification rejects siblings of $ref seems to be because of the concern about these potentially overriding validation criteria in different references to the same underlying definition in conflicting ways. It is safe for purely JSON schema validation purposes to ignore siblings of $ref which don't affect validation as others have already observed.

This opens the possibility of defining the OpenAPI as a superset of the JSON schema where all additional tags must begin with a whitelisted set of prefixes which get stripped out to reduce the contents to a standard JSON schema document. This allows the OpenAPI file to be used both as is by tools which understand the additions or will ignore them. For tools with a strict interpretation of JSON schema we require an additional tool in the toolchain which will strip out this well defined set of extensions to to produce a new set of JSON schema specification compliant input files.

Although this does require an extra tool in the chain it does break the log jam by decoupling the two specifications, leaving OpenAPI much more free to evolve as it needs. This might help with a much wider set of issues.

Following the principle that things should be closed for modification but open for extension it would have been more helpful if the JSON schema specification forbad only tags it understood as siblings of $ref, but that is beyond the scope of this group.

[handrews]

@nickreevesuk json-schema-org/json-schema-spec#98 , the "$use" proposal mentioned earlier which is under consideration for Draft 07 in large part because of support from folks from OAI. Options for implementing it would include either a whitelist (all annotation keywords are on the whitelist and can appear in the "with" clause) or blacklist approach (all validation keywords are on the blacklist and _cannot_ appear in the "with" clause). Either approach could be extensible,a nd both could be used.

It is easy enough for OAI to implement "$use" as an extension which could then fold into JSON Schema proper, just keep us posted on any implementation choices you make.

[timburks]

I ran into this issue recently and add the following as a practical note:

Scanning the descriptions in the apis.guru OpenAPI directory, I found two descriptions that put "description" fields alongside "$ref".

• APIs/authentiq.io/0.1.0/swagger.yaml
• APIs/rebilly.com/2.1/swagger.yaml

I expect that this will be a fairly common mistake.

We might guard against this by enforcing schema validation early. For example, public archives like apis.guru might only publish API models that pass validation.

But hand-written source files are what they are.

For better or worse, I was able to accept these descriptions in my schema-driven reader by generating my reader from a modified schema that adds a "description" property to the "jsonReference" definition.

    "jsonReference": {
      "type": "object",
      "required": [
        "$ref"
      ],
      "additionalProperties": false,
      "properties": {
        "$ref": {
          "type": "string"
        },
        "description": {
          "type": "string"
        }
      }
    }

[webron]

Pointing to specific lines in those would help, because I can't find a problem.
That said, there's nothing to catch in the validation. It's not wrong to have additional properties, they're just ignored. There's nothing invalid about that.

[MikeRalphson]

For example, public archives like apis.guru might only publish API models that pass validation.

We do. :) What @webron said.

[ePaul]

I think there is a real use case for having description near references – a description for a type is not suitable as description for all properties of this type. That is why people put a description there – even knowing than any tools will ignore it, it is useful for human readers.

[Relequestual]

We had a similar discussion over at JSON Schema as @ePaul explains. The description is ignored when with a $ref, but actually it's still useful for human readers of the raw documents.

[advance512]

Wouldn't it be nice having this valuable information as part of visual OpenAPI clients, like Swagger UI, and not just for a human incidentally reading the raw JSON schema?

[Relequestual]

As stated earlier, we are looking into adding a new key word to JSON Schema core which will allow this sort of thing.
JSON reference was kind of rolled into JSON Schema.

Regardless I think this is probably more of an upstream issue from Swagger / OAI, as they are not defining the behaviour.

[tadhgpearson]

We had a similar discussion over at JSON Schema as @ePaul explains. The description is ignored when with a $ref, but actually it's still useful for human readers of the raw documents.

Absolutely - see the initial discussion of this subject on the JSON Schema group
The proposal to adapt to this in a generic way is JSON Schema #98, but I would tend to agree that the description seems to be by-far the most common use case.

If apis.guru provides strong support to indicate this issue applies almost exclusively to the description, and it the solution could be much simpler if only support for the description were required, I'd encourage you to go for it!

[MikeRalphson]

If apis.guru provides strong support to indicate this issue applies almost exclusively to the description...

I'm not sure if you meant APIs.guru the collection or APIs.guru the maintainer (hi!), but anyway here is some analysis.

Of 5,182 instances of $ref having other properties:
4,356 are description only
283 are x-ms-client-flatten
138 are title only
131 are description and readOnly
107 are description and title
102 are description and x-ms-client-flatten
20 are properties and required
16 are example
7 are type
6 are description and type
6 are readOnly
5 are description and x-ms-client-name
3 are description and externalDocs
2 are required

The results may be skewed somewhat by the Google specs being particularly bad offenders of $ref + description.

If anyone's interested I could run the same analysis over our entire test corpus of nearly 50,000 definitions.

[handrews]

If anyone's interested I could run the same analysis over our entire test corpus of nearly 50,000 definitions.

@MikeRalphson I would love to see this information, and if you could update json-schema-org/json-schema-spec#98 with it that would be great. The main reason that proposal hasn't gone in is that a few people are refusing to support it on the grounds that they want to be able to use it with all validation keywords. My argument has been that annotation keywords (description, title, default, and examples from the validation spec, readOnly from the hyper-schema spec) are their own distinct use case that has substantially broader support for overrides than validation. Your data so far would seem to support that.

I'm curious about the type usage, though- any patterns there?

[Relequestual]

For refernece, myself and @handrews are maintainers of JSON Schema.

[MikeRalphson]

json-schema-org/json-schema-spec#98 updated.

I'm curious about the type usage, though- any patterns there?

They seem mainly to be type: object where the type property hasn't been defined in the referenced definition. I'm guessing they may be remnants of cut-and-pasting where a definition has been extracted for re-use? The larger number across the 50K definitions might not bear that out though.

[slhenty]

Having read all these comments, and being faced with a real-world desire to share definitions but make the description meaningful at the point of reference, I find myself aligning with @nickreevesuk's observation that treating OpenAPI as a superset of JSONSchema solves the immediate needs of the OpenAPI community without limiting existing tooling, nor future adherence to JSONSchema should it finally acquire a recognized method of mixing in / overriding shared references.

I appreciate the respective OpenAPI / JSONSchema teams wanting to 'get this right', but providing a stepping stone would help the client community now. At the end of the day, I just want to use Swagger as a single source of truth for API definition and documentation - as it stands, the current $ref is failing that charge. The community (myself included) appears to gravitate naturally to adding a description field to the referencing object, whether as override or mixin... I say treat that as the signal it is.

[fehguy]

I just think it's a tough one. If we go against JSON schema the expect that _all_ JSON schema tooling will not work, and therefore we'd be better off building our own syntax, since the benefit of tooling reuse is zero.

I personally think the $ref stand-alone constraint in JSON schema is garbage, at least for our purposes. Makes for ugly definitions and causes too many tears.

[handrews]

Here are all of the proposed solutions to date, including just allowing "definitions" and "title" alongside "$ref": https://github.com/json-schema-org/json-schema-spec/labels/additionalProperties (not all of these are strictly about additionalProperties, that's just the biggest complaint so I used it for the label)

Most people involve recognize the problem, the deadlock is on choosing the solution. I am pushing hard to resolve this in the next draft, as I view it as the "elephant in the room" of JSON Schema and one of the biggest barriers to further adoption.

[handrews]

Anyway, please feel free to weigh in on the various proposals. Community support matters for these things.

[Relequestual]

@handrews I would support putting this issue as 1st priority for draft-7. It's a clear issue, and something people want. I confess I have a LOT of catching up and other JSON Schema related work to do though... and I'll be at less than 100% strength for a while! ;)

But seirously, if this isn't on the draft-7 milestone, it should be. I'll avoid further thoughts as this is OAI issue and not in JScheam repo.

[pierslawson]

I'm afraid to say I'm new to OAI having found my way here due to a difficulty I'm having with swagger. However, could I also suggest that you add "xml" to the list of items possibly supported alongside a $ref. The current approach means that XML overrides can only be applied to basic properties. As far as I can see, if a property is defined using a $ref, it is impossible for me to override its XML name. e.g. the example below would rename id to Id but not content to Content in the XML world.

"Document": {
  "type": "object",
  "properties": {
    "id": {
      "type": "string",
      "xml": {
        "name": "Id"
      }
    },
    "content": {
      "$ref": "#/definitions/Content",
      "xml": {
        "name": "Content",
        "attribute": false
      }
    }
  }
 }

There is a similar issue with naming the elements within an array... if the "items" contains a $ref I can't change the name of the XML element that wraps each item in the array.

[tadhgpearson]

Anyway, please feel free to weigh in on the various proposals. Community support matters for these things.

This is a tough one. On one hand, I have only ever had a need to overwrite the _description_ column to date. Based on the feedback from @MikeRalphson, it looks like this is the most common use case. However, I can see why a schema creator may also want to overwrite _example_, _required_ or _readOnly_... and once schema creators become used to the idea of overwriting descriptions of imported $refs, they will probably want that functionality for all the elements.

The key issue for me is ease-of-use of the two solutions, which is putting the overwrite value beside the ref (the fictional example from the top) vs using a specific overwrite object such as with.

Ultimately, I believe we should consider

1. Ease of use for schema creators
2. Ease of use for tool creators

For schema creators, the fictional contract design seems easier because it's intuitive, so long as all element can be overwritten. For example, if _description_ can be overwritten but _example_ cannot, that would seem counter-intuitive. On the other hand, adding an extra with element doesn't seem terribly burdensome to for schema creators, and the fact that it's more explicit may make schemas easier to maintain in the long run.

For tool creators, with seems much easier. It's an explicit overwrite element in the parsing tree, with a defined functionality. Simply overwriting in-line, like our fictional contract, requires a set of rules and exceptions to be programmed into every schema reader.

Given the pros and cons of both options, I would be OK with either, but would lean towards using the with element, or some equivalent. Hope that helps.

[pierslawson]

@tadhgpearson do they have to be considered as overrides? A description alongside a $ref could be seen as an override of the referenced type's description or it could be viewed as a description of the property itself.

In @JimSangwine original example, I think the descriptions he provides are better considered as a description of the property rather than a replacement for the address type's description (i.e. in addition to the address's own description). For example, if an address has a description "5 lines of free form text with a UK postcode", that is still valid, alongside the additional "The premises at which electricity is provided" that is describing this particular usage of the type.

In my case of having a problem with xml, the information held within xml is definitely referring to the property, not the type of the property (i.e. not the $ref)... in which case it certainly isn't meant to be an override of any xml information within the referenced type.

Does this enhancing the property (rather than overriding the type) effect whether you think introducing with is best or not?

Another thought...

It seems to me that the "path of least surprise" would push more towards the description having a common location regardless of whether a property happens to have a type or a $ref. By which I mean, given this example:

{
    addresses: [
        "homeAddress": {
            "type": "string",
            "description": "The premises at which electricity is provided."
        }
    ]
}

If I wanted to change it from type based to $ref based, I would probably assume that I would only have to change one line:

{
    addresses: [
        "homeAddress": {
            "$ref": "#/definitions/Address",
            "description": "The premises at which electricity is provided."
        }
    ]
}

Finding I have to introduce a whole new node (with) and move values into it would be a _surprise_ to me at least ;-)

[tadhgpearson]

As someone who wants a clear, consistent behavior for my specification, I think that having a rule that appends 2 descriptions, like you described, very weird. In this case, a set of rules would be determining how to separately defined fields, potentially far from each other in the specification file, are merged. I don't think that's an obvious behavior, especially in the context of JSON Schema & Swagger, where each field tends to be explicitly defined.

Regarding the location of the fields, I agree with you @pierslawson - with nothing specified today, as we can see from @MikeRalphson's example, specification creators do put the description in the same node as the $ref. There are so many examples of it that I think we can consider it to be intuitive or _unsurprising_ behavior.

The question I would ask from here is, if the spec was changed to define a with sub-node that allows specification creators to overwrite properties from the $referred object, would that still feel intuitive? Or would they constantly be looking at the manual, trying to figure out why it's not working?
To me, I think the with specification is actually quite good: I don't find it burdensome to remember, it forces me to bundle the overwriting elements together, and gives me a keyword to find where overwrites occur. However, that's just my own little opinion!

[pierslawson]

@tadhgpearson I wouldn't necessarily expect the descriptions to be appended... I'd leave it down to the UI to decide how they should be displayed as it can access them both. It appears to me that swagger ui could, for example, just display the property level description where it currently displays the description of a primitive property and continue to show the referenced type's description within its model section.

I guess my thoughts come from not thinking of this as "overwriting" the referenced type's properties... rather it is documenting this use of the referenced type. If it was "overwriting" then either it is indicating this is a "sub class" of the referenced type (which doesn't feel right as we aren't intending to change its inherent makeup) or it is changing the global definition of the referenced type (which doesn't feel right either as each $ref to the referenced type would change that global definition, in turn leaving only one as the winner).

[pierslawson]

My worry about the proposal to limit the additional elements allowed alongside a $ref (as mentioned above by @handrews) is that won't help me with my xml issue and somebody is bound to want to add more items to the list... such as readonly etc. that are extensions to the underlying JSON Schema.

Does anyone know the reasoning behind the restriction being there in the first place?

[Relequestual]

I believe the restriction was initially to keep the JSON Schema specification simple.

It's actually pretty complex in terms of rule making to determine what behaviour should be expected when merging elements under various conditions. I won't go into details as that's been done a few times now already over on the JSON Schema repo issues. BUT, it is something we want to solve. (I believe we've added it to the draft-7 milestone) What would be most useful for now is if you could document exactly and as simply as possible, what your use case is.

[handrews]

Originally, JSON Reference was a separate specification intended for use independent from JSON Schema. This general problem is definitely a top priority for draft-07.

[pierslawson]

Funnily enough I went back to look at when I was last involved with the JSON Schema back in 2010... and my contributions then were around $ref!

I think at the time JSON references were just a concept rather than something somebody had tried to specify in their own specification. Version 04 dropped my wording to simply refer to the 2011 draft specification for JSON Reference. The even later draft specs for JSON Schema seem to have adopted a more stringent usage.

I'm not sure what thinking went into the change between versions 03 and 04... whether it was a well thought through adoption of the JSON Reference draft spec or just an assumption that the JSON Reference spec must capture the same concept that had been in version 03 of the JSON Schema spec. Either way... I think life might have been easier if we had adopted a type: schema alongside string, number etc. Then the reference could have been encapsulated in its own node.

Enough on that... I'll work on a use case.

[pierslawson]

@Relequestual - quick background: I have an existing API that can provide HTML, JSON and XML representations. I'm migrating it to ASP.Net Core and decided to take advantage of https://github.com/domaindrivendev/Swashbuckle.AspNetCore to create a resource that would expose a swagger.json document. It worked great except when I looked at the sample XML generated by swagger.io.

So here is the full contents of swagger.json for a pretty basic API that mixes up string based properties and object based properties:

{
  "swagger": "2.0",
  "info": {
    "title": "People Service",
    "version": "v1"
  },
  "basePath": "/",
  "paths": {
    "/parents/{id}": {
      "get": {
        "tags": [ "Parents" ],
        "summary": "Retrieves a specific parents.",
        "consumes": [],
        "produces": [ "application/xml", "application/json" ],
        "parameters": [
          {
            "name": "id",
            "in": "path",
            "required": true,
            "type": "string"
          }
        ],
        "responses": {
          "200": {
            "description": "Parent retrieved.",
            "schema": { "$ref": "#/definitions/parent" }
          }
        }
      }
    }
  },
  "definitions": {
    "parent": {
      "type": "object",
      "properties": {
        "childStr": {
          "type": "string",
          "xml": {
            "name": "ChildStr"
          }
        },
        "childObj": {
          "$ref": "#/definitions/childObject",
          "xml": {
            "name": "ChildObj"
          }
        },
        "childStrings": {
          "type": "array",
          "items": {
            "type": "string",
            "xml": { "name": "ChildString" }
          },
          "xml": {
            "name": "ChildStrings",
            "wrapped": true
          }
        },
        "childObjects": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/childObject",
            "xml": { "name": "ChildObject" }
          },
          "xml": {
            "name": "ChildObjects",
            "wrapped": true
          }
        }
      },
      "xml": {
        "name": "Parent"
      }
    },
    "childObject": {
      "type": "object",
      "properties": {
        "value": {
          "type": "string",
          "xml": {
            "attribute": true
          }
        }
      }
    }
  }
}

The sample XML produced by swagger.io is currently:

<?xml version="1.0" encoding="UTF-8"?>
<Parent>
    <ChildStr>string</ChildStr>
    <childObj value="string">
    </childObj>
    <ChildStrings>
        <ChildString>string</ChildString>
    </ChildStrings>
    <ChildObjects>
        <ChildObjects value="string">
        </ChildObjects>
    </ChildObjects>
</Parent>

As per the current spec, there are two warnings about two of the xml values being ignored as they are alongside a $ref.

I would like to see the following XML generated (i.e. those xml nodes not being ignored and simply treated the same way as the xml nodes alongside the string based values are treated):

<?xml version="1.0" encoding="UTF-8"?>
<Parent>
    <ChildStr>string</ChildStr>
    <ChildObj value="string">
    </ChildObj>
    <ChildStrings>
        <ChildString>string</ChildString>
    </ChildStrings>
    <ChildObjects>
        <ChildObject value="string">
        </ChildObject>
    </ChildObjects>
</Parent>

In case you can't spot the differences:

1. childObj is now ChildObj
2. ChildObjects contains a list of ChildObject not a list of ChildObjects

This is pretty similar use case to the original one highlighted by @JimSangwine

[handrews]

@pierslawson I can't speak to what happened between drafts 03 and 04. No one who was active with JSON Schema at the time is still involved.

For draft-05, the goal with core and validation was improved clarity, so it just pulled in exactly what the separate JSON Reference draft specified. And we did not significantly change that for draft-06. The only real change that resulted from pulling it into the JSON Schema spec (again) is that we specified that it can only be used where a schema is expected (there is also a proposal for draft-07 hyper-schema to allow it where a link description object is expected as well).

Loosening the $ref rules is one of several proposals on the table for draft-07.

[Relequestual]

@pierslawson Is the xml part something designated by swagger? not seen it before in a json schema.

[pierslawson]

@Relequestual it is part of the Open API Specification https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#xml-object which swagger is built against.

[nickreevesuk]

They key issue is that $ref represents a BINARY relationship with two ends the referee and the referent. There are many genuine needs to attach meta data describing the referee not the referent. So if Customer entity has a reference to an Address you need a descriptions such as "this address is the billing address" or "this address is the electricity supply address". These are not overriding the description of the referent (what an address is), but giving information about the specific use in the referee (what this address instance represents). This goes beyond description to many other pieces of meta data describing the use in the referee (whether that reference is mandatory, what version it was introduced at ...) which can legitimately vary between references to the same referent.

Thus any complaints around not being able to allow other sibling properties to $ref because they would conflict with ones on the referent are specious. You simply need to define the semantics of these properties as applying to the specific use in the referee, leaving the referent unchanged. If you want to add properties relating to the referent those would be done in the referent at its point of definition.

[Relequestual]

@nickreevesuk That's an easy to understand and compelling use case. We're ON IT.

You could get around this currentley by having two layers of references I believe.

[ePaul]

You could get around this currentley by having two layers of references I believe.

I don't think so ... each layer of $ref ignores all other properties.

What I've seen sometimes is misusing allOf:

Customer:
  type: object
  properties:
    billingAddress:
      description: "the address to which the bill is sent"
      allOf:
       - $ref: '#/definitions/Address'
    supplyAddress:
      description: The address where the electricity is supplied
      allOf:
       - $ref: '#/definitions/Address'

[Relequestual]

mmm. I'm not sure quite what I was thinking =/

[RicoSuter]

I'm working on the .NET Swagger toolchain NSwag and I have the same issue (also see https://github.com/Azure/autorest/issues/2652).

Currently the toolchain produces

    supplyAddress:
      description: The address where the electricity is supplied
      allOf:
       - $ref: '#/definitions/Address'

Why is this a misuse? The description of the parameter is not an override but a description of the property - the description of the Address describes the schema/class itself.

My options are:

• Use $ref with additional properties: Violates JSON Reference spec - invalid in swagger-editor, etc.
• Use allOf: Valid Swagger spec but some tools cannot handle this (correctly) (e.g. they treat it as inheritance instead of reference).
• Add an option in the toolchain to either produce $ref or allOf depending on the tool which is consuming the spec: I really want to avoid this because most users will not understand the option and for most spec consumers it is not that simple to find out how they handle this scenario (i.e. it is hard to decide what the correct option is).

I still think @IvanGoncharov proposal is the best solution. Its important for me to find consensus here so that we can invite all tools to behave the same way and we can link to this decision.

For example, we could define the following rules which everyone should follow:

• For consumers (e.g. code generators):
  
  
  
  • If a property or parameter object has no direct type and allOf contains only one reference, then treat it like a $ref (not as inheritance, etc.)
  
  
  • If a property or parameter object has a $ref and additional properties do not ignore them but use them as if it where defined with an allOf (backward-compatiblity)
  
  
• For producers (e.g. Swagger generators):
  
  
  
  • Only use $ref if the property or parameter object has no additional properties, otherwise use allOf instead of $ref with only one $ref and no type on the parameter or property object
  
  

I think it is important to define the rules for this scenario somewhere, so that every tool can be implemented in the same way. As it is currently, a Swagger spec is not really universally usable...

Has this been fixed with Open API 3?

[handrews]

@RSuter and all: The forthcoming draft-07 of JSON Schema (hopefully next month) will have a bit more guidance on keywords like title, description, default, examples, readOnly, and secret, which we are calling _annotation_ keywords. This isn't really a new mandatory implementation requirement, and is compatible with earlier drafts (for whatever keywords were present in the earlier drafts- only the first three of those were in draft-04, plus readOnly was in Hyper-Schema at the time).

We are defining how an implementation can and should collect multiple values of these keywords when they apply to the same part of the instance, and making it more clear that applications should interpret the results as needed.

For instance, a system could discard all descriptions except the top-most one (e.g. keeping the one outside of an allOf and ignoring the ones within it). Or a system could append them all into a multi-paragraph description. The JSON Schema spec does not determine this, so OpenAPI could choose a particular behavior on top of this basic value-collection behavior for consistency.

Then, if OpenAPI chooses something like "concatenate all descriptions in depth-first order", or even something more content-specific like "check for a section number and order them accordingly", OpenAPI document authors can write descriptions with those processing rules in mind.

The PR is https://github.com/json-schema-org/json-schema-spec/pull/424 if anyone would like to give feedback.

This is much simpler than defining schema-merging semantics, which continues to be a bitterly divisive topic. Hopefully it will be addressed in draft-08 (either added or definitively rejected, but either way not in a state of limbo anymore). Although I'm not sure who is going to run that draft on the core and validation side as I will be focusing on hyper-schema.

[darrelmiller]

I have a slightly different proposal to address this particular problem. And perhaps this should be in the JSON-Schema repo instead. Let me know @handrews

Considering the example shown by @RSuter above:

supplyAddress:
      description: The address where the electricity is supplied
      allOf:
       - $ref: '#/definitions/Address

and building on @nickreevesuk 's description of $ref as a relationship between what I'll call the source and target because I will keep confusing referee and referent :-)

Let's look at an example of a media type that already does transclusion much like we are trying to do. Consider a standard HTML page:

<html>
    <link rel="stylesheet" title="My bedazzling style" href="stylesheet.css" />
</html>

The link tag defines a relationship between the source HTML page and target CSS file to be transcluded.
The additional attributes on the link tag further qualify the relationship. They DO NOT change the target CSS. In fact the attributes on the link tag are not related to the structure of the target object. They are native properties of the link itself.

My understanding is that much of the pain we are experiencing is related to us trying to modify the target of the $ref using properties in the source. Do we really need to do that? Could we annotate the object that contains the $ref property with additional metadata to describe the relationship?

e.g.

supplyAddress:
      $ref: '#/definitions/Address
      title: The address where the electricity is supplied

I realize this is not currently valid JSON Schema, and it looks very similar to what has already been proposed and what many people already do, ignoring the fact that it is currently invalid. However, the important distinction is that the property I have added would need to be a defined property of a "reference object". It is not an attempt to merge with the target object. It is an attribute of the relationship.

This approach addresses a major concern of mine that I haven't seen much conversation around. If I am writing tooling that loads an OpenAPI description into memory, when I have 1400 operations that all $ref a shared parameter, do I maintain that reference in the object graph, or do I inline the parameter definition? Obviously just maintaining the reference is ideal. However, if I have to merge a property that is overridden in every operation, I may have little choice but inline the parameter objects. Either that or create some fancy decorator object that wraps the target object.

If the reference object itself can maintain the additional annotations, then there only needs to be a single instance of the target object.

Now, what attributes are appropriate for this kind of reference object I don't know, but we do have some precedent on the subject in RFC5988.

[handrews]

@darrelmiller let's keep the discussion here for just a bit, and if it really digs into $ref proposals I'll find the right JSON Schema issue for you to add to (or let you know if you should file a new one).

This approach addresses a major concern of mine that I haven't seen much conversation around. If I am writing tooling that loads an OpenAPI description into memory, when I have 1400 operations that all $ref a shared parameter, do I maintain that reference in the object graph, or do I inline the parameter definition? Obviously just maintaining the reference is ideal. However, if I have to merge a property that is overridden in every operation, I may have little choice but inline the parameter objects. Either that or create some fancy decorator object that wraps the target object.

Yeah, this about sums it up. There's some confusion here, too, as the old separate JSON Reference I-D allowed implementations to replace the reference with the target, that actually doesn't quite work in JSON Schema (due to base URI inhereitance and meta-schema working within the target file / structure rather than inheriting across the $ref).

Let me float a syntax here, _in addition to the current syntax_, and then explain it in the context of recent JSON Schema work:

$ref:
    href: "#/definitions/Address"
    title: "The address where the electricity is supplied."
    description: "Lots and lots of text here ..."

The idea here is that you would still be able to use $ref as you do now, with a string value that is a URI reference, for simple cases. But if you want to get full link functionality, then the value of $ref is an object letting you specify a lot more attributes. This particular syntax is a very, _very_ stripped-down subset of Hyper-Schema's LDO, which will likely get some more target attributes defined that could be used here (might as well argue them in one place...)

To be absolutely clear: none of the URI templating or input description keywords would be allowed in this context. The idea is not to use Hyper-Schema itself, just a compatible subset. While rel is required in Hyper-Schema, it seems unlikely to be useful here, so I would probably assign it the default value of "related" (which is an IANA-registered relation). I suppose anchor could be used, but I can't imagine why it would be off the top of my head.

What I like about this is that it's not possible to confuse this with a schema merge strategy, and it suggests the correct mental model of web linking.

[handrews]

If this gets any traction, we can file it as a proposal on JSON Schema. $ref changes are a source of contention, but this at least aligns $ref with an existing well-known concept.

[darrelmiller]

@handrews That approach works for me too. I was adding to the existing reference object just to reduce the nesting a bit, but your solution is definitely clearer as to what is going on.

[handrews]

@darrelmiller cool! I'm going to think on it for a couple of days, and if it holds up, I'll file it in the JSON Schema repo. You're of course welcome to open an issue yourself if you'd rather. I've kind of exhausted the capacity of everyone else to review things at the moment with the top-to-bottom hyper-schema rewrite :-P

Anyone: feel free to remind me if I don't do something about it within a week.

[Relequestual]

@handrews Did you make any suggestions on this within the JSON Schema repo? (I have nearly 2k github notification emails... forgive me for not knowing or checking ;) )

(I guess that means you should also probably notify me on slack or something if you reply!)

[handrews]

@Relequestual I plan to file it this week or next. Draft-07 is out as of yesterday! I'm tidying up the web site and putting together migration notes and will announce it more broadly. Then this is one of several draft-08 backlog items I need to file that I was avoiding doing b/c I did not want to start a big discussion while wrapping up draft-07.

[handrews]

@darrelmiller @Relequestual I've referenced this from a JSON Schema issue. The JSON Schema issue is rather abstract, focusing on how $ref should be processed. I'd like to see how that goes, as well as a separate proposal for deferred evaluation keywords that may solve a lot of re-use problems (warning- that issue is even more abstract, laying out a full JSON Schema processing model and proposing extensions).

With draft-07 we formally organized JSON Schema keywords by behavior into the (in some cases overlapping) classifications of applicability, assertions, and annotations. This has given us a better framework to talk about changes to JSON Schema behavior.

I'm hoping to get these processing model questions resolved early in draft-08 and then move into concrete discussions of what to do about specific ongoing re-use concerns. We'll see how that goes.

[handrews]

_[Apologies for some duplicate info in various issues as I update with JSON Schema developments- I'm never sure if people, particularly lurkers, are following all the issues or just selected ones]_

@darrelmiller it looks like we will probably go with something more like your proposal for adjacent keywords.

Specifically:

• $ref is being defined as delegation (its results are the results of the target schema)
• we have clarified how annotations like title, description, default, etc. are returned through schema results alongside of the validation outcome
• this clarification also provides more guidance on what additional semantics an application can define, such as "a title adjacent to $ref applies to the relation rather than the target."

Basically, we'll define the process so that it's easier to figure out how the information is arranged in the schema, and avoid defining the usage semantics too closely to give downstream systems the ability to define a wide range of reasonable semantics.

As for the link serialization approach, I think it's still reasonable to think of $ref as a hyperlink, but moving to a specific syntax related to hyper-schema LDOs adds complexity without adding much functionality. Let's see how things work with the simpler approach first.

[tadhgpearson]

That sounds good @handrews, but could you give us a specific example of how this would look in practice?

[handrews]

@tadhgpearson this isn't finalized yet, but the general idea is that in addition to the boolean validation outcome, JSON Schema implementations should also return annotation data (although disabling annotation collection for optimizing validation-only should probably be allowed). Then, applications (in this context, all OpenAPI-based tools are applications) can make their own choices about how to handle things.

So let's have a schema:

{
    "type": "object",
    "properties": {
        "foo": {
            "title": "Title adjacent to reference",
            "description": "Lots of text",
            "readOnly": true,
            "$ref": "#/definitions/fooDef"
        }
    },
    "definitions": {
        "fooDef": {
            "title": "Title in reference target",
            "description": "Even more text"
        }
    }
}

With an instance of:

{
    "foo": "whatever"
}

The annotation results might look like this (it's still up in the air, for instance I literally made up how $ref appears in the JSON Pointer-ish thing while typing this comment):

{
    "/foo": {
        "title": {
            "/properties/foo/$ref/title": "Title in reference target",
            "/properties/foo/title": "Title adjacent to reference"
        },
        "description": {
            "/properties/foo/$ref/description": "Even more text",
            "/properties/foo/description": "Lots of text"
        },
        "readOnly": true
    }
}

"/foo" is the instance pointer- this is the place in the instance to which the annotations apply.

Under that, the property names are the names of each annotation. All of these are also the keyword names that produced them, but in theory you could do something that doesn't map directly (please allow me to wave my hands on that- it's related to a complicated proposal that's under discussion and may not end up being relevant).

Under each annotation, the format is determined by the annotation.

readOnly just has a boolean, because it's explicitly stated (as of draft-07) that the overall value of readOnly is a logical OR of all applicable values. So there's no need to keep track of which value came from where- just let the application know the final outcome.

On the other hand, the way "title" and "description" are shown here is probably the default behavior. For those, "/properties/foo/... are schema pointers, more or less. Pointing across $ref is a bit weird and I need to figure out if that's the right approach. But however it ends up looking, you will be able to tell where the annotation came from so your application can decide how to use it.

So let's say your application decides that a local title always wins over a referenced title, but descriptions should just be concatenated as if each were a paragraph. There is enough information in the above format that you can tell that "Title adjacent to reference" is the title that wins, and then you can construct "Lots of text\n\nEven more text" from the various descriptions.

This is because there are many ways that an application might want to handle multiple titles / descriptions / defaults / etc: closest to root wins, most specific wins, combine them somehow, throw an error, whatever. Depending on the application and the keyword, any of these could be reasonable.

I would expect a system like OpenAPI to pick strategies for consistency. So then an OpenAPI implementation could use a regular old JSON Schema implementation to gather the annotations, then examine them to resolve any ambiguities however OpenAPI thinks they should be resolved. And then other applications can still use JSON Schema and resolve ambiguities differently.

There will no doubt be further debate about how much to standardize and how much to defer to applications, but that's a lot better than the current situation where there's not even any guidance on what sort of decisions are valid to make, much less any implementation requirements to facilitate making them.

[handrews]

@tadhgpearson & everyone:

json-schema-org/json-schema-spec#523 discusses keywords alongside of $ref
json-schema-org/json-schema-spec#530 discusses more of the process of annotation collection

Both are targeted for draft-08, hopefully by the end of May.

[Relequestual]

Keywords alongside $ref now has an open PR against. So I can't imagine it won't be included in draft-8.

[sgeeroms]

I found this thread, and for me it should be possible to added description sibling next to a ref$ tag.
This is mainly usefull if you reference a simple types and not objects.

If you define a base type like for example:

  IPAddressV4:
    type: string
    format: ipv4
    pattern: '^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$'

And you use this type in a object, for example

  DualPortServer:
    type: object
    description: Server with two ports     
    required:
      - ipAddress1
    properties:
      managementPort:
        $ref: '#/definitions/IPAddressV4'
        description: LAN management port for configuring the server
      ipAddress1:
        $ref: '#/definitions/IPAddressV4'
        description: Primary (mandatory) transfer ethernet IP address.
      ipAddress2:
        $ref: '#/definitions/IPAddressV4'
        description: Secundary (optinal) transfer ethernet IP address.

Maybe I am not doing this the right way? But if so it would be nice to be able to set a specific description.

[hkosova]

@sgeeroms please see this comment.

[sgeeroms]

@hkosova, Thanks a lot! Didn't know the "allOf" could be used for simple fields as well.

[bobvanderlinden]

I ran into this issue because swagger 1.2 _does_ support property descriptions where there is a reference to a different type:

{
  "properties": {
    "myProperty": {
      "type": "MyModel",
      "description": "Description of property"
    }
  }
}

In addition, swagger-converter (swagger 1.2 -> openapi 2.0 converter) generates the following from this definition:

{
  "properties": {
    "myProperty": {
      "$ref": "#/definitions/MyModel",
      "description": "Description of property"
    }
  }
}

This is not according to the openapi 2.0 specification. api-spec-converter gives a warning for the description field, as it resides next to $ref. Other tools often ignore the description.

[ghost]

Hi
My Project has user defined classes for Duration, Cost, etc and these are being used as nested fields in other schema classes and I need to give description to each nested fields which is referenced to that class and I am using swagger-annotation to generate docs. I tried the allOf but it didn't work.

E.g:
class Duration{
int value;
int text;
}

@Schema
class OfficeSchedule{
int val;
String name;
Duration startDuration;
Duration endDuration;

public int getVal(){
return val;
}

public String getName(){
return name;
}

@Schema(description = "start duration of office schedule", accessMode = AccessMode.READ_ONLY)
public Duration getStartDuration() {
return startDuration;
}

@Schema(description = "end duration of office schedule", accessMode = AccessMode.READ_ONLY)
public Duration getEndDuration() {
return endDuration;
}
}

I tried using allOf but Json stillis not giving referenced class property description and updating the same schema description and the last property with

Json Schema Output

"Duration": { "title": "Duration", "type": "object", "properties": { "double": { "type": "number", "format": "double" }, "bigDecimal": { "type": "number" } }, "description": "end duration of office schedule", "readOnly": true, "allOf": [ { "$ref": "#/components/schemas/Duration" } ] }

Above example is a sample and almost all classes declared for our project are being used internally in other classes as fields. Can anyone help with this I need to provide referenced property level descriptions?

[Arbuste]

Hi, just adding one instance of reusaility need here.

My own personal need is to change the required fields of an object depending on which service it is requested in.

I tried to use the allOf trick but as I expected this is not well handled by the client generator. Anyway, I think this is not the purpose of the allOf element.

[RicoSuter]

I'd expect that this (OpenAPI 3)

"startLocation": {
    "description": "The location of the trip start.",
    "nullable": true,
    "oneOf": [
        {
            "$ref": "#/components/schemas/Location"
        }
    ]
},

or this (Swagger 2)

"startLocation": {
    "description": "The location of the trip start.",
    "allOf": [
        {
            "$ref": "#/components/schemas/Location"
        }
    ]
},

should be allowed, but eg. swagger-codegen does not allow this. Why is one of these samples not the current way to go? Or something similar to avoid $ref with additional properties?

[hkosova]

In Schema Objects, $ref siblings will be supported in OAS 3.1 which will use the JSON Schema 2019-09. As mentioned in JSON Schema 2019-09 release notes, it now allows keywords alongside $ref.

However, this change does not affect $refs outside of Schema Objects, such as $refs in parameters or responses.

Related: #2099

[philsturgeon]

As @hkosova mentions, siblings are now welcome in schema objects, and will likely be allowed in various other places around the OpenAPI spec.