Dredd: Un-expected behaviour when using `$ref` inside the definitions

[davidgengenbach]

Describe the bug
When using $ref in a model definition, the schema is not validated by dredd. Instead it passes without testing the response at all.

To Reproduce
See https://gist.github.com/davidgengenbach/cf0b962e8276785f006fdba0c41a9bdc

When de-referencing (= removing the $ref and inlining the definition), the behaviour is correct and response is validated again.
Unfortunately, de-referencing is not possible in some cases, eg. when having recursive/cyclic definitions.

Expected behavior
$refs should be resolved correctly as specified in the documentation. With other tools, such $refs work.

What is in your dredd.yml?

swagger: "2.0"
info:
  title: HttpBin
  description: Test
  version: 1.0.0
basePath: /
paths:
  /get:
    get:
      summary: Test
      responses:
         200:
            description: OK
            schema:
              $ref: "#/definitions/HttpBinResponse"
definitions:
  HttpBinResponse:
    type: object
    # The response does contain more properties, thus dredd should fail the tests
    additionalProperties: false
    # This should fail the tests for sure, since this property is not given in the response
    required:
      - SomeWrongAttributeWhichIsNotInResponseButRequired
    properties:
      SomeWrongAttributeWhichIsNotInResponseButRequired:
        type: string
      SomeReference:
        # This reference breaks the validation - by adding it, the schema tests pass without even validating anything
        $ref: '#/definitions/SomeReference'
  SomeReference:
    type: object
    properties:
      property:
          type: string

What's your dredd --version output?

dredd v5.4.4 (Linux 4.19.10-300.fc29.x86_64; x64)

Does dredd --level=debug uncover something?
No

Can you send us failing test in a Pull Request?
I'd love to if you give me some hints where to look at.

[kylef]

Had a quick look into this and wanted to share my findings so far. The latest version of the Swagger parser is emitting the following JSON Schema:

{
    "definitions": {
        "HttpBinResponse": {
            "additionalProperties": false,
            "required": [
                "SomeWrongAttributeWhichIsNotInResponseButRequired"
            ],
            "type": "object",
            "properties": {
                "SomeReference": {
                    "$ref": "#/definitions/SomeReference"
                },
                "SomeWrongAttributeWhichIsNotInResponseButRequired": {
                    "type": "string"
                }
            }
        },
        "SomeReference": {
            "type": "object",
            "properties": {
                "property": {
                    "type": "string"
                }
            }
        }
    },
    "allOf": [
        {
            "$ref": "#/definitions/HttpBinResponse"
        }
    ]
}

Which would appear to be correct JSON Schema derived from the Swagger Schema. Going to a site like https://www.jsonschemavalidator.net/ and pasting the schema and the response body, I am greeted with an error as expected.

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Connection": "close", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.60.0"
  }, 
  "origin": "::1", 
  "url": "https://httpbin.org/get"
}

It would appear that the library used in Dredd for JSON Schema valdiation (https://github.com/geraintluff/tv4) is not behaving as expected. I would gather it might be a problem with allOf support. tv4 does say they support references. I think the next step would be to confirm if there is a problem with tv4 with the given body and schema.

[kylef]

It may also be easier to debug the problem with API Blueprint as we can directly alter the JSON Schema easier as it isn't derived from Swagger:

# HTTPBin

## GET /get

+ Response 200 (application/json)
    + Schema

            {
                "definitions": {
                    "HttpBinResponse": {
                        "additionalProperties": false,
                        "required": [
                            "SomeWrongAttributeWhichIsNotInResponseButRequired"
                        ],
                        "type": "object",
                        "properties": {
                            "SomeReference": {
                                "$ref": "#/definitions/SomeReference"
                            },
                            "SomeWrongAttributeWhichIsNotInResponseButRequired": {
                                "type": "string"
                            }
                        }
                    },
                    "SomeReference": {
                        "type": "object",
                        "properties": {
                            "property": {
                                "type": "string"
                            }
                        }
                    }
                },
                "allOf": [
                    {
                        "$ref": "#/definitions/HttpBinResponse"
                    }
                ]
            }

[GabbyChang]

Same here

[bokece]

Is there a workaround regarding this open issue? The only solution i see is to update the specifications file; remove the $ref references and add the nested object definitions manually. In complex objects this is time-consuming though. So a workaround or a fix would really help.

[davidgengenbach]

Is there a workaround regarding this open issue? The only solution i see is to update the specifications file; remove the $ref references and add the nested object definitions manually. In complex objects this is time-consuming though. So a workaround or a fix would really help.

In the bug description I also explained this approach:

„When de-referencing (= removing the $ref and inlining the definition), the behaviour is correct and response is validated again.
Unfortunately, de-referencing is not possible in some cases, eg. when having recursive/cyclic definitions.“

Dereferencing can be done automatically, e.g. https://apitools.dev/swagger-parser/docs/swagger-parser.html#dereferenceapi-options-callback

[bokece]

Thank you a lot @davidgengenbach. You are right. I would like to have a "universal" solution as you mentioned this is not always possible.

[kylef]

@davidgengenbach This problem has been resolved in Dredd 13.0.1:

$ dredd test.yaml https://httpbin.org
fail: GET (200) /get duration: 409ms
info: Displaying failed tests...
fail: GET (200) /get duration: 409ms
fail: body: data should NOT have additional properties
body: data should NOT have additional properties
body: data should NOT have additional properties
body: data should NOT have additional properties
body: At '/SomeWrongAttributeWhichIsNotInResponseButRequired' Missing required property: SomeWrongAttributeWhichIsNotInResponseButRequired

request:
method: GET
uri: /get
headers:
    User-Agent: Dredd/13.0.0 (Darwin 18.7.0; x64)

body:



expected:
headers:

statusCode: 200
bodySchema: {"allOf":[{"$ref":"#/definitions/HttpBinResponse"}],"definitions":{"HttpBinResponse":{"type":"object","additionalProperties":false,"required":["SomeWrongAttributeWhichIsNotInResponseButRequired"],"properties":{"SomeWrongAttributeWhichIsNotInResponseButRequired":{"type":"string"},"SomeReference":{"$ref":"#/definitions/SomeReference"}}},"SomeReference":{"type":"object","properties":{"property":{"type":"string"}}}}}


actual:
statusCode: 200
headers:
    date: Mon, 17 Feb 2020 13:11:23 GMT
    content-type: application/json
    content-length: 281
    connection: close
    server: gunicorn/19.9.0
    access-control-allow-origin: *
    access-control-allow-credentials: true

bodyEncoding: utf-8
body:
{
  "args": {},
  "headers": {
    "Content-Length": "0",
    "Host": "httpbin.org",
    "User-Agent": "Dredd/13.0.0 (Darwin 18.7.0; x64)",
    "X-Amzn-Trace-Id": "Root=1-5e4a90fb-c35d0a8c564756f813ab4e9d"
  },
  "origin": "89.36.68.62",
  "url": "https://httpbin.org/get"
}



complete: 0 passing, 1 failing, 0 errors, 0 skipped, 1 total
complete: Tests took 414ms