Datamodel-code-generator: Exception: A Parser can not resolve classes when using `$ref` if references are nested

Created on 13 Jan 2021 · 9Comments · Source: koxudaxi/datamodel-code-generator

Describe the bug
Parser can not resolve classes when using $ref if the path is nested even though the reference is a valid json pointer.

File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\__main__.py", line 281, in main
    generate(
  File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\__init__.py", line 259, in generate
    results = parser.parse()
  File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\parser\base.py", line 332, in parse
    _, sorted_data_models, require_update_action_models = sort_data_models(
  File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\parser\base.py", line 110, in sort_data_models
    return sort_data_models(
  File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\parser\base.py", line 110, in sort_data_models
    return sort_data_models(
  File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\parser\base.py", line 110, in sort_data_models
    return sort_data_models(
  [Previous line repeated 97 more times]
  File "c:\users\sonali.ingale\appdata\local\programs\python\python39\lib\site-packages\datamodel_code_generator\parser\base.py", line 158, in sort_data_models
    raise Exception(f'A Parser can not resolve classes: {unresolved_classes}.')
Exception: A Parser can not resolve classes: [class: Person references: {'Dog', 'Cat'}]

To Reproduce

Example schema:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Person",
    "type": "object",
    "properties": {
        "name": {
            "title": "name",
            "type": "string"
        },
        "pet": {
            "title": "pet",
            "type": "object",
            "oneOf": [
                {
                    "$ref": "#/definitions/Pets/Cat"
                },
                {
                    "$ref": "#/definitions/Pets/Dog"
                }
            ]
        }
    },
    "definitions": {
        "Pets": {
            "Cat": {
                "title": "Cat",
                "type": "object",
                "required": [
                    "pet_type",
                    "hunts",
                    "age"
                ],
                "properties": {
                    "pet_type": {
                        "enum": [
                            "Cat"
                        ]
                    },
                    "hunts": {
                        "type": "boolean"
                    },
                    "age": {
                        "type": "string"
                    }
                }
            },
            "Dog": {
                "title": "Dog",
                "type": "object",
                "required": [
                    "pet_type",
                    "bark",
                    "breed"
                ],
                "properties": {
                    "pet_type": {
                        "enum": [
                            "Dog"
                        ]
                    },
                    "bark": {
                        "type": "boolean"
                    },
                    "breed": {
                        "type": "string"
                    }
                }
            }
        }
    },
    "additionalProperties": false
}

Used commandline:

$ datamodel-codegen --input pets.json --input-file-type=jsonschema --output pets.py

Expected behavior
I was expecting following things which is working when I do not encapsulate "Cat" and "Dogs" inside "Pet".

Create the classes "Cat" and "Dog"
Add "pet" attribute with Union of "Cat" and "Dog"

Version:

windows
Python version: 3.9.0
datamodel-code-generator version:0.6.18

Additional context
I tried to pull the code and run this example but I am getting the same error. I have few large schemas which are encapsulating references for readability. After editing the schema according to datamodel-code-generator, it works wonderfully 👍 . Thanks for this module.

released

Source

sonali686

Most helpful comment

@koxudaxi
I've just tested and it works correctly.
Thank you) This library is awesome!)

MikeZaharov on 25 Jan 2021

🎉1 😄1

All 9 comments

@sonali686
I'm sorry. I have overlooked a notification for this issue.

OK, I understand why Cat and Dog models are not generated.
This parse finds JSONSchema objects in each path.
But,definitions/Pets is not object.

I just decided on how to implement it.
If $ref is a JSON pointer and It is not parsed, the parser parses it after all models parsed.

I will support it. please wait...

koxudaxi on 18 Jan 2021

@sonali686
I fixed it!! And I have released a new version as 0.6.22.

koxudaxi on 19 Jan 2021

👍1

Hi! I have the similar problem for the following case:
test.json

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "test.json",
  "description": "test",
  "type": "object",
  "required": [
    "test_id",
    "test_ip",
    "result"
  ],
  "properties": {
    "test_id": {
      "type": "string",
      "description": "test ID"
    },
    "test_ip": {
      "$ref": "base_test.json#/definitions/first"
    }
  }
}

base_test.json:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "base_test.json",
  "description": "test",
  "type": "object",
  "definitions": {
    "first": {
      "$ref": "#/definitions/second"
    },
    "second": {
      "type": "string"
    }
  }
}

Used commandline:

datamodel-codegen --input test.json --input-file-type=jsonschema --output model.py

Error:

raise Exception(f'A Parser can not resolve classes: {unresolved_classes}.')
Exception: A Parser can not resolve classes: [class: TestTestTestEvent references: {'First'}], [class: First references: {'Second'}].

The issue can be solved by editing ref link:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "base_test.json",
  "description": "test",
  "type": "object",
  "definitions": {
    "first": {
      "$ref": "base_test.json#/definitions/second"
    },
    "second": {
      "type": "string"
    }
  }
}

But i don't sure it's the right solution

Version:

OS: Ubuntu 16.04
Python version: 3.7
datamodel-code-generator version: 0.6.23

MikeZaharov on 22 Jan 2021

Hi @MikeZaharov,
Thank you for the bug report.
I just fixed it.
And, I have released a new version 0.6.25
I tested your schemas with the version.
model.py

# generated by datamodel-codegen:
#   filename:  test.json
#   timestamp: 2021-01-24T16:08:55+00:00

from __future__ import annotations

from pydantic import BaseModel, Field


class Second(BaseModel):
    __root__: str


class First(BaseModel):
    __root__: Second


class Model(BaseModel):
    test_id: str = Field(..., description='test ID')
    test_ip: First

koxudaxi on 24 Jan 2021

@koxudaxi
I've just tested and it works correctly.
Thank you) This library is awesome!)

MikeZaharov on 25 Jan 2021

🎉1 😄1

@koxudaxi
I am using latest version (0.8.2) of the library and delighted to see many new options. Thanks for the awesome library once again :)

I came across one more scenario where library fails to resolve classes while using $ref.

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Person",
    "type": "object",
    "properties": {
        "name": {
            "title": "name",
            "type": "string"
        },
        "pet": {
            "title": "pet",
            "type": "object",
            "oneOf": [
                {
                    "$ref": "#/definitions/Pets/Cat"
                },
                {
                    "$ref": "#/definitions/Pets/Dog"
                }
            ]
        }
    },
    "definitions": {
        "CatBreed":{
            "C1":
            {
                "title":"C1",
                "type": "object",
                "properties":
                {
                    "hunts": {
                        "type": "boolean"
                    },
                    "age": {
                        "type": "string"
                    }
                }
            },
            "C2":
            {
                "title":"C2",
                "type": "object",
                "properties":
                {
                    "hunts": {
                        "type": "boolean"
                    },
                    "age": {
                        "type": "string"
                    }
                }
            }
        },
        "DogBreed":{
            "D1":
            {
                "title":"D1",
                "type": "object",
                "properties":
                {
                    "bark": {
                        "type": "boolean"
                    },
                    "age": {
                        "type": "string"
                    }
                }
            },
            "D2":
            {
                "title":"D2",
                "type": "object",
                "properties":
                {
                    "hunts": {
                        "type": "boolean"
                    },
                    "age": {
                        "type": "string"
                    }
                }
            }
        },
        "Pets": {
            "Cat": {
                "title": "Cat",
                "type": "object",
                "required": [
                    "pet_type",
                    "age"
                ],
                "properties": {
                    "pet_type": {
                        "enum": [
                            "Cat"
                        ]
                    },
                    "breed": {
                        "title": "breed",
                        "type": "object",
                        "oneOf": [
                            {
                                "$ref": "#/definitions/CatBreed/C1"
                            },
                            {
                                "$ref": "#/definitions/CatBreed/C2"
                            }
                        ]
                    }
                }
            },
            "Dog": {
                "title": "Dog",
                "type": "object",
                "required": [
                    "pet_type",
                    "breed"
                ],
                "properties": {
                    "pet_type": {
                        "enum": [
                            "Dog"
                        ]
                    },
                    "breed": {
                        "title": "breed",
                        "type": "string",
                        "oneOf": [
                            {
                                "$ref": "#/definitions/DogBreed/D1"
                            },
                            {
                                "$ref": "#/definitions/DogBreed/D2"
                            }
                        ]
                    }
                }
            }
        }
    },
    "additionalProperties": false
}

Is it possible to support this scenario as well? Is it possible to resolve any reference if it is a valid json pointer? Currently I am getting same exception as mentioned in my original post.

sonali686 on 24 Feb 2021

@sonali686
Wow, I don't expect the pattern.
The code-generator resolves JsonPointer after parsing all non-JsonPointer objects.
Your example has JsonPointer is in JsonPointer.
It means the nested JsonPointer is not resolved.

I have fixed the code with my hands instantly.
It can generate it :)
I will push the complete fixed version tomorrow.
Thank you.

I put generated models.
Did you expect the models?

# generated by datamodel-codegen:
#   filename:  person.json
#   timestamp: 2019-07-26T00:00:00+00:00

from __future__ import annotations

from enum import Enum
from typing import Any, Optional, Union

from pydantic import BaseModel, Field


class CatBreed(BaseModel):
    __root__: Any


class DogBreed(BaseModel):
    __root__: Any


class Pets(BaseModel):
    __root__: Any


class PetType(Enum):
    Cat = 'Cat'


class PetType1(Enum):
    Dog = 'Dog'


class C1(BaseModel):
    hunts: Optional[bool] = None
    age: Optional[str] = None


class C2(BaseModel):
    hunts: Optional[bool] = None
    age: Optional[str] = None


class D1(BaseModel):
    bark: Optional[bool] = None
    age: Optional[str] = None


class D2(BaseModel):
    hunts: Optional[bool] = None
    age: Optional[str] = None


class Cat(BaseModel):
    pet_type: PetType
    breed: Optional[Union[C1, C2]] = Field(None, title='breed')


class Dog(BaseModel):
    pet_type: PetType1
    breed: Union[D1, D2] = Field(..., title='breed')


class Person(BaseModel):
    name: Optional[str] = Field(None, title='name')
    pet: Optional[Union[Cat, Dog]] = Field(None, title='pet')

koxudaxi on 24 Feb 2021

I have fixed the code with my hands instantly.
It can generate it :)
I will push the complete fixed version tomorrow.
Thank you.

I put generated models.
Did you expect the models?

# generated by datamodel-codegen:
#   filename:  person.json
#   timestamp: 2019-07-26T00:00:00+00:00

from __future__ import annotations

from enum import Enum
from typing import Any, Optional, Union

from pydantic import BaseModel, Field


class CatBreed(BaseModel):
    __root__: Any


class DogBreed(BaseModel):
    __root__: Any


class Pets(BaseModel):
    __root__: Any


class PetType(Enum):
    Cat = 'Cat'


class PetType1(Enum):
    Dog = 'Dog'


class C1(BaseModel):
    hunts: Optional[bool] = None
    age: Optional[str] = None


class C2(BaseModel):
    hunts: Optional[bool] = None
    age: Optional[str] = None


class D1(BaseModel):
    bark: Optional[bool] = None
    age: Optional[str] = None


class D2(BaseModel):
    hunts: Optional[bool] = None
    age: Optional[str] = None


class Cat(BaseModel):
    pet_type: PetType
    breed: Optional[Union[C1, C2]] = Field(None, title='breed')


class Dog(BaseModel):
    pet_type: PetType1
    breed: Union[D1, D2] = Field(..., title='breed')


class Person(BaseModel):
    name: Optional[str] = Field(None, title='name')
    pet: Optional[Union[Cat, Dog]] = Field(None, title='pet')

@koxudaxi

Yes I expected these models. Thanks for providing the solution so fast. 👍

sonali686 on 25 Feb 2021

@sonali686
Thank you for your response!!

I have released a new version as 0.8.3!!

koxudaxi on 25 Feb 2021

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Nested additionalProperties result in empty value type being generated

languitar · 3Comments

additionalProperties always result in Any Python type

languitar · 3Comments

json schema: minus in name results in invalid python module name

FlorianLudwig · 3Comments

Support for remote $refs

ioggstream · 6Comments

Pass through example strings

languitar · 3Comments