Datamodel-code-generator: Does not inherit all the classes mentioned in `allOf`

[rushabh-v]

Describe the bug
It is not extending all the mentioned classes in the allOf attribute of the schema.

To Reproduce
I am running this on the directory: https://github.com/AnalyticalGraphicsInc/czml-writer/tree/master/Schema. But it does not inherit all the classes as I mentioned above. For example, one of the schemas is as follows,

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "https://analyticalgraphicsinc.github.io/czml-writer/Schema/Color.json",
    "title": "Color",
    "description": "A color. The color can optionally vary over time.",
    "allOf": [
        { "$ref": "InterpolatableProperty.json" },
        { "$ref": "DeletableProperty.json" },
        { "$ref": "ValueProperties/RgbaValueProperty.json" },
        { "$ref": "ValueProperties/RgbafValueProperty.json" },
        { "$ref": "ValueProperties/ReferenceValueProperty.json" }
    ],
    "type": [
        "array",
        "object"
    ],
    "items": {
        "$ref": "#"
    },
    "properties": {
        "rgba": {
            "$ref": "Values/RgbaValue.json",
            "description": "The color specified as an array of color components `[Red, Green, Blue, Alpha]` where each component is an integer in the range 0-255."
        },
        "rgbaf": {
            "$ref": "Values/RgbafValue.json",
            "description": "The color specified as an array of color components `[Red, Green, Blue, Alpha]` where each component is a double in the range 0.0-1.0."
        },
        "reference": {
            "$ref": "Values/ReferenceValue.json",
            "description": "The color specified as a reference to another property."
        }
    }
}

It generates the class as follows:

# generated by datamodel-codegen:
#   filename:  Color.json
#   timestamp: 2021-03-29T05:46:25+00:00

from __future__ import annotations

from typing import Optional

from pydantic import BaseModel, Field

from . import Cylinder, LabelStyle


class Color(BaseModel):
    """
    A color. The color can optionally vary over time.
    """

    rgba: Optional[Cylinder.RgbaValue] = Field(
        None,
        description='The color specified as an array of color components `[Red, Green, Blue, Alpha]` where each component is an integer in the range 0-255.',
    )
    rgbaf: Optional[Cylinder.RgbafValue] = Field(
        None,
        description='The color specified as an array of color components `[Red, Green, Blue, Alpha]` where each component is a double in the range 0.0-1.0.',
    )
    reference: Optional[LabelStyle.ReferenceValue] = Field(
        None, description='The color specified as a reference to another property.'
    )

Expected behavior
It should have inherited the classes DeletableProperty and InterpolatableProperty

Additional context
Those classes have been generated and are residing in the same directory. It's just that they are not being imported and inherited.

[koxudaxi]

@rushabh-v
Thank you for creating this issue.
The schema has allOf and properties. So, the code-generator detected it as an object model and not allOf model.
I can fix it.

However, The schema has array attributes too.
The pattern is a little complex.

"type": [
        "array",
        "object"
    ],
    "items": {
        "$ref": "#"
    },

I guess the combined type is written as Union in pydantic.
Unfortunately, these models can't resolve by pyndatic. :(

from __future__ import annotations
from typing import Union

from pydantic import BaseModel


class ModelA(BaseModel):
    name: str = None


class ModelB(BaseModel):
    age: int = None


class AllOfModel(ModelA, ModelB):
    description: str = None


class Model(BaseModel):
    __root__: Union[Model, AllOfModel] = None


Model.update_forward_refs()

print(Model(__root__=AllOfModel(name='dog', age=1)))

output

Traceback (most recent call last):
  File "/Users/koudai/PycharmProjects/pythonProject10/main.py", line 25, in <module>
    print(Model(__root__=AllOfModel(name='dog', age=1)))
  File "pydantic/main.py", line 398, in pydantic.main.BaseModel.__init__
  File "pydantic/main.py", line 1034, in pydantic.main.validate_model
  File "pydantic/fields.py", line 723, in pydantic.fields.ModelField.validate
  File "pydantic/fields.py", line 899, in pydantic.fields.ModelField._validate_singleton
  ...
    File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/abc.py", line 98, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison

[koxudaxi]

I thought about schema again 🤔
I don't know whether we can combine allOf, properties, and items.
If someone knows the spec then please write the detail here 🙏
Thank you.

[rushabh-v]

Hi @koxudaxi, Thanks for responding!

Could you explain why allOf, properties and items can't be combined? I think understanding that will help me in my project, even if we can't get it to work here.

And I am willing to contribute if it is possible to get it working here.

Thanks!

[koxudaxi]

@rushabh-v

Thank you for your reply 😄

Could you explain why allOf, properties and items can't be combined? I think understanding that will help me in my project, even if we can't get it to work here.

I don't know to work combined schema properties and items :(
But, I write a combined schema as a sample.

  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": [
        "array",
        "object"
    ],
    "items": {
        "$ref": "#"
    },
       "properties": {
        "name": {
           "type": "string"
        }

Does the pattern allow an array and object?

[ {"name": "dog"}, {"name": "cat"}]

And

{"name": "dog"}

If the understanding is correct then I will be able to implement it...

I expect the model if allOf and properties are defined. I think the model is no problem.

class Color(InterpolatableProperty, DeletableProperty, RgbaValueProperty, RgbafValueProperty, ReferenceValueProperty):
    """
    A color. The color can optionally vary over time.
    """

    rgba: Optional[Cylinder.RgbaValue] = Field(
        None,
        description='The color specified as an array of color components `[Red, Green, Blue, Alpha]` where each component is an integer in the range 0-255.',
    )
    rgbaf: Optional[Cylinder.RgbafValue] = Field(
        None,
        description='The color specified as an array of color components `[Red, Green, Blue, Alpha]` where each component is a double in the range 0.0-1.0.',
    )
    reference: Optional[LabelStyle.ReferenceValue] = Field(
        None, description='The color specified as a reference to another property.'
    )

items is written as root model in Pydantic
for example,

class Colors(BaseModel):
   __root__: List[Color]

we can write the below model.

class Model(BaseModel):
   __root__: Union[Color, Colors]

Added:
This idea is changed from the previous comment https://github.com/koxudaxi/datamodel-code-generator/issues/399#issuecomment-809122566
I misunderstood when I write the comment. sorry.

[rushabh-v]

Thanks for your willingness to fix this. I don't have the context to imagine the amount of work it will require but please let me know if you need any help.

Thank you :)

[koxudaxi]

I always welcome PR!! But, I have already started fixing it 😅
Thank you.

[koxudaxi]

I merged a fixed for combined allOf and object.
Next, I will implement a feature for combined object and array .
a better way is...

class Colors(BaseModel):
   __root__: Unoin[List[Color], Color]

[rushabh-v]

Hey, that was quick. Thanks :smile:

[koxudaxi]

@rushabh-v
I'm implementing the feature.
But, I met a new problem.
Some inherited class is __root__ model.
It means Pydantic can't treat these models.
We can only one choice to resolve the problems
It's to ignore inherited __root__ models.

What did you think of the way?

Runtime error

  ...
    class Color1(
  File "pydantic/main.py", line 301, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/main.py", line 198, in pydantic.main.validate_custom_root_type
ValueError: __root__ cannot be mixed with other fields

Generated Code

# generated by datamodel-codegen:
#   filename:  https://raw.githubusercontent.com/AnalyticalGraphicsInc/czml-writer/master/Schema/Color.json
#   timestamp: 2021-03-31T06:23:57+00:00

from __future__ import annotations

from datetime import datetime
from typing import Any, List, Optional, Union

from pydantic import BaseModel, Field


class InterpolatableProperty(BaseModel):
    epoch: Optional[datetime] = Field(
        None,
        description='The epoch to use for times specified as seconds since an epoch.',
    )
    interpolationAlgorithm: Optional[str] = Field(
        'LINEAR',
        description='The interpolation algorithm to use when interpolating. Valid values are "LINEAR", "LAGRANGE", and "HERMITE".',
    )
    interpolationDegree: Optional[float] = Field(
        1, description='The degree of interpolation to use when interpolating.'
    )
    forwardExtrapolationType: Optional[str] = Field(
        'NONE',
        description='The type of extrapolation to perform when a value is requested at a time after any available samples. Valid values are "NONE", "HOLD", and "EXTRAPOLATE".',
    )
    forwardExtrapolationDuration: Optional[float] = Field(
        0.0,
        description='The amount of time to extrapolate forward before the property becomes undefined. A value of 0 will extrapolate forever.',
    )
    backwardExtrapolationType: Optional[str] = Field(
        'NONE',
        description='The type of extrapolation to perform when a value is requested at a time before any available samples. Valid values are "NONE", "HOLD", and "EXTRAPOLATE".',
    )
    backwardExtrapolationDuration: Optional[float] = Field(
        0.0,
        description='The amount of time to extrapolate backward before the property becomes undefined. A value of 0 will extrapolate forever.',
    )


class DeletableProperty(BaseModel):
    delete: Optional[bool] = Field(
        None,
        description='Whether the client should delete existing samples or interval data for this property. Data will be deleted for the containing interval, or if there is no containing interval, then all data. If true, all other properties in this property will be ignored.',
    )


class RgbaValueProperty(BaseModel):
    __root__: Any = Field(
        ...,
        description='The base schema for a property whose value may be written as an array of color components `[Red, Green, Blue, Alpha]` where each component is in the range 0-255.',
    )


class RgbafValueProperty(BaseModel):
    __root__: Any = Field(
        ...,
        description='The base schema for a property whose value may be written as an array of color components `[Red, Green, Blue, Alpha]` where each component is in the range 0.0-1.0.',
    )


class ReferenceValueProperty(BaseModel):
    __root__: Any = Field(
        ...,
        description='The base schema for a property whose value may be written as a reference to another property.',
    )


class RgbaValue(BaseModel):
    __root__: List = Field(
        ...,
        description='A color specified as an array of color components `[Red, Green, Blue, Alpha]` where each component is in the range 0-255. If the array has four elements, the color is constant. If it has five or more elements, they are time-tagged samples arranged as `[Time, Red, Green, Blue, Alpha, Time, Red, Green, Blue, Alpha, ...]`, where Time is an ISO 8601 date and time string or seconds since epoch.',
        title='Rgba',
    )


class RgbafValue(BaseModel):
    __root__: List = Field(
        ...,
        description='A color specified as an array of color components `[Red, Green, Blue, Alpha]` where each component is in the range 0.0-1.0. If the array has four elements, the color is constant. If it has five or more elements, they are time-tagged samples arranged as `[Time, Red, Green, Blue, Alpha, Time, Red, Green, Blue, Alpha, ...]`, where Time is an ISO 8601 date and time string or seconds since epoch.',
        title='Rgbaf',
    )


class ReferenceValue(BaseModel):
    __root__: str = Field(
        ...,
        description='Represents a reference to another property. References can be used to specify that two properties on different objects are in fact, the same property.',
    )


class Color1(
    InterpolatableProperty,
    DeletableProperty,
    RgbaValueProperty,
    RgbafValueProperty,
    ReferenceValueProperty,
):
    rgba: Optional[RgbaValue] = Field(
        None,
        description='The color specified as an array of color components `[Red, Green, Blue, Alpha]` where each component is an integer in the range 0-255.',
    )
    rgbaf: Optional[RgbafValue] = Field(
        None,
        description='The color specified as an array of color components `[Red, Green, Blue, Alpha]` where each component is a double in the range 0.0-1.0.',
    )
    reference: Optional[ReferenceValue] = Field(
        None, description='The color specified as a reference to another property.'
    )


class Color(BaseModel):
    __root__: Union[List[Color1], Color1] = Field(
        ...,
        description='A color. The color can optionally vary over time.',
        title='Color',
    )

[rushabh-v]

We can only one choice to resolve the problems
It's to ignore inherited __root__ models.

I agree, I think ignoring __root__ models would not matter that much.

[koxudaxi]

@rushabh-v
I have released a new version as 0.10.1
Would you please test it?

[rushabh-v]

Thanks for the super quick fix @koxudaxi :)

Sure, just give me a minute, I'll test it and come back to you.

[rushabh-v]

Hi @koxudaxi, there is one little problem.

Consider a schema where "$schema": "http://json-schema.org/draft-07/schema#", in this case it will raise an index error on these lines because ref.split('#', 1)[-1] will return an empty string.

[koxudaxi]

@rushabh-v
I guess ref is $ref. $schema is not parsed. 🤔
Did you meet an error? I want to see it.

[rushabh-v]

Sure, here is the traceback,

Traceback (most recent call last):
  File "/home/rushabh/anaconda3/lib/python3.7/site-packages/datamodel_code_generator/__main__.py", line 421, in main
    empty_enum_field_name=config.empty_enum_field_name,
  File "/home/rushabh/anaconda3/lib/python3.7/site-packages/datamodel_code_generator/__init__.py", line 329, in generate
    results = parser.parse()
  File "/home/rushabh/anaconda3/lib/python3.7/site-packages/datamodel_code_generator/parser/base.py", line 382, in parse
    self.parse_raw()
  File "/home/rushabh/anaconda3/lib/python3.7/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1111, in parse_raw
    self._resolve_unparsed_json_pointer()
  File "/home/rushabh/anaconda3/lib/python3.7/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1129, in _resolve_unparsed_json_pointer
    self.parse_json_pointer(self.raw_obj, reserved_ref, path_parts)
  File "/home/rushabh/anaconda3/lib/python3.7/site-packages/datamodel_code_generator/parser/jsonschema.py", line 1140, in parse_json_pointer
    if path[0] == '/':  # pragma: no cover
IndexError: string index out of range

[koxudaxi]

Thanks. Also, I need an input file or the URL too.
I'm sorry I must sleep now. I will check the problem tomorrow.

[rushabh-v]

No problem, take your time :smile:

I got this error while using the codegen on an entire directory at once. Download Schema.zip and run it on the Schema directory. The command I am using is,

datamodel-codegen  --use-schema-description --input /path/to/Schema/ --input-file-type jsonschema --output ./generated/

[koxudaxi]

Thank you for giving me the schema.

I have understood why the error is happened.
The parser can't resolve actual $ref.
The files have $id attributes to indicate self url.
But, the file is in our local disk.
The parser is confused what file should read ....

I will fix it.

[koxudaxi]

@rushabh-v
I'm sorry for late my work.
I have released a new version as 0.10.2
I believe fixed it.

[rushabh-v]

Hi, works like a charm now :100:

Thanks for fixing it :)

Is the naming to the extra class given by one of these PRs. For example Color1 in this comment. If yes, then in my opinion we can name it like _ClassName instead of ClassName1.

It's only a suggestion, it's okay if you think ClassName1 is better, it was just my opinion.

[koxudaxi]

@rushabh-v
Thank you for testing it.

Is the naming to the extra class given by one of these PRs. For example Color1 in this comment. If yes, then in my opinion we can name it like _ClassName instead of ClassName1.

I agree. I think I should change the generated name for the class.
But, I feel _ClassName is a private class on python coding style. It's not correct in this context.
We don't know ClassName is a singular name? Or a name for the list?
ClassNameModel may be better than ClassName1.
However, the choice is not good.
If someone suggests a good name then I will change it.

[rushabh-v]

Hmm, reasonable.

No problem. And thanks again for fixing it, closing the issue!