Datamodel-code-generator: Support generating models from json-schemas defined in multiple files

[soider]

Is your feature request related to a problem? Please describe.
Now the tool cli API supports only single file as input. I'd suggest to extend it to have directory as as option. This would help with generating models from json-schema (or may be even from other formats as well). For example, let's have the following schemas (see below).

Since the tool doesn't support directory as input I make a separate CLI call for both file. In that case the output would have:

• model generated from object.json for Object
• model generated from object.json for referenced Metadata
• model generated from metadata.json

Even more, if we have 2 and more files, which, like object.json refer to the same external reference, we will have copy of the model in every output file.

// object.json
{
  "$schema": "http://json-schema.org/schema#",
  "type": "object",
  "title": "Object",
  "properties": {
    "metadata": {
      "$ref": "metadata.json#/"
    }
  },
  "required": [
    "metadata",
  ]
}

// another file in the same dir
{
  "$schema": "http://json-schema.org/schema#",
  "type": "object",
  "title": "Object",
  "properties": {
    "somefield": {
      "type": "integer"
    }
  },
  "required": [
    "metadata",
  ]
}

Describe the solution you'd like
In the described case I would like to avoid duplicated models output for the references.

Describe alternatives you've considered
May be, another alternative is to recursively call the datamodel-codegen in case of reference to the external file (then even if we had made few calls to generate the same reference from different referees we would just override the previous result in a separate file, instead of generating inline pydantic model-duplicate).

Additional context
Add any other context or screenshots about the feature request here.

[koxudaxi]

@soider
Thank you for creating this issue.
I guess the problem is a bug. :thinking:
I expect that the code-generator can create a referenced model from another JSONSchema.
In your case, If a user gives object.json then, the generator has to create a Metadata model from metadata.json.
Originally, the user doesn't need to pass multiple files.

[koxudaxi]

@soider
I fixed the bug and I published a new version as 0.5.34

I put an example of your use-case.

Files

$ tree /tmp/multiple_files                                                     
/tmp/multiple_files
├── metadata.json
└── object.json

/tmp/multiple_files/object.json

{
  "$schema": "http://json-schema.org/schema#",
  "type": "object",
  "title": "Object",
  "properties": {
    "metadata": {
      "$ref": "metadata.json#/"
    }
  },
  "required": [
    "metadata"
  ]
}

/tmp/multiple_files/metadata.json

{
  "$schema": "http://json-schema.org/schema#",
  "type": "object",
  "title": "Object",
  "properties": {
    "somefield": {
      "type": "integer"
    }
  },
  "required": [
    "metadata"
  ]
}

Run a command

$ datamodel-codegen --input /tmp/multiple_files/object.json --input-file-type jsonschema

Output

# generated by datamodel-codegen:
#   filename:  object.json
#   timestamp: 2020-09-19T16:12:29+00:00

from __future__ import annotations

from typing import Optional

from pydantic import BaseModel


class Metadata(BaseModel):
    somefield: Optional[int] = None


class Object(BaseModel):
    metadata: Metadata

[soider]

Hello! Thanks, it fixes the bug, but still doesn't solve the original issue.

For example, imagine you have directory "schemas" with 50 files in it. Some of them are:

• standalone (not used as reference source or target)
• reference targets
• reference sources

The tool doesn't support passing the whole directory as input, so you need to run tool separately for every file.
Most probably it would be something like shell script for loop. But with this approach the code generated would have duplicates.
For example:

• file A refs file B
• file C refs file B
• file D refs file A

The output would be:

• a.py with A and B
• c.py with C and B
• d.py with D, A and B

But if the tool would support using directory as the context it could be possible to build dependency tree and avoid generating duplicates.

[koxudaxi]

@soider
Thank you for explaining the detail.
I have understood your thought.

OK, I try it. But, an internal parser may not able to interpret multiple input files. :thinking:

[JosXa]

There should be some Context data class that is passed around in the library to keep track of types that have already been generated and reference those if necessary, I guess

[koxudaxi]

I have implemented that the parser can reference the created model.
But, I have not checked complex pattern. :(
I will do it this week.

[aexvir]

Hi there,
I'm having a similar issue and I think it's mostly related to what's being reported here, so I'll just chime in. Please let me know if it's different enough so it should go on a different issue.

I'm trying to use this tool to generate definitions from files like the example at the end of this comment. As you can see... it's a pretty nested structure, relying strongly on relative imports. And this is just an example, in our application we have many more levels and complicated importing.

I've taken a look at the code and using the debugger to find out what are the main issues preventing this structure from being used, as it's fully supported by prance for example, and this project also uses prance underneath.

I've found this line as the main culprit for not being able to have files following relative refs properly. If that conditional is removed completely, allowing all that following chunk of code to run then it suddenly starts outputting python files for them.

But then I'm finding other issues, like problems parsing iso-4217 formatted schemas, or some other broken ref links that I seem related to wrongly assumed basepaths, but I'm assuming this might be more related to the way how I'm wrapping this tool for my own usage...

So... the final question is... is this issue also taking in account scenarios like the one I'm exposing? Or should this rather go on its own issue?

example

schemas/common.yaml

Currency:
  type: string
  description: 3 letter currency code as defined by ISO-4217.
  example: EUR

schemas/finance/options.yaml

SingleOption:
  type: object
  additionalProperties: false
  properties:
    amount:
      type: number
      example: 10.5
    currency:
      $ref: "../common.yaml#/Currency"

api/paths/charges.yaml

ChargesPath:
  get:
    summary: Get charges
    responses:
      "200":
        content:
          application/json:
            schema:
              $ref: "../../schemas/finance/options.yaml#/SingleOption"

openapi.yaml

openapi: 3.0.2
...
paths:
  "/charges/{charge_id}/option":
  $ref: "api/paths/charges.yaml#/ChargesPath"

[koxudaxi]

@aexvir
Sorry for replying so late.
Thank you for creating the issue.

I've taken a look at the code and using the debugger to find out what are the main issues preventing this structure from being used, as it's fully supported by prance for example, and this project also uses prance underneath.

I got a report that prance has bugs. And I dedicated we use prance as a validator. https://github.com/koxudaxi/datamodel-code-generator/issues/105

I've found this line as the main culprit for not being able to have files following relative refs properly. If that conditional is removed completely, allowing all that following chunk of code to run then it suddenly starts outputting python files for them.

We should resolve the problem.

So... the final question is... is this issue also taking in account scenarios like the one I'm exposing? Or should this rather go on its own issue?

The current version doesn't support the files.
the code-generator doesn't parse under paths and unrelated other files.

But, The PR supports external files.
Only JsonSchema

I am surprised that people write the schema in many ways.
I want to support all ways which people need.

[koxudaxi]

@aexvir
I thought about your scenarios.
the parser must check under paths for detecting external models.
If you want it then I will implement it.

[koxudaxi]

@soider
I have released a new version as 0.6.0
The version can parse multiple JsonSchema files.
You can give a directory to --input argument.
Input directory:
https://github.com/koxudaxi/datamodel-code-generator/tree/master/tests/data/jsonschema/multiple_files

Output directory:
https://github.com/koxudaxi/datamodel-code-generator/tree/master/tests/data/expected/main/multiple_files

[JosXa]

Does this only work for JsonSchema as of now? I have a use case here where it would be super handy to have for normal JSON.

By the way, f**k this is so cool!! 🚀

[koxudaxi]

@JosXa

Does this only work for JsonSchema as of now? I have a use case here where it would be super handy to have for normal JSON.

Yes, The feature is only for JsonSchema.
Could you please tell me the use case?

[themmes]

I have a similar issue with the following structure. Note that my ref structure might not be optimal for this library, I have the flexibility to switch e.g. absolute/relative paths etc.

The error I'm getting is (with some max recursion errors):

Exception: A Parser can not resolve classes: [class: Person references: {'pet.JsonPet'}].

Example schema:

/schema
  person.json
  /definitions
    pet.json

{
  "$id": "person.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Person",
  "type": "object",
  "properties": {
    "first_name": {
      "type": "string",
      "description": "The person's first name."
    },
    "last_name": {
      "type": "string",
      "description": "The person's last name."
    },
    "age": {
      "description": "Age in years.",
      "type": "integer",
      "minimum": 0
    },
    "pets": {
      "type": "array",
      "items": [
        {
          "$ref": "definitions/pet.json#Pet"
        }
      ]
    },
    "comment": {
      "type": "null"
    }
  },
  "required": [
      "first_name",
      "last_name"
  ],
}

{
  "$id": "pet.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Pet",
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "age": {
      "type": "integer"
    }
  }
}

[themmes]

The problem was a mistake in my example, the reference should be "$ref": "definitions/pet.json#/Pet" and then it works perfectly!

[themmes]

@koxudaxi one more question, how should the $ref look like when there are definitions that rely on other definitions (both in the definitions folder)? I tried both definitions/fur.json#/Fur and fur.json#/Fur, in both cases the Python code refers to a Fur definition, but there exists no fur.py

/schema
  person.json
  /definitions
    pet.json
    fur.json

person.json is same as above..

{
  "$id": "pet.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Pet",
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "age": {
      "type": "integer"
    },
    "fur": {
      "$ref": "fur.json#/Fur"
    }
  }
}

{
  "$id": "fur.json",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Fur",
  "type": "string",
    "enum": [
        "Short hair",
        "Long hair"
    ],
}

[koxudaxi]

@themmes
I expected the Fur will be created in fur.py. if you give directory to CLI then it creates fur.py.
But, it doesn't work fine 😢
I will fix it.

[themmes]

Hi @koxudaxi, thank you that would be great! Do you know what is causing the issue? Let me know if I can help

[koxudaxi]

@themmes
I'm sorry. I was busy with a week.

I'm checking your schema and the error.
This description may be invalid.

"$ref": "definitions/pet.json#/Pet"

It may be

"$ref": "definitions/pet.json#"

This object has a title. But, the title is not a key.

Did you know the answer?

[themmes]

I'm surprised by your suggestion, because in the previous case (without the fur.json detail) adding Pet to the reference was the solution.

I've tried removing the Pet from the reference (and the same for Fur). But that results in the following error:

"/Users/tom/miniconda3/envs/py38/lib/python3.8/site-packages/datamodel_code_generator/parser/base.py", line 99, in sort_data_models
    raise Exception(f'A Parser can not resolve classes: {unresolved_classes}.')
Exception: A Parser can not resolve classes: [class: Person references: {'pet.Json'}], [class: Pet references: {'fur.Json'}].

[koxudaxi]

I'm surprised by your suggestion, because in the previous case (without the fur.json detail) adding Pet to the reference was the solution.

I've tried removing the Pet from the reference (and the same for Fur). But that results in the following error:

I talked about correct JsonSchema.
This code generator has a bug that it can't generate a model without Pet.
Because I didn't know the spec of JsonSchema. I'm Sorry.

[koxudaxi]

@themmes
I fixed the problem.
I have released a new version as 0.6.9
I added the unittest for your case.

https://github.com/koxudaxi/datamodel-code-generator/pull/272/files#diff-d9f70133872d7ebd127ebb3e8841261a498af1054748c868ea1555bec762097b

https://github.com/koxudaxi/datamodel-code-generator/pull/272/files#diff-d9f70133872d7ebd127ebb3e8841261a498af1054748c868ea1555bec762097b

[themmes]

Wow you are really fast! Thank you! I can confirm your PR is working for

datamodel-codegen --input person.json --input-file-type jsonschema --output person.py

However if we try to run it for the folder there are some incorrect file references

datamodel-codegen --input external_files_in_directory --input-file-type jsonschema --output external_files_output

FileNotFoundError: [Errno 2] No such file or directory: '/Users/tom/dev/datamodel-test/external_files_in_directory/fur.json'

[koxudaxi]

@themmes
Good catch!!
I understand it.
OK, I will fix it tomorrow.

[koxudaxi]

@themmes
I have released a new version as0.6.10.
It supports this use-case.
$ datamodel-codegen --input external_files_in_directory --input-file-type jsonschema --output external_files_output

I tested nested modules.

https://github.com/koxudaxi/datamodel-code-generator/blob/70f3ed0adb3310235b8e87989034e30a00647976/tests/data/expected/main/main_nested_directory/person.py#L11-L21

[themmes]

Nice! I tested it with some of my json-schemas, for one case it gives incorrect Python.

I created the following example, if a definition defines an array with objects with some properties the codegenerator creates two objects Friends and Friend, however it seems to put Friends in the toplevel __init__.py instead of definitions/friends.py and therewith breaking imports.

external_files_in_directory.zip

[koxudaxi]

@themmes
Thank you for reporting your case.
I will fix it 😉

[koxudaxi]

@themmes
I have fixed it for your use-case.
https://github.com/koxudaxi/datamodel-code-generator/pull/277/files#diff-9515230296b694c373b1d55f73319d2b632887f54935a9baa5513cb4455e59e1

The version is released as 0.6.11

[themmes]

That is amazing, works really well now. Thank you!

[koxudaxi]

I believe the issue is solved.
If some see any problem then, please create a new issue.

Thank you.

[koxudaxi]

I'm surprised by your suggestion, because in the previous case (without the fur.json detail) adding Pet to the reference was the solution.

I've tried removing the Pet from the reference (and the same for Fur). But that results in the following error:

"/Users/tom/miniconda3/envs/py38/lib/python3.8/site-packages/datamodel_code_generator/parser/base.py", line 99, in sort_data_models
    raise Exception(f'A Parser can not resolve classes: {unresolved_classes}.')
Exception: A Parser can not resolve classes: [class: Person references: {'pet.Json'}], [class: Pet references: {'fur.Json'}].

Hi, @themmes
I don't know you recently use the library.
But, I tell you breaking changes in the next version (0.9.0). https://github.com/koxudaxi/datamodel-code-generator/issues/357
The next version may not parse your JSONSchema. Because it needs valid $ref.

It means You should change $ref to valid $refwhich I talked you about it https://github.com/koxudaxi/datamodel-code-generator/issues/215#issuecomment-737098137

Example

"$ref": "definitions/pet.json#/Pet"
to
"$ref": "definitions/pet.json#"

Changed test case

https://github.com/koxudaxi/datamodel-code-generator/pull/354/files#diff-dee29a249ed79c3bccb632aa80578f100630d38f2af93974e2a3f71695c32460

[themmes]

@koxudaxi Thanks a lot for the heads up, thats very kind 🙏 ! I'll make the changes like you suggested and will test it soon with this new version.

[koxudaxi]

@themmes
I will notify you when the new version is released.
Thank you very much!! 😄

[aexvir]

As always, thanks for the effort on this @koxudaxi 🙂️
But unfortunately I'm a bit confused by this message you posted today...
valid $ref seem to be able to handle specific elements inside a single definition

$ref Syntax or this other reference
According to RFC3986, the $ref string value (JSON Reference) should contain a URI, which identifies the location of the JSON value you are referencing to. If the string value does not conform URI syntax rules, it causes an error during the resolving. Any members other than $ref in a JSON Reference object are ignored.

Check this list for example values of a JSON reference in specific cases:

• Local Reference – $ref: '#/definitions/myElement' # means go to the root of the current document and then find elements definitions and myElement one after one.
• Remote Reference – $ref: 'document.json' Uses the whole document located on the same server and in the same location.
  
  
  
  • The element of the document located in the same folder – $ref: 'document.json#/myElement'
  
  

So I'm not sure why this is not a valid $ref and should be avoided 🤔️

[koxudaxi]

@aexvir
Your thought about $ref is right. But, You misunderstand my changes.
I had to explain the detail of invalid $ref.
The test case did reference "$ref": "file_a.json#/ModelA"
https://github.com/koxudaxi/datamodel-code-generator/blob/81595f2b73157cffb290431818a481dc8dcaf744/tests/data/jsonschema/multiple_files/file_a.json#L1-L13

ModelA is the value of title. I guess It is an invalid $ref.
Of course, "ModelA": {...} is valid $ref.
But, I wrote invalid $ref in this test-case. 😢

[aexvir]

@koxudaxi oh... I see
My bad then, sorry for the noise

The problem is that I'm still struggling with refs working properly, with completely valid definitions, but I'll make a new issue with some reproducible example, that way it should be easier to analyze 🙂️

Thanks a lot!

[koxudaxi]

@aexvir
Don’t worry about it😄

The problem is that I'm still struggling with refs working properly, with completely valid definitions, but I'll make a new issue with some reproducible example, that way it should be easier to analyze 🙂️

Thank you very much!!
I think examples and test-case support this project. 🚀

[koxudaxi]

I have released a new version 0.9.0. 🎉

@themmes
I believe the version can parse your new schema.

@aexvir
The new version is changed internally design.
I hope that the version can resolve all complex references.