Datamodel-code-generator: Calling datamodel-code-generator programatically

[HuiiBuh]

Background Information

I would like to use the datamodel-code-generator in a program which automates the extraction of data-models of an existing API documentation. Unfortunately the API documentation is not in a JSON-format, but a simple Website.
So the plan is to parse the Website and create a JSON Schema with the information I gained.

Actual Feature Request

I plan to generate the Pydantic Models with the extracted JSON schema. The question is if it is possible to use datamodel-code-generator programmatically.

# I generate the json schema
json_schema: str = extract_from_website()

# And pass it to the datamodel-code-generator which in turn generates the pydantic_models
pydantic_models: str = generate(json_schema, ...some_args)
               # or whatever fits best

I had a quick look at the code, but the generate function does not look like it is intended to be used that way. So would it be possible to modify it in a way so you can use the library from python and not just as a command line tool?

[koxudaxi]

@HuiiBuh
Thank you for creating this issue.

This function can generate model code from json schema.

https://github.com/koxudaxi/datamodel-code-generator/blob/e91ae4c6dc3fd6a1dfb561e1def2e681ec9ac2be/datamodel_code_generator/__init__.py#L132
But, the function doesn't return a string. We should change the output argument to get models as strings.
There is a workaround that you can output to a temporary directory with
this function https://docs.python.org/3/library/tempfile.html#tempfile.TemporaryDirectory
And, you can read model code from a generated file in the directory.

[HuiiBuh]

Thanks for the reply.
I have seen this method, but did not not how to use it exactly.

Perhaps a docstring, which describes how to use it and a bit of documentation would be nice.
I think this would not only be use useful for my use case, but also for potential contributors which have a good starting point.

At the moment I can use datamodel-code-generator by just executing it with the subprocess module and afterwards reading the created file, but it would be nice to use it like a normal python library.

[koxudaxi]

@HuiiBuh
OK, I write an example to generate models

from pathlib import Path
from tempfile import TemporaryDirectory
from datamodel_code_generator import InputFileType, generate

json_schema: str = """{
    "type": "object",
    "properties": {
        "number": {"type": "number"},
        "street_name": {"type": "string"},
        "street_type": {"type": "string",
                        "enum": ["Street", "Avenue", "Boulevard"]
                        }
    }
}"""

with TemporaryDirectory() as temporary_directory_name:
    temporary_directory = Path(temporary_directory_name)
    output = Path(temporary_directory / 'model.py')
    generate(
        json_schema,
        input_file_type=InputFileType.JsonSchema,
        input_filename="example.json",
        output=output,
    )
    model: str = output.read_text()
print(model)

the result of print(model)

# generated by datamodel-codegen:
#   filename:  example.json
#   timestamp: 2020-12-21T08:01:06+00:00

from __future__ import annotations

from enum import Enum
from typing import Optional

from pydantic import BaseModel


class StreetType(Enum):
    Street = 'Street'
    Avenue = 'Avenue'
    Boulevard = 'Boulevard'


class Model(BaseModel):
    number: Optional[float] = None
    street_name: Optional[str] = None
    street_type: Optional[StreetType] = None

Perhaps a docstring, which describes how to use it and a bit of documentation would be nice.
I think this would not only be use useful for my use case, but also for potential contributors which have a good starting point.

I agree.
I want to add examples to documents.

[HuiiBuh]

Thanks for the quick reply.
That looks really great and does not feel to much like a workaround.

And thanks for the your lib which really saves me a lot of time

[koxudaxi]

I just add a page for using datamodel-code-generator as a module.
https://koxudaxi.github.io/datamodel-code-generator/using_as_module/

[HuiiBuh]

Thanks that looks great

[hardbyte]

Thanks for adding this documentation! I wanted to simply get the pydantic class back to use for validation so I ended up loading the python code from the temporary file like this:

def pydantic_model_from_json_schema(json_schema: str,
                                    class_name='MyPydanticModel', **generate_kwargs):
    """ Generate a Pydantic Model from a loaded json schema.

    :param json_schema: Source json schema to create Pydantic model from.
    :param class_name: The pydantic class name, e.g., "Item".
    :param generate_kwargs: Any extra arguments to pass to datamodel_code_generator.generate
    :return: the newly created and loaded pydantic class
    """
    # Ref: https://github.com/koxudaxi/datamodel-code-generator/issues/278
    with TemporaryDirectory() as temporary_directory_name:
        temporary_directory = Path(temporary_directory_name)
        output = Path(temporary_directory / 'model.py')
        generate(
            json_schema,
            input_file_type=InputFileType.JsonSchema,
            input_filename="spec.json",
            class_name=class_name,
            output=output,
            **generate_kwargs
        )
        module_name = "models"
        module_path = str(output)
        spec = importlib.util.spec_from_file_location(module_name, module_path)
        module = importlib.util.module_from_spec(spec)
        sys.modules[spec.name] = module
        spec.loader.exec_module(module)
        return getattr(module, class_name)

Usage like this:

json_schema: str = '''{
    "type": "object",
    "properties": {
        "number": {"type": "number"},
        "street_name": {"type": "string"},
        "street_type": {"type": "string",
                        "enum": ["Street", "Avenue", "Boulevard"]
                        }
        }
    }'''

Model = pydantic_model_from_json_schema(json_schema)
print(Model)
print(Model(number=None, street_name="test", street_type=None))

# raises validation error:
#print(Model(street_name=['not a valid string']))

[koxudaxi]

@hardbyte
Interesting!!

I want to add your example to the documents.
Also, I think this function is beneficial 😄
Because some users hope to create and import generated models dynamically.
https://github.com/koxudaxi/datamodel-code-generator/issues/195#issuecomment-689357216
The function doesn't cover all cases like modular models. But, It is a good start point 🚀

Thank you very much.