Datamodel-code-generator: testing with https://www.schemastore.org/json/

[tstorek]

Hello,

thanks again for this piece of software. I have a little project for that I would like to use the code generator.
The basic idea is the following:

1. parse existing data models from https://www.schemastore.org/json/ or ontologies in the form of *.ttl or *.owl to create pydantic models.
   
   
   
   1. create instances of my models from a large dictionary of data
   
   
   2. validate the data via pydantic, at least with the build-in validators
   
   
   3. store the data in a mongodb based RestAPI, using the pydantic serialization mechanisms.
   
   

As a first step, I was playing around with the data models from https://www.schemastore.org/json expecting them to work out of the box.
However, I faced several issues with this:

1. In most cases the schemas provide titles that cannot directly be parsed as ClassName. This is solvable by creating my own ClassName. By somehow pre-parsing the file and generate my own Name and set it to the parser using the data model generator with CLI. However, in most cases the "Top-Layer" Model does not even be created because one might even be more interested in the underlying submodels the schema consists of.
2. Furthermore, I experienced that several types are not parsed correctly. Maybe it would be good to use 'Any' as default type instead of simply failing.

Suggestion to solve:

• Writing a test that parses a randomized subset of https://www.schemastore.org/json. This would at least provide sophisticated testing data schemas to test for integrity.

Thanks for your help

Cheers

[koxudaxi]

@tstorek
Thank you for creating this issue.
I feel great suggestion.

In most cases the schemas provide titles that cannot directly be parsed as ClassName. This is solvable by creating my own ClassName. By somehow pre-parsing the file and generate my own Name and set it to the parser using the data model generator with CLI. However, in most cases the "Top-Layer" Model does not even be created because one might even be more interested in the underlying submodels the schema consists of.

I agree. But, I don't know top-layer is not needed on users. They may need it.
And, The code generator can set converted class-name. But, the converted name may not best name. So, I think users should set names for models.

Furthermore, I experienced that several types are not parsed correctly. Maybe it would be good to use 'Any' as default type instead of simply failing.

Would you please tell me the schemas?
I want to fix the code for it.

Writing a test that parses a randomized subset of https://www.schemastore.org/json. This would at least provide sophisticated testing data schemas to test for integrity.

I will try it 😄

[tstorek]

@koxudaxi
Thanks, for the response. I agree that top-layer needs to persist, but it may be not of greatest interest. Nevertheless, it would be great to define a callback for generating ClassNames for referred schemas. Hence, it would be possible to parse linked schemas.

In fact, I tried to parse the schema from here: https://github.com/FIWARE/data-models/tree/master/specs/Building
that I do not maintain myself. The problem is that the schema is referenced to https://fiware.github.io/data-models/common-schema.json#/definitions/GSMA-Commons which I failed to parse. I think it was a combination of classname parsing and other errors.

[koxudaxi]

, it would be great to define a callback for generating ClassNames for referred schemas. Hence, it would be possible to parse linked schemas.

I'm sorry, What is callback ? Do you mean set className converted name from the title?

Is the error this?

$ datamodel-codegen --url 'https://fiware.github.io/data-models/common-schema.json#/definitions/GSMA-Commons' --class-name Model --output /tmp/models
Parse $id failed. $id=https://geojson.org/schema/Geometry.json
 mapping values are not allowed in this context
  in "<unicode string>", line 9, column 25

[tstorek]

For the callback: https://en.wikipedia.org/wiki/Callback_(computer_programming)

Or short you pass a function as an argument to another function or class. Afterward, this function is able to execute the function if it can understand its signature. My idea would be to pass a function that receives the string as an argument, parses it for instance to a python conform classname with something like removing all spaces and make next word Uppercase. The format can be then freely defined by the user, whereas a default callback could be reasonable. Finally, the callback returns this processed string back to the original function. In this case the 'generate' function. All this happens within 'generate'.

Basically this would look like:
```python:

from typing import Callable
import regex as re

def callback(title: str):
# remove special characters with replace with spaces using regualar expression
classname = re.sub('[^A-Za-z0-9]+', ' ', title)
# Make string CamelCase and join for pep8
classname = ''.join(x for x in classname.title() if not x.isspace())
return classname

def generate(
class_name: str = None,
custom_classname_gen: Callable = None):
title = 'title_from_schema'
if class_name:
class_name = class_name
elif custom_classname_gen:
class_name = custom_classname_gen(title)
else:
class_name = title

create_class(name=class_name)

For my parsing problem I used this code:

```python:

import os
import shutil
import requests
from pathlib import Path
from tempfile import TemporaryDirectory
from datamodel_code_generator import InputFileType, generate

if __name__ == '__main__':
    path = Path(os.getcwd()).joinpath("../datamodels")
    path.mkdir(parents=True, exist_ok=True)
    with TemporaryDirectory() as temp:
        temp = Path(temp)
        #url = 'https://smart-data-models.github.io/dataModel.Building/Building/schema.json'
        url = 'https://smart-data-models.github.io/data-models/common-schema.json#/definitions/GSMA-Commons'
        r = requests.get(url)
        with open(temp.joinpath('schema.json'), 'wb') as f:
            f.write(r.content)
        output = Path(temp).joinpath('model.py')
        input = temp.joinpath('schema.json')
        generate(
            input_=input,
            input_file_type=InputFileType.JsonSchema,
            output=output,
            #class_name='SmartDataModels_Building'
            class_name='SmartDataModels_GSMA'

        )
        #filename = path.joinpath('smart_data_models_building.py')
        filename = path.joinpath('smart_data_models_gsma.py')
        shutil.move(str(output), str(filename))

here is my error:

Traceback (most recent call last):
  File "D:/Projects/N5GEH/n5geh.tools.FiLiP/filip/utils/smartdatamodels_generator.py", line 31, in <module>
    generate(
  File "D:\Python\anaconda3\envs\Fiware\lib\site-packages\datamodel_code_generator\__init__.py", line 213, in generate
    results = parser.parse()
  File "D:\Python\anaconda3\envs\Fiware\lib\site-packages\datamodel_code_generator\parser\base.py", line 253, in parse
    self.parse_raw()
  File "D:\Python\anaconda3\envs\Fiware\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 847, in parse_raw
    self._parse_file(self.raw_obj, obj_name, path_parts, path_parts)
  File "D:\Python\anaconda3\envs\Fiware\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 873, in _parse_file
    self.parse_raw_obj(key, model, [*path_parts, '#/definitions', key])
  File "D:\Python\anaconda3\envs\Fiware\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 810, in parse_raw_obj
    self.parse_obj(name, JsonSchemaObject.parse_obj(raw), path)
  File "D:\Python\anaconda3\envs\Fiware\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 823, in parse_obj
    self.parse_root_type(name, obj, path)
  File "D:\Python\anaconda3\envs\Fiware\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 601, in parse_root_type
    data_type: DataType = self.get_data_type(obj)
  File "D:\Python\anaconda3\envs\Fiware\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 266, in get_data_type
    return _get_data_type(obj.type, obj.format or 'default')
  File "D:\Python\anaconda3\envs\Fiware\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 255, in _get_data_type
    json_schema_data_formats[type_][format__],
KeyError: 'idn-email'

[koxudaxi]

@tstorek
Thank you for explaining the idea.
I want to implement the approach. Also, I will prepre camelCaseGenerator as default custom_classname_gen

For my parsing problem I used this code:

What version of the datamodel-code-generator did you use?
I recommend using the latest version 0.9.4

I could generate models by version 0.9.4. Of course, I copied your code.
But, I got some warning. I will support the formats.

```
/Users/koudai/PycharmProjects/pythonProject10/venv/lib/python3.9/site-packages/datamodel_code_generator/parser/jsonschema.py:340: UserWarning: format of 'idn-email' not understood for 'string' - using default
warn(
Parse $id failed. $id=https://smart-data-models.github.io/data-models/common-schema.json
mapping values are not allowed in this context
in "", line 9, column 25
/Users/koudai/PycharmProjects/pythonProject10/venv/lib/python3.9/site-packages/datamodel_code_generator/parser/jsonschema.py:340: UserWarning: format of 'time' not understood for 'string' - using default
warn(
````

[tstorek]

@koxudaxi
I used 0.6.9 for the error above.
After updating to the newest version I receive the same warning as you.
Now I can generate models as well, yeah! 👍 However, I am surprised to find for instance Email and Email1 as models. This is also the case for some of the other models. The definitions of both models seems to be the same.

Also for the geometries the Enum for the choices is split up into multiple classes. Isn't there a way to automatically merge them instead of generating a new class? I mean the Model should be still GeometryItem. Why isn't the model reused? I mean the reference is both times the same. Would it be reasonable to cache the ref-models first until all recursive loops are finished and then generate the models afterwards? I think that would also make parsing far more efficient, doesn't it?

[koxudaxi]

@tstorek
Great!!

However, I am surprised to find for instance Email and Email1 as models. This is also the case for some of the other models. The definitions of both models seems to be the same.

umm...
It's a troublesome problem...
$id may confuse code generator.
There is a workaround. You can generate models without duplicate Email objects when getting schema from URL directly

  from urllib import parse
  generate(
      input_=parse.urlparse('https://smart-data-models.github.io/data-models/common-schema.json'),
      input_file_type=InputFileType.JsonSchema,
      output=output,
      #class_name='SmartDataModels_Building'
      class_name='SmartDataModels_GSMA',
  )

I will investigate the duplicate problem.

[tstorek]

Thank you! I think we should keep this open until the other tests are implemented, right?

[koxudaxi]

Yes, I fixed the problem in #394. But, I haven't released it. I will do it this weekend
I will close the issue after I release it.
Thank you.

[koxudaxi]

@tstorek
I pushed a fixed version.
You can generate models from the downloaded file.

If the title is not a valid Python class name then the parser creates a camel class name automatically.
Also, The version supports a custom class name generator
https://github.com/koxudaxi/datamodel-code-generator/blob/939b8d7dac892afc84645ac2c9ed2270c6309799/tests/test_main.py#L2163-L2174

[rushabh-v]

Hi @koxudaxi,

First of all thanks for the amazing tool. I am getting the following error,

File "/home/rushabh/anaconda3/lib/python3.7/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "yaml/_yaml.pyx", line 707, in yaml._yaml.CParser.get_single_node
  File "yaml/_yaml.pyx", line 726, in yaml._yaml.CParser._compose_document
  File "yaml/_yaml.pyx", line 905, in yaml._yaml.CParser._parse_next_event
yaml.scanner.ScannerError: mapping values are not allowed in this context
  in "<unicode string>", line 16, column 14

You have mentioned this error twice in this thread. Do you know the potential cause?

Thanks!

[koxudaxi]

@rushabh-v
Thank you for your reports!!
I guess pyyaml failed to parse your schema.
Would you please share the schema?

[rushabh-v]

Hi, thanks for your response. I found my mistake and got it working now. But I think there is a problem with the inheritance of the schemas mentioned in allOf.

I believe it's out of this issue's scope, so I have opened another one at #399. Please take a look :)

Thanks!

[tstorek]

@koxudaxi thanks, for fixing this. One last thing : Can I also enforce classname generation. That would interesting to keep consistency in codestyle. Since snake_would be also a valid python class name but not pep8.

[koxudaxi]

@tstorek

Can I also enforce classname generation.

Do you mean a top-level model? Or, all generated models?
Top-level model name is generated by title_to_class_name()
https://github.com/koxudaxi/datamodel-code-generator/blob/f9c2179bedd805ec1e1b7e405adbc4debecbe31c/datamodel_code_generator/parser/jsonschema.py#L1057-L1062

Added:
I think we should not do enforce it.
Because The behavior will be changed from the previous versions.

[tstorek]

@koxudaxi well, i do not mean to implement it as default behavior rather then just havin the option setting classname_generator as behavior. I mean for all classnames.

[koxudaxi]

@tstorek
I think your suggestion is good.
OK, I will implement that the code-generator applies custom_class_name_generator to all class names.
Thank you.

[koxudaxi]

@tstorek
I have released a new version 0.10.1 that applies custom_class_name_generator on all class names.