Datamodel-code-generator: json_schema_data_formats fail when looking up 'array'

[javierabosch]

When trying to convert file data.json to model.py, I get a
keyerror on line 171 of jsonschema.py when the type in types is 'array'.

To Reproduce

Example schema:
```json

[javierabosch]

When I amended the json_schema_data_formats and added:

'array': {'default': Types.array}

and added Types.array: DataType(type='list'), to typ_map in types.py

this fixed the issue.

I am not sure if this fixes any corner cases, but I do believe json 'array' maps to a list.

[koxudaxi]

@javierabosch
Thank you for creating this issue.

and added Types.array: DataType(type='list'), to typ_map in types.py

This approach is not bad, However, Python List needs sub types like List[str]
The generator use a method which can parse array object to resolve subtypes of List.
https://github.com/koxudaxi/datamodel-code-generator/blob/01e9e58e92e44c385133b80c01caa475548929cd/datamodel_code_generator/parser/jsonschema.py#L201

I would use the method to parse your array object. I don't know why It was not used in your case.
Would you please put the json file?
I want to resolve this problem.

[javierabosch]

Hi @koxudaxi ,

The json file is quite large. Is there anyway I can email it to you?

I saw the function you referenced in the code, but I was just monkey patching as I went along, hoping down the pipeline it would use that function.

I sent the json file to your email.

[koxudaxi]

@javierabosch
Thank you for your email.
I have fixed a problem which is a nested array.
I released a new version as 0.5.11

But, I found other problems to generate models from your json data.

1. Need calling update_forward_refs for MediaItem after all dependency models are defined.
   I will fix the problem when I refactor a function referencing class which is related to the issue https://github.com/koxudaxi/datamodel-code-generator/issues/145
   It means a feature is broken which is referencing class.

2. change types of some field to Optional
   the code generator uses genson to convert Json data to JSON schema. the problem happens is in genson.
   You should change types to Optional manually after you generate models.

[koxudaxi]

@javierabosch

1. Need calling update_forward_refs for MediaItem after all dependency models are defined.

Sorry, I mistake. It's no problem.

I believe this issue was resolved. Can I close this issue?

[JosXa]

Hi, I'm on 0.5.34 and the KeyError: 'array' issue still appears. I was able to track it down to the following case:

{
  "my_obj": [
    {
      "items": null
    },
    {
      "items": []
    }
  ]
}

It doesn't matter whether the null or the [] is defined first.

The schema generated from this is:

{
    "$schema": "http://json-schema.org/schema#",
    "type": "object",
    "properties": {
        "my_obj": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "items": {
                        "type": [
                            "array",
                            "null"
                        ]
                    }
                },
                "required": [
                    "items"
                ]
            }
        }
    },
    "required": [
        "my_obj"
    ]
}

Traceback:

Traceback (most recent call last):

  File "C:\projects\josxabot\app\modules\josxa_private\pydantic_model_gen\directory_watcher.py", line 92, in run_conversion
    generate(
    └ <function generate at 0x00000251AF436CA0>

  File "C:\Users\INT002327\AppData\Local\pypoetry\Cache\virtualenvs\josxabot-c3BmTbt9-py3.8\lib\site-packages\datamodel_code_generator\__init__.py", line 185, in generate
    result = parser.parse()
             │      └ <function Parser.parse at 0x00000251AF436A60>
             └ <datamodel_code_generator.parser.jsonschema.JsonSchemaParser object at 0x00000251AF20E6A0>

  File "C:\Users\INT002327\AppData\Local\pypoetry\Cache\virtualenvs\josxabot-c3BmTbt9-py3.8\lib\site-packages\datamodel_code_generator\parser\base.py", line 351, in parse
    self.parse_raw()
    │    └ <function JsonSchemaParser.parse_raw at 0x00000251B1703430>
    └ <datamodel_code_generator.parser.jsonschema.JsonSchemaParser object at 0x00000251AF20E6A0>

  File "C:\Users\INT002327\AppData\Local\pypoetry\Cache\virtualenvs\josxabot-c3BmTbt9-py3.8\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 687, in parse_raw
    self.parse_raw_obj(obj_name, raw_obj, [])
    │    │             │         └ {'$schema': 'http://json-schema.org/schema#', 'type': 'object', 'properties': {'my_obj': {'type': 'array', 'items': {'type': ...
    │    │             └ 'Model'
    │    └ <function JsonSchemaParser.parse_raw_obj at 0x00000251B17033A0>
    └ <datamodel_code_generator.parser.jsonschema.JsonSchemaParser object at 0x00000251AF20E6A0>

  File "C:\Users\INT002327\AppData\Local\pypoetry\Cache\virtualenvs\josxabot-c3BmTbt9-py3.8\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 671, in parse_raw_obj
    self.parse_object(name, obj, path)
    │    │            │     │    └ []
    │    │            │     └ JsonSchemaObject(items=None, uniqueItem=None, type='object', format=None, pattern=None, minLength=None, maxLength=None, minim...
    │    │            └ 'Model'
    │    └ <function JsonSchemaParser.parse_object at 0x00000251B1703040>
    └ <datamodel_code_generator.parser.jsonschema.JsonSchemaParser object at 0x00000251AF20E6A0>

  File "C:\Users\INT002327\AppData\Local\pypoetry\Cache\virtualenvs\josxabot-c3BmTbt9-py3.8\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 424, in parse_object
    fields = self.parse_object_fields(obj, path)
             │    │                   │    └ []
             │    │                   └ JsonSchemaObject(items=None, uniqueItem=None, type='object', format=None, pattern=None, minLength=None, maxLength=None, minim...
             │    └ <function JsonSchemaParser.parse_object_fields at 0x00000251B16E0F70>
             └ <datamodel_code_generator.parser.jsonschema.JsonSchemaParser object at 0x00000251AF20E6A0>

  File "C:\Users\INT002327\AppData\Local\pypoetry\Cache\virtualenvs\josxabot-c3BmTbt9-py3.8\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 331, in parse_object_fields
    array_field, array_field_classes = self.parse_array_fields(
                                       │    └ <function JsonSchemaParser.parse_array_fields at 0x00000251B17030D0>
                                       └ <datamodel_code_generator.parser.jsonschema.JsonSchemaParser object at 0x00000251AF20E6A0>

  File "C:\Users\INT002327\AppData\Local\pypoetry\Cache\virtualenvs\josxabot-c3BmTbt9-py3.8\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 458, in parse_array_fields
    type=self.parse_object(
         │    └ <function JsonSchemaParser.parse_object at 0x00000251B1703040>
         └ <datamodel_code_generator.parser.jsonschema.JsonSchemaParser object at 0x00000251AF20E6A0>

  File "C:\Users\INT002327\AppData\Local\pypoetry\Cache\virtualenvs\josxabot-c3BmTbt9-py3.8\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 424, in parse_object
    fields = self.parse_object_fields(obj, path)
             │    │                   │    └ ['my_obj']
             │    │                   └ JsonSchemaObject(items=None, uniqueItem=None, type='object', format=None, pattern=None, minLength=None, maxLength=None, minim...
             │    └ <function JsonSchemaParser.parse_object_fields at 0x00000251B16E0F70>
             └ <datamodel_code_generator.parser.jsonschema.JsonSchemaParser object at 0x00000251AF20E6A0>

  File "C:\Users\INT002327\AppData\Local\pypoetry\Cache\virtualenvs\josxabot-c3BmTbt9-py3.8\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 391, in parse_object_fields
    field_types = self.get_data_type(field)
                  │    │             └ JsonSchemaObject(items=None, uniqueItem=None, type=['array', 'null'], format=None, pattern=None, minLength=None, maxLength=No...
                  │    └ <function JsonSchemaParser.get_data_type at 0x00000251B169AF70>
                  └ <datamodel_code_generator.parser.jsonschema.JsonSchemaParser object at 0x00000251AF20E6A0>

  File "C:\Users\INT002327\AppData\Local\pypoetry\Cache\virtualenvs\josxabot-c3BmTbt9-py3.8\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 190, in get_data_type
    return [

  File "C:\Users\INT002327\AppData\Local\pypoetry\Cache\virtualenvs\josxabot-c3BmTbt9-py3.8\lib\site-packages\datamodel_code_generator\parser\jsonschema.py", line 192, in <listcomp>
    json_schema_data_formats[t][format_],
    │                        │  └ 'default'
    │                        └ 'array'
    └ {'integer': {'int32': <Types.int32: 2>, 'int64': <Types.int64: 3>, 'default': <Types.integer: 1>}, 'number': {'float': <Types...

KeyError: 'array'

[JosXa]

@koxudaxi would you like me to create a fresh issue for this?

[JosXa]

My super hacky workaround:

def normalize_null_arrays(obj: Any):
    """ Recursively replace "null" dict values with empty lists if another item in the same list contains an empty [] """
    if isinstance(obj, dict):
        for val in obj.values():
            normalize_null_arrays(val)
        return

    if not isinstance(obj, list):
        return

    key_has_null_map = dict()
    key_has_array_map = dict()

    for i in obj:
        if not isinstance(i, dict):
            break

        for k, v in i.items():
            if v is None:
                key_has_null_map[k] = True
            if v == []:
                key_has_array_map[k] = True

    for i in obj:
        if not isinstance(i, dict):
            break

        for k, v in i.items():
            if k in key_has_array_map and k in key_has_null_map and v is None:
                i[k] = []

normalize_null_arrays(some_json_data)

[koxudaxi]

@JosXa
Thank you for reporting the problem.

would you like me to create a fresh issue for this?

Yes, please.

I will check the problem tomorrow.

[JosXa]

Alright, will do 👍