Swagger-codegen: OutOfMemoryError

[robert-mygind]

Description

When running codegen I get an OutOfMemoryError

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at io.swagger.codegen.examples.ExampleGenerator.resolvePropertyToExample(ExampleGenerator.java:160)
at io.swagger.codegen.examples.ExampleGenerator.resolveModelToExample(ExampleGenerator.java:251)
at io.swagger.codegen.examples.ExampleGenerator.resolvePropertyToExample(ExampleGenerator.java:210)
at io.swagger.codegen.examples.ExampleGenerator.resolvePropertyToExample(ExampleGenerator.java:161)
at io.swagger.codegen.examples.ExampleGenerator.generate(ExampleGenerator.java:49)
at io.swagger.codegen.DefaultCodegen.fromOperation(DefaultCodegen.java:2195)
at io.swagger.codegen.DefaultGenerator.processOperation(DefaultGenerator.java:864)
at io.swagger.codegen.DefaultGenerator.processPaths(DefaultGenerator.java:791)
at io.swagger.codegen.DefaultGenerator.generateApis(DefaultGenerator.java:418)
at io.swagger.codegen.DefaultGenerator.generate(DefaultGenerator.java:730)
at io.swagger.codegen.cmd.Generate.run(Generate.java:285)
at io.swagger.codegen.SwaggerCodegen.main(SwaggerCodegen.java:35)

Swagger-codegen version

Just checkout from github and build

Swagger declaration file content or url

https://api.reepay.com/swagger.json

Command line used for generation

java -jar modules/swagger-codegen-cli/target/swagger-codegen-cli.jar generate -i https://api.reepay.com/swagger.json -l php -o ~/Projects/reepay-php

Operation system and Java

OS X 10.11.6
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

[wing328]

I was able to repeat the issue with similar errors:

[main] INFO io.swagger.codegen.AbstractGenerator - writing file /tmp/Projects/reepay-php/SwaggerClient-php/docs/Model/WebhookUpdateRequest.md
Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
    at io.swagger.codegen.examples.ExampleGenerator.resolvePropertyToExample(ExampleGenerator.java:160)
    at io.swagger.codegen.examples.ExampleGenerator.resolveModelToExample(ExampleGenerator.java:251)
    at io.swagger.codegen.examples.ExampleGenerator.resolvePropertyToExample(ExampleGenerator.java:210)
    at io.swagger.codegen.examples.ExampleGenerator.generate(ExampleGenerator.java:49)
    at io.swagger.codegen.DefaultCodegen.fromOperation(DefaultCodegen.java:2188)
    at io.swagger.codegen.DefaultGenerator.processOperation(DefaultGenerator.java:864)
    at io.swagger.codegen.DefaultGenerator.processPaths(DefaultGenerator.java:791)
    at io.swagger.codegen.DefaultGenerator.generateApis(DefaultGenerator.java:418)
    at io.swagger.codegen.DefaultGenerator.generate(DefaultGenerator.java:730)
    at io.swagger.codegen.cmd.Generate.run(Generate.java:285)
    at io.swagger.codegen.SwaggerCodegen.main(SwaggerCodegen.java:35)

I'll take a deeper look tomorrow by investigating the spec.

[dariota]

@wing328 The issue is in #/definitions/CardGatewayAgreement/properties/card_types/. It defines maxItems as MAX_INT, which causes that issue here for obvious reasons.

I suspect a fix may be to simply cap the number of items in an example array - there's probably not much value in a 2 billion item example.

[aiham]

I've hit this with the javascript language. Had to revert to v2.2.2 because v2.2.3 is broken because of #5845

[wing328]

@dariota thanks for pointing out the exact root cause of the issue. May I know if you've time to contribute a PR with the fix?

[dariota]

Not until mid January, but I can take it up then.

[xmeng1]

typescript-angular same issue

[atbe]

Python codegen same issue

Affected versions:

• 2.3.1
• 2.3.0

[xmeng1]

according to my test (MacOS 10.13.3)
if I assign more than 8G for VM (such as -Xmx9g, -Xmx8200m, less than 8200m may lead to Exception in thread "main" java.lang.OutOfMemoryError: Java heap space, strange), the max of the object array can be 2147483645 (Integer.MAX_VALUE - 2). but I think setting the size of the object in the example too high is not necessary. Add the code as follow before this line can solve the issue. However, I am not sure is it better to limit the Object array size here, or there will be other issues occur. @wing328 What do you think?

if(arrayLength>10000) {
    arrayLength=10000;
}

The simple code to test the Obejct array size limit:

public class ObjectArraySizeTestHighXmx {
    public static void main(String args[]) {
        Integer arrayLength = Integer.MAX_VALUE;
        Object[] objectProperties = null;

        // objectProperties = new Object[2396729];
        // System.out.print(objectProperties.length);
        // step = 1
        for (int i=arrayLength; i>1;i=i-1) {
            try {
                objectProperties = new Object[i];
            } catch (OutOfMemoryError e) {
                System.out.print("Out of memory: " + String.valueOf(i) + "\n");
                // System.out.print(e.toString());
            }
            if (objectProperties != null) {
                System.out.print("Find the limit by step 1: " + String.valueOf(i) + "\n");
                System.out.print(objectProperties.length + "\n");
                break;
            }
        }
    }
}

run

javac ObjectArraySizeTestHighXmx.java
java -Xmx9g ObjectArraySizeTestHighXmx  > ObjectArraySizeTestHighXmx-xmx9g.log

the log file will be

Out of memory: 2147483647
Out of memory: 2147483646
Find the limit by 1: 2147483645
2147483645

[dariota]

I don't see why we shouldn't limit the size of the example - what benefit would you get from looking at a 2 billion item example array (with just generated default data in it) that you wouldn't get from a 10,000 (or even 100) item array?

Of course, it would be nice to replace this in-memory creation of a massive array with a lazily generated array that's written as it's produced and massively reduces the memory footprint, but you'd still have an absolutely gigantic file for little to no value (bonus points when you try to view it and crash your browser).

[xmeng1]

@dariota Yes, I also think it is not necessary to use large size of example array here. But do you think is it better to hardcode a size in here? or is there any mechanism to define this size?

[dariota]

For maximal configurability it should be something the user can set (with a sensible default), but at the same time I'm not a big fan of excessive numbers of options overwhelming users.

As a short-term fix, hardcoding a size will let the people having this issue actually use the specs they're trying to use. I'd argue for inclusion of a warning when the number of examples is reduced (or noting the max more directly in the generated output when the count's reduced).

Longer term, pending determination of whether a config option to set the max number of items in the example should be added, a proper fix should be done, perhaps replacing the generation of an array with something like an Iterator that generates the output on the fly when the file's being written, to enable these huge examples should people want them (and choose to do so via the option) for some strange reason.
In fact, if a replacement of the array with an Iterator is possible without huge amounts of modification right now, that plus a hardcoded limit would be the best approach in my eyes.

[wing328]

if(arrayLength>10000) {
arrayLength=10000;
}

Please file a PR with the suggested fix.

My first suggestion is to use LOGGER.warn to show a warning message so that users will be notified of the issue.

[xmeng1]

ok, I will submit a quick PR for this then we can discuss it further.

[aiham]

Turns out the spec I was using (which was auto generated with swagger-jersey2-jaxrs) has "maxItems": 2147483647 in it. This is because the API implementation uses java.util.List which swagger-jersey2-jaxrs determines the maximum size as Integer.MAX_VALUE, and ExampleGenerator.java uses that size to generate the example array:

int arrayLength = null == ((ArrayProperty) property).getMaxItems() ? 2 : ((ArrayProperty) property).getMaxItems();
Object[] objectProperties = new Object[arrayLength];

[konths]

Any update on this issue?
I am still facing the problem.

[aiham]

I deleted maxItems from the spec if it was equal to Integer.MAX_VALUE. it's not a useful constraint in that case, neither to the generated code or any code relying on the spec to validate requests.

[cbornet]

I think the array length should be computed as a value between minItems and maxItems as close as possible from a magic number (2).

[robert-mygind]

Any update to this ?

[xmeng1]

@robert-mygind you can remove or reduce the number of "maximum" : 10000000, in your JSON spec currently to solve this issue.

[grandeemme]

it's possible to exclude the example generator?

[dariota]

@wing328 could you take at #7625 when you get a chance?

[robert-mygind]

@robert-mygind you can remove or reduce the number of "maximum" : 10000000, in your JSON spec currently to solve this issue.

I removed "maximum" : 10000000 and also "maxItems" : 2147483647 so now it works.

[frantuma]

fixed in #9553