Crystal: Heredoc with more than 300 lines fails to compile on Crystal 0.32.0

[lbguilherme]

The recent Crystal 0.32.0 release includes the new String.interpolation as an alternative codegen for what used to be based on String.build (see #8400).

This is causing issues for me on heredocs with interpolation:

bug = <<-END
x
#{2}
x
x
... 300 more lines with just "x"...
x
x
END

Fails to compile with:

In /tmp/bug.cr:1:5

 1 | bug = <<-END
         ^------------
Error: instantiating 'String.class#interpolation(String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String, String)'


In /usr/share/crystal/src/string.cr:4536:27

 4536 | def self.interpolation(*values : String)
                                ^
Error: tuple size cannot be greater than 300 (size is 301)

I noticed from reading the parser that heredocs are represented as a StringInterpolation nodes with each line as a individual string in this case. I believe there should be an AST transformation on top of StringInterpolation that will merge any two successive string literals into one, avoiding the issue if I'm just writing a long heredoc. Is this the correct approach to fix this?

[straight-shoota]

AST transformation would be an improvement. But it's not a fix. With the instight on tuple size limitation, we need to reconsider the entire String.interpolation idea. String interpolation literals should not be limited to 300 parts (i.e. 150 interpolations with fixed strings in between). It might seem like a large number for this, but it's not really. Especially considering that the source code can be generated.
I'm sure we should avoid such a arbitrary limitation on string interpolation, which is essentially a regression introduced by changing the internal implementation. We need to find a way to fix this or return to pre-String.interpolation.

[lbguilherme]

Maybe we could keep String.interpolation but fallback to the old implementation if there are more than X elements? This allows optimizing the more common edge cases, but still works on all cases.

[bcardiff]

@lbguilherme did you use those huge heredocs and discover the issue upon release, or did you actually built it to stress the limit?

[lbguilherme]

@bcardiff I use then for code generation. Here is one of the affected files: https://github.com/cubos/sdkgen/blob/master/src/target/java_android.cr

[j8r]

This 300 limitation is due to the Tuple limitation of max 300 elements 🤔

[asterite]

I think we should merge string literals in interpolations, I consider this a bug. I wouldn't worry about the 300 interpolated elements just now.

[asterite]

Also regarding the 300 tuple limitation, we should probably do that and also check for that stuff in recursive methods or contexts. Hard, but I think doable.