Ksql: KSQL 5.2.1 UDF load failing

[stewartbryson]

When upgrading to CP 5.2.1, all UDF loads that previously worked in 5.1.2 are failing. So I went back to basics and compiled the multiply function from the Confluent documentation, and that failed as well.

Notes:
I am using gradle and the shadowJar plugin. This worked successfully in 5.1.2.

I have tested using both of the following dependency statements:
compile 'io.confluent.ksql:ksql-udf:5.2.1'
compile 'io.confluent.ksql:ksql-udf:5.1.2'
I wanted to see if the issue was only with the new dependency, or the new KSQL Server. It seems to be the new KSQL Server, as either dependency (5.1.2 or 5.2.1) fails with the same error.

The error I'm getting during the startup of the KSQL Server is below. I've also attached my uber JAR created by gradle. I had to zip it first for GitHub... but inside the ZIP is a JAR.

INFO Adding UDF name='multiply' from path=/etc/ksql-server/ext/functions-1.0.0.jar (io.confluent.ksql.function.UdfLoader:211)
WARN Failed to add UDF to the MetaStore. name=multiply method=public long Multiply.multiply(long,long) (io.confluent.ksql.function.UdfLoader:220)
io.confluent.ksql.util.KsqlException: Failed to compile udf wrapper class for public long Multiply.multiply(long,long)
at io.confluent.ksql.function.UdfCompiler.compile(UdfCompiler.java:96)
at io.confluent.ksql.function.UdfLoader.lambda$handleUdfAnnotation$8(UdfLoader.java:214)
at io.github.lukehutch.fastclasspathscanner.scanner.ScanSpec$9.lookForMatches(ScanSpec.java:1390)
at io.github.lukehutch.fastclasspathscanner.scanner.ScanSpec.callMatchProcessors(ScanSpec.java:696)
at io.github.lukehutch.fastclasspathscanner.FastClasspathScanner.scan(FastClasspathScanner.java:1606)
at io.github.lukehutch.fastclasspathscanner.FastClasspathScanner.scan(FastClasspathScanner.java:1678)
at io.github.lukehutch.fastclasspathscanner.FastClasspathScanner.scan(FastClasspathScanner.java:1704)
at io.confluent.ksql.function.UdfLoader.loadUdfs(UdfLoader.java:141)
at io.confluent.ksql.function.UdfLoader.lambda$load$2(UdfLoader.java:111)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at io.confluent.ksql.function.UdfLoader.load(UdfLoader.java:111)
at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:333)
at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:307)
at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:85)
at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:50)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList$SubList.rangeCheck(ArrayList.java:1225)
at java.util.ArrayList$SubList.get(ArrayList.java:1042)
at java.util.Collections$UnmodifiableList.get(Collections.java:1309)
at com.squareup.javapoet.ClassName.<init>(ClassName.java:52)
at com.squareup.javapoet.ClassName.<init>(ClassName.java:43)
at com.squareup.javapoet.ClassName.enclosingClassName(ClassName.java:76)
at com.squareup.javapoet.CodeWriter.lookupName(CodeWriter.java:358)
at com.squareup.javapoet.ClassName.emit(ClassName.java:237)
at com.squareup.javapoet.CodeWriter.emit(CodeWriter.java:253)
at com.squareup.javapoet.CodeBlock.toString(CodeBlock.java:95)
at io.confluent.ksql.function.UdfTemplate.generateCode(UdfTemplate.java:55)
at io.confluent.ksql.function.UdfCompiler.generateCode(UdfCompiler.java:251)
at io.confluent.ksql.function.UdfCompiler.compile(UdfCompiler.java:90)
... 25 more

functions-1.0.0.jar.zip

[apurvam]

@hjafarpour can you triage this as on call? Is it a real regression?

[apurvam]

@stewartbryson can you share the exact build setup you used which reproduces this?

[stewartbryson]

Here's the repo: https://github.com/RedPillAnalytics/kafka-examples/tree/confluent-blog

FYI... the link is to the "confluent-blog" branch... make sure you check that one out.

Build with: ./gradlew functions:build. The JAR will have the decode() function in it (which is what the Spock tests are for), but I have also put the multiply() function from the Confluent docs in there just to make sure it wasn't something to do with my code. The decode() function written in Groovy and the multiply() function written in Java both produce the same error.

This setup worked with 5.1.2. The only difference I made for 5.2.1 is the dependency. However... as I mentioned above, the behavior is the same regardless of which dependency I use. So then... the only real difference is the loading into the 5.2.1 environment versus the 5.1.2 environment.

Thanks.

[hjafarpour]

yes, this is a regression in 5.2 resulted from using new package for building udfs. I had a discussion with @agavra and he will be looking into this.

[agavra]

Nice! We've identified a bug in https://github.com/square/javapoet with regards to type resolution with empty packages. I'll submit a PR to fix this and then hopefully fix it in javapoet! Cheers 🍻

[agavra]

https://github.com/square/javapoet/pull/723 the corresponding fix in javapoet

[agavra]

The next release of 5.2.x will have the fix!