Graal: Graal 19.1.1 with Scala and Play Framework: java.lang.IllegalStateException: Must use either different key or iv for GCM encryption

[henricook]

Hi all,

Unfortunately this bug is really hard for me to recreate, so i've included information that i'm using Scala, SBT and Play Framework.

During day-to-day use when i'm using Graal 19.1.1 with sbt run and Play Framework in Dev Mode, and play's using Netty instead of akka-http - I get this error. I have to completely clean and rebuild to make things keep working. It often recurs.

I see it whenever a client tries to hit my Play backend to do something over HTTPS - it completely errors out the communication

This only happens with Graal after 19.0.0. Graal 19.0.0 works fine.

I'm hoping someone will see this and say "oh it's connected to this crypto issue" - obviously the setup i've described is super specific and hard to recreate so apologies for that

[thomaswue]

We have not seen this before. So we might have to indeed try to recreate/reproduce the setup. Are you running with CE or EE? Can you run with -XX:TieredStopAtLevel=1 (effectively disabling the GraalVM compiler) and see if it still occurs? Maybe also just -XX:-UseJVMCICompiler flag instead to check whether it is an issue specific to the JDK.

[henricook]

Thanks for this @thomaswue, I'll give this a try next week and maybe have a crack at a smaller test case

[henricook]

I'm using GraalEE. I still see it regularly on my main project. I've been unable to recreate it on a vanilla (playframework/play-scala-seed) project, with netty and https enabled and with me just curl'ing the default endpoint over HTTPS _ad infinitum_ at random intervals

I'll wait for it to recur on my main project and try -XX:TieredStopAtLevel=1 as suggested to see if it helps

[henricook]

So far on my 'main project' i've confirmed that I get the issue, stop sbt shell, start sbt shell, and the issue recurs very quickly (stopped, started and saw again 5 times). When i stop the sbt shell and restart it with -D-XX:TieredStopAtLevel=1 - the issue no longer occurs.

Without the flag i can also stop the issue for a while by cleaning but it comes back

[thomaswue]

OK, thanks for testing. This could be quite a serious issue on our side. What is the processor config? Is the issue reliable enough that you could help us by testing a few more flag combinations?

[henricook]

I wish i could recreate it more reliably. Happy to test multiple flag configs, next time I see it I plan on giving -XX:-UseJVMCICompiler a try - please feel free to suggest any more.

I hope to soon be in a position to bring this code home with me as well, which will mean I can test the issue more thoroughly outside of client on-site time.

Hopefully i'm not ignorant when I say processor config is a quad core w/hyperthreading? :-D I feel like that might not be the question you're asking

Thinking about it we do have a truck load of custom scala compiler flags, i'm about to add these to these to my sample project

[thomaswue]

Testing whether one of these two flags helps, would be interesting:
-Dgraal.Vectorization=false
-Dgraal.OptDuplication=false

My main interest regarding the processor is whether it has AVX and/or AVX2 support. For this we would need to know the exact processor version. On a Mac, this can be obtained for example via "sysctl -n machdep.cpu.brand_string".

[henricook]

Thanks, i'll take a look once on-site. When i manage to bring the code home it'll also be interesting to see if the issue occurs for me on my home machine, which is slightly more bleeding edge than the one on-site.

[henricook]

Hmm, i had the issue with -XX:TieredStopAtLevel=1 today. I didn't clean after enabling the flag though, do you think that's important?

To repeat what i've said before i've never had it with vanilla openjdk, and in this case switching back to openjdk made it go away

I mention it in case it's relevant, but i'm also running two copies of sbt at the same time on this project, each one with its own target folder - so if there was some kind of graal cache directory that stored things by project root or similar i'd imagine one of the sbt copies could be overwriting the other? something like that? Just a thought

[thomaswue]

If it also occurs with TieredStopAtLevel=1, then it cannot be related to the GraalVM compiler itself, but has to be related to the JDK version or something else in the runtime.

[henricook]

Very curious, do you think it's important to clean my target directory when enabling the flag? So nothing is cached?

[thomaswue]

Depends whether it increases the likelihood of the error to occur. When having a faulty config and a correct config, one would basically try to localize the error by searching for the smallest change that would trigger it. The issue with non-deterministic errors is that one cannot be sure if the error really does not occur for a certain configuration.

[henricook]

I'm afraid i just can't figure out how to recreate this - i'll close the issue but see if one day i come across the cause. Anecdotally i haven't seen it in the last week since upgrading to the latest patch graal release... but that might just be a change in how i've been developing for a particular feature.

[thomaswue]

OK, thanks anyway for investigating and let us know if this comes up again!