Quarkus: Extension for Flyway

[Sanne]

As suggested by @cristhiank on:

• https://stackoverflow.com/questions/55233662/how-add-flyway-dependency-and-build-native-image-quarkus-java-lang-classnotfound
• https://github.com/cristhiank/quarkus-flyway-bug

[kenfinnigan]

We had one for Thorntail, so should be easy enough

[Sanne]

This is now assigned to @cristhiank , assigning to myself as mentor.

[Sanne]

I haven't tested this, but I suspect that all you need is to have a class similar to this one:

@TargetClass(className="org.flywaydb.core.internal.util.FeatureDetector")
@Substitute
public final class FlywayFeatureDetectionSubstitutions {

    @Substitute
    public boolean isApacheCommonsLoggingAvailable() {
        return false;
    }

    @Substitute
    public boolean isSlf4jAvailable() {
        return false;
    }

    @Substitute
    public boolean isSpringJdbcAvailable() {
        return false;
    }

    @Substitute
    public boolean isJBossVFSv2Available() {
        return false;
    }

    @Substitute
    public boolean isJBossVFSv3Available() {
        return false;
    }

    @Substitute
    public boolean isOsgiFrameworkAvailable() {
        return false;
    }

    @Substitute
    public boolean isAndroidAvailable() {
        return false;
    }

}

This should be packaged in the "runtime" module of a Flyway Quarkus extension.

[cristhiank]

Ok, got it, let me work on it and I will let you know.
thanks @Sanne

[cristhiank]

@Sanne please take a look to the first approach to implement this on my repo:

https://github.com/cristhiank/quarkus/commit/e87f3f36c6c04da142da440d680ad776e62dd749

I ended up doing some more substitutions but it is working now.

[cristhiank]

I also updated the test repo I made to use it and it is working now https://github.com/cristhiank/quarkus-flyway-bug/commit/ce3d2d99d546b92cc3a656675af81914916d1291

[Sanne]

thanks @cristhiank ! it's looking great.

I haven't tried it out yet, I'll trust you on that for now.. I had some minor comments.
Also, could you remove the @authors tags? apparently we're going to remove them all.

Looks like the main thing missing is some basic integration tests.

Please send a pull request when you're happy with it?

[cristhiank]

Thanks @Sanne I will remove the tags, these are the defaults from my IDE 😣 . I will implement the integration tests. What am I suppose to test in this case ? I don’t know how to test the Substitutions but I will find the way, help is appreciate it as always 🙂

[Sanne]

To test things what we usually do is create a new module under /integration-tests/.

Since we can't use JUnit to test the native code, what we normally do is have an application which exposes a REST endpoint, and uses the to-be-tested component to return an "OK". This can be trivially done as we provide several helpers to do this easily.

Since you're creating a new extension, you could add a new module - even if you're going to have just one or two trivial tests that's ok, it helps to test extensions in isolation.

Have a look at, for example, integration tests for the pgsql jdbc driver:

• https://github.com/quarkusio/quarkus/tree/master/integration-tests/jpa-postgresql

some points of interest:

1. relatively easy to start a postgresql database via the maven plugin at https://github.com/quarkusio/quarkus/blob/master/integration-tests/jpa-postgresql/pom.xml#L178
2. most slow tests - like the ones requiring docker or building native-images are disabled by default
3. there's an easy to use @SubstrateTest annotation to test the binaries: https://github.com/quarkusio/quarkus/blob/master/integration-tests/jpa-postgresql/src/test/java/io/quarkus/example/test/JPAFunctionalityInGraalITCase.java
4. and an easy QuarkusTest annotation to test the same thing, but in JVM: https://github.com/quarkusio/quarkus/blob/master/integration-tests/jpa-postgresql/src/test/java/io/quarkus/example/test/JPAFunctionalityTest.java
5. the REST endpoint is very easy to define: https://github.com/quarkusio/quarkus/blob/master/integration-tests/jpa-postgresql/src/main/java/io/quarkus/example/jpa/JPAFunctionalityTestEndpoint.java#L40
6. remember that the endpoint implementation needs to be in /src/main/java, not in tests.. as we'll have to build it as an application ;)

[Sanne]

P.S. forgot to say: of course this makes sure you can test the full compilation of an "application" using Flyway, and so verify indirectly that all substitutions are doing the intended job. You'll have to find a way to assert that Flyway did something - maybe import something on the database, then assert for that?

[cristhiank]

Great, I got it, I will work in it and will be back with the results. Thanks for all your help.

[cristhiank]

Ready for your revision @Sanne :)

[Sanne]

hi @cristhiank , what happened with the PR?

I suspect you had some trouble as you sent it from the master branch and possibly tried to update it to latest. I would suggest to create a new branch for any changes you make.

[cristhiank]

Hi @Sanne, that was exactly what happened, It was my mistake not branching from start.

But, besides that, I'm having another issue. I need to add resources because Flyway reads .sql files from classpath, I already created a build step to produce a SubstrateResourceBuildItem but it is not working because some of the needed resources are in the Flyway jar :( .
So I added a "application archive marker" to try to add those resources but it is not working either.

Please take a look at it in my dev branch and let me know what could I do to add those resources to the native image. Thanks!!

https://github.com/cristhiank/quarkus/blob/adb810d3199816fbdbd1fbdc6c825dd33d5293f8/extensions/flyway/deployment/src/main/java/io/quarkus/flyway/FlywayBuildStep.java

[cristhiank]

@Sanne , I just found out that the latest Flyway beta version removes those external resources https://github.com/flyway/flyway/blob/c50652d51315bfc0c2d1b0fba2548e8c05074eb6/flyway-core/src/main/java/org/flywaydb/core/internal/database/h2/H2Database.java#L103

[Sanne]

@cristhiank there's a little mistake in the FLYWAY_METATADA_TABLE_FILENAME constant : it should be createMetaDataTable.sql instead of createMetadataTable.sql.

That fixes the problem with resource loading ;)

[cristhiank]

Shame on me.... 🤣 Thanks @Sanne . I will make the changes and will be back.

[Sanne]

regarding this block:

resources.add("db/migration/*.sql");

to be fair I'm not sure, but I suspect we don't do "*" expansions. In case that doesn't work, you will need that method to actually look for all matching resources, then list them all individually.

[cristhiank]

@Sanne yes, you are right, expansions won't work, I'm working on that.
I have to add each application .sql file as an SVM resource and I will also have to make some substitution to avoid Flyway trying to discover migrations dynamically.

[Sanne]

perfect, looks like you're on track.

[cristhiank]

@Sanne I need some help here, what would be the best approach to get a list for the resources I added in the FlywayBuildStep as Flyway expects to "discover" them dynamically, but it cannot be done in "native mode".

At this moment I am adding all the resources to the image here:
https://github.com/cristhiank/quarkus/blob/35a549815b378c59ea83fe25745f46a8e0f03b38/extensions/flyway/deployment/src/main/java/io/quarkus/flyway/FlywayBuildStep.java#L78

But at runtime, I have to pass the list of available resources to Flyway. How could I do that ?

https://github.com/cristhiank/quarkus/blob/35a549815b378c59ea83fe25745f46a8e0f03b38/extensions/flyway/runtime/src/main/java/io/quarkus/flyway/runtime/QuarkusPathLocationScanner.java#L28

[Sanne]

at runtime you can not load nor scan resources: you only have the binary image (the application file), and it will only include the resources we explicitly listed for inclusion at build time.

So migration scripts need to be either:

• included explicitly in the build step
• be external files, possibly discovered from some Path.

It looks like the first case should be covered by your work already?

[Sanne]

I should clarify: by "at runtime" I mean of course "at runtime in native mode". You can still allow doing such things for standard JVM runtimes and dev-mode.

[cristhiank]

Yes, I understand, at this moment I am adding the resources to the image. But how can I access that list “at runtime” ? Because the list is generated at build time and Flyway has the concept of “Scanner” that discovers resources. So my approach is to substitute all the scanners for a single “QuarkusScanner” that will return the fixed list of resources included in the image. But I don’t know how to get the list at runtime.

[Sanne]

ah, good point.

There are a couple of very interesting things you can do. Unfortunately the specific need of accessing this stored state from within the scope of a substitution makes it hard. Other extensions can benefit from automatic bytecode recording by using io.quarkus.runtime.annotations.Template, but in this case we can't easily access the recorded template.

The simplest solution is probably to generate and store an additional single resource which you can then load in a static block within QuarkusScanner: see io.quarkus.deployment.builditem.GeneratedResourceBuildItem to generate one.

Alternatively you could use gizmo to generate the an array as a constant in bytecode, then have QuarkusScanner access this reflectively. You can find examples of constant generation in io.quarkus.deployment.steps.ConfigurationSetup but these are not at all simple; I'd probably just generate the resources myself.

There's an even simpler solution, but you would need to rewrite method private List<String> discoverApplicationMigrations(Path root) as private List<String> discoverApplicationMigrations(). Maybe it's ok to enforce a strong convention on these files needing to be resources in a very specific place which can't be reconfigured?

If you're ok with that, you could simply introduce a static constant in QuarkusScanner; make sure its initializer loads the list:

public final class QuarkusPathLocationScanner implements ResourceAndClassScanner {
     private static final List<String> ALL_SQL_RESOURCES = discoverApplicationMigrations();

 ...
 }

[cristhiank]

Cool, will do some research and I will let you know when I have it done. Thanks!

[cristhiank]

Hello @Sanne, it looks like I finally made it. Please let me know what you think. It's my first time working with Graal/SVM it was quite a challenge.