Jooq: Implement KotlinGenerator

[lukaseder]

A dedicated KotlinGenerator could greatly improve the jOOQ experience for Kotlin developers. We could generate some specialised implementations, including:

• [x] Data class support for POJOs (#6345, #10288)
• [x] Property support in POJOs (see above), interfaces, and records. Properties are more idiomadic than JavaBeans style getters and setters
• [x] Globally available objects should be package objects, not companion objects. This raises new questions due to name clashes, as e.g. sequences, udts, tables would all be located together in a schema package. Some dialects (e.g. PostgreSQL) forbid this, but others may not.
• [x] ~Property support in stored procedure call objects~
• [x] ~Property support in tables? This would allow fo providing views of a table type...~
• [x] ~Option to handle nullability in generated types.~: Implemented separately: #10212

The above list may be enhanced

[ssijak]

Well, Kotlin is really gaining traction. And with Spring 5 adding first class support I think it is reasonable to add Kotlin generator because I would guess JOOQ is most frequently used by Spring people.

[lukaseder]

because I would guess JOOQ is most frequently used by Spring people.

That's a bold claim - I'd love to see some numbers on that ;-)

[ssijak]

@lukaseder Maybe I worded it badly to sound like JOOQ is used more than any other database library (Hibernate, MyBatis, whatever) with Spring. What I meant was that majority of JOOQ uses probably comes from Spring + JOOQ combo. Mainly because of how Spring use is widespread, and because of ease of starting with it by using spring-boot-starter-jooq. There is also vertx starter and they have kotlin integration also.
But, this could be completely wrong, it is just me guessing. But anyway Kotlin provides very easy interop with Java so that using it with something like Dropwizard or any other microframework is easy.

[lukaseder]

What I meant was that majority of JOOQ uses probably comes from Spring + JOOQ combo

I understood you correctly, and trust me, jOOQ was a thing before Spring Boot ;-) Anyway, let's discuss the actual feature request.

jOOQ already works super well with Kotlin, see e.g.
https://blog.jooq.org/2017/05/18/10-nice-examples-of-writing-sql-in-kotlin-with-jooq/

The "advantage" of a KotlinGenerator would be that generated code would look more native. There aren't too many other advantages from my experience with the ScalaGenerator, so whether adding this additional complexity is really a good idea will need to be seen.

[PCasafont]

Record classes could be much more strictly typed, for example only nullable columns would be nullable.
We could also have default values in their constructors so they can be called with named arguments.
We could benefit from the [] operators as well, instead of having explicit declarations for each nth column... or they could be field aliases.
Generally, the usage code would look much, much cleaner.

[lukaseder]

Record classes could be much more strictly typed, for example only nullable columns would be nullable.

Yeah, about that :-) Maybe an ORM can get away with this "lie", but a SQL framework cannot. In SQL, every type is nullable. NOT NULL is really a (check) constraint, not a type modifier. Something that may be stored as non-nullable can quickly become nullable if you apply outer join, union, or any sort of expression on a column. See item #1 here:
https://blog.jooq.org/2017/07/25/5-things-you-may-not-have-known-about-jooq/#thing1

We could also have default values in their constructors so they can be called with named arguments.

That would be really really cool, indeed!

We could benefit from the [] operators as well, instead of having explicit declarations for each nth column... or they could be field aliases.

But you can, already! Record.get() and Record.set() are defined precisely for that.

Generally, the usage code would look much, much cleaner.

I have my sincere doubts. There was little gain from the ScalaGenerator, but tons of maintenance overhead. I'd like to see many more advantages, though (such as the named / defaulted parameters)

[PCasafont]

Well, in java everything is nullable as well, and kotlin is just adding a compile time check. But I get your point.
I'll be back after some further experimenting with JOOQ+kotlin

[lukaseder]

kotlin is just adding a compile time check

It's not just a check, it's a guarantee. If a method guarantees that it returns String (as opposed to String?), your client code will simply be wrong, because it won't handle (and in the case of Kotlin is incapable of handling) the occasional NPE that will arise.

A simple example:

class AddressRecord : UpdatableRecordImpl<Address> {
    var name: String get() ...
}

Now, since AddressRecord extends jOOQ's UpdatableRecordImpl, what will you initialise that name to when creating a new record? Exactly, the only reasonable answer is null. But it cannot and should not be null according to the signature.

Yet, null is a reasonable value prior to inserting the record, because it could be generated by a trigger or default value in the database only upon storing the record. null is also a reasonable value if you left join address to customer, and a customer doesn't have an address. All the properties will be null.

[PCasafont]

I understand your concerns, and lateinit comes to the rescue! As long as you don't call its get when it wasn't yet set/initialized it won't crash.
The only bad part is that it's not supported on primitives such as int or double, but there are work arounds (initial value at 0, -1 or MIN_VALUE is one of them).

[lukaseder]

Yeah, I had seen lateinit in the past. I don't see the point of that. Either you're going to ensure non-nullability in a language, or you're not going to do it. I mean, in DI contexts, people have started to conclude that field injection isn't good, constructor injection is the way to go.

After all, if accessing this lateinit property can raise another type of exception (which isn't called NPE, but effectively is), then why bother with all of this?

[PCasafont]

It's basically to be able to have not nullable attributes which won't be set in the constructor. Not the best practice, but that can be very handy in JOOQ. After all, if in a query some of the properties isn't set, you should _never_ try to access it.

[lukaseder]

It's basically to be able to have not nullable attributes which won't be set in the constructor. Not the best practice, but that can be very handy in JOOQ

I understand the idea behind it ;) But what on earth would you initialise an address with? Temporary Street? Enter Street name here? :)

After all, if in a query some of the properties isn't set, you should never try to access it.

And what's a good indicator in a type system to indicate that the property isn't set? Nullability!

[PCasafont]

If you want null to be a normal state for the field's value, yes indeed, but if you want it never to be null unless it's not yet initialized (or you don't plan to initialize nor use it in some cases) lateinit does a pretty good job.

[lukaseder]

if you want it never to be null unless it's not yet initialized (or you don't plan to initialize nor use it in some cases)

Look... :) What you do on your side, that's your business. But the library will never pretend that some cases is an acceptable exception. It will not guarantee non nullability if it is a lie.

[PCasafont]

There was no need to go personal, I was just stating how kotlin works and its potential advantages. There's no need to call a programming language a liar either, just because it offers extra flexibility on very common cases without having to sacrify null safety.
Anyways, thanks for all your feedback, the conversation has been productive despite the looks.

[lukaseder]

There was no need to go personal

There was nothing personal in my words. What I meant is that library users (e.g. you) can make more assumptions about the particular application of language features including nullability guarantees in particular contexts than the library itself. The library has to make sure that code works for everyone under all assumptions. The only safe choice here is not to use the Kotlin nullable/non-nullable type language feature for this use-case.

There's no need to call a programming language a liar either

You're reading stuff between the lines here. What I called a lie is the application of a Kotlin language feature in a situation where it is not suitable. It's a "lie" just like this (pardon the hyperbolic example):

/**
 * Will never return null.
 */
public String foo() {
    return null;
}

I hope this made it more clear. Again, nothing personal or offensive intended here.

Cheers
Lukas

[PCasafont]

Thanks for the clarification. To be honest I am not very fond of lateinit, especially given its lack of support for primitives, so yes; a first approach with nullables everywhere would be nice already.
Hopefully we can convince you someday, but until then we can always auto convert the generated java source in our projects.

[alexvas]

There is one be additional to-be-feature that would have considerable impact on a build time of a jOOQ-ed application. It is an ability to give internal visibility modifier to generated code. That would exclude generated code out of project's ABI.

To elaborate it a bit further, one might expect that common application would incapsulate interactions with database in some dedicated module, i.e. Data Access Layer (DAL). If DAL's API will not expose "inner" database structure, that would add some degree of agility to the application's architecture: one might refactor DB structure, leaving API intact.

There is no way to tell build system (Gradle, Bazel, etc.) that classes or methods do not participate in public module's API except to mark them with special modifier. This is what Kotlin's internal visibility modifier is exactly for. At the _configuration_ phase, when build tool could omit complex (as complex as application - DB interaction might be) generated code out of consideration and take in account only actual (refined) API, the configuration would be done much faster. Also, if changes on the low level (application - DB interaction) do not incur API change, other modules would not depend on DAL in the _compile_ phase. That would allow to compile different modules in parallel and further speed up the project's build.

[lukaseder]

Generating classes with properties for stored procedur IN and OUT parameters is very convenient!

Consider these screenshots (from Twitter: https://twitter.com/lukaseder/status/1260939631316684800, I don't have the text form anymore):

Stored procedure definition in HSQLDB

Kotlin API usage

Log output

This is definitely a big plus over the Java generated code, especially in the presence of defaulted parameters!

[lukaseder]

There are a few caveats with the "improved" stored function support. Of course, this syntax can be used to produce a more "idiomatic" experience in Kotlin, especially to help use defaulted parameters. But this can also be done with the standard way to invoke routines in jOOQ.

Whereas the above suggestion would allow for running arbitrary, unrelated code in this lambda.

I don't like it. I've reverted the implementation for now.

[lukaseder]

The only remaining bullet here is:

Globally available objects should be package objects, not companion objects

This is a bit tricky.

• Companion objects allow for keeping things similar to the Java generated code, and allow for properly namespacing each object type individually, but they result in annoying Companion identifiers in imports, e.g.: import org.jooq.codegen.test.x.db.DefaultSchema.Companion.DEFAULT_SCHEMA. Also, there seems to be no way to "import-on-demand" from a companion object: Companion.*
• Auxiliary classes, such as the Indexes class, also don't allow for "import-on-demand". It's not possible to import Indexes.*
• Package objects produce the best user experience ("import-on-demand"), and seem to be most kotlin idiomatic, but they risk producing collisions between object types, e.g. when a table and a primary key and a sequence and a UDT share the same name. Moving them in a subpackage is a possibility, but if we do this e.g. for tables, there are alredy contents in that subpackage, so there's new risk for collisions that we don't have in Java. For example a single-letter table name T would produce both the class T and the package object T in the same package.

While users can influence package names of (some) generated objects, they cannot decide themselves what layout we choose to generate, so we have to make that decision for them, now.

I'm in favour of a package object based solution with careful avoidance of collisisons in package names (specifically, tables)

[lukaseder]

So far, the assumption when generating these Tables.java, Indexes.java and other files was hard-coded that there will be such a file and class with static methods (or object with defs in Scala) on the same level as the schema.

We'll need some additional infrastructure to abstract over these global references files. There's a pre-existing feature request to allow for abstracting over them via generator strategies, as always: #6183.

That way, we can solve this problem quite elegantly for Kotlin.

[lukaseder]

There's also a disadvantage when using package objects. We might no longer be able to import the generated keys, because we can no longer qualify them using Keys.PK_123. There's no partial qualification in Kotlin, such as e.g. in Scala. So, we'll have to import the PK_123 itself, and might run into more conflicts, leading to more full qualification...

[Gama11]

So, we'll have to import the PK_123 itself, and might run into more conflicts, leading to more full qualification...

I think you could also import as, letting you choose a different name rather than needing the full path.

[lukaseder]

I think you could also import as, letting you choose a different name rather than needing the full path.

This would only further complicate things. The PK_123 identifier is either imported or fully qualified: simple

[lukaseder]

So, package constants work quite nicely and feel much better for most object types (e.g. Domains.kt, Keys.kt, Sequences.kt).

Last caveat, the Tables.kt and Routines.kt files cannot be located in the <catalog>.<schema>.tables and <catalog>.<schema>.routines packages, because of the conflicts they would produce with classes of the same name as the constants. E.g.

CREATE TABLE t (i int);

Would produce:

• Class <catalog>.<schema>.tables.T (in T.kt)
• Constant <catalog>.<schema>.tables.T (in Tables.kt)

A new auxiliary subpackage could work here, e.g.

• Class <catalog>.<schema>.tables.T (in T.kt)
• Constant <catalog>.<schema>.tables.references.T (in Tables.kt)

This logic would be implemented in DefaultGeneratorStrategy::getGlobalReferencesJavaPackageName (see also #6183) and can be overridden by user-written strategies, of course.

[lukaseder]

We'll need to review our tutorial eventually and create a new one where users can switch between Java/Scala/Kotlin, pointing out the little differences.