Triplea: Use LTS Java (JDK11) or JDK13?

[DanVanAtta]

Question is whether to use the 6 month release cadence version of Java, which is Java13 now, or stick to LTS?

LTS has benefits, notably we won't be pressured to upgrade as often. On the other hand, we do want to try and release every 3 to 6 months, which matches the cadence and we've seen (now and before) a number of problems with being on an out-dated JDK. Since we are bundling JDK with the distributable, I think it's at least a viable option to go with latest Java. We should not also exclude as well the benefit that it's an appeal for developers to be working with the latest java and not stay on JDK11 for the next 'n' years.

[RoiEXLab]

I don't have any strong objections staying at JDK 11 for now, but that is mainly because JDK 12-13 don't offer too many new possibilities from our perpective.
There were many changes under the hood that aimed to make the openJDK easier to maintain, but the only 2 language additions (switch expressions and text blocks) are still not out of preview, meaning that the compiler won't compile without any additional flags.
However currently it looks like switch expressions will finally get out of preview in JDK 14, so that might be a reason to upgrade for the syntactic sugar.

TL;DR: I don't think we should try to go for every new JDK version ASAP, but at the same time we shouldn't hesitate to upgrade to a newer version if we think the APIs or some sort of feature would be nice to have.

[ron-murhammer]

Agree. I think its probably easier to stick with LTS unless we have a compelling reason to upgrade to the intermediate versions.

[DanVanAtta]

I'm not sure LTS is necessarily easier. We had an example with the Travis build issue, we also had many support problems staying on JDK8 and certs. The non-LTS is mostly a cost because the support for patches is not as long, but the release rate and support cycle do more or less match our release cadence, that means each release we can mostly jump to a newer version, hopefully with minimal changes, and take advantage of new features and not worry about problems that arise from old JDK. We also can simplify our build and only test/verify one JDK instead of verifying multiple.

[DanVanAtta]

We should really take a deep look at this. Why is LTS easier? What constitutes a compelling reason? Do we really need a compelling reason?

I'm still seeing Java shops on Java 1.5 and advertising "we are on Java 1.8!", in part because "there was no compelling reason". Yet, it's all old-tech now.

Non-LTS means we commit to upgrading perhaps once a year, as security patches will not be produced for as long. It's just a different pattern where we'll stay up to date.

If we go LTS, then I wonder when exactly we would upgrade? I suspect the answer will be when we are close to end-of-life for the LTS, which means we stay on that version for years.

Given our release cadence, I think it's an option to go with latest, I wonder if there is really strong detailed reason not to. Are we concerned about compatibility issues? Don't we already address those by building with LTS and the latest already?

I'm leaning a bit towards going with latest, I think that may prove to be simpler and more enjoyable. We get to take advantage of latest features, and it would have avoided this latest JDK11 TLS1.0 problem, and we would have avoided the (quite painful) cert problem we had with JDK8, and considering we already build against the latest anyways, I'm not sure if we are actually losing anything. IN other words, if we build against the latest already, and are sure to keep that working, it seems we are paying the price of any downside for latest already without getting the benefit.

[DanVanAtta]

I think maybe a pro/con list will help. I'll give it a start, please add pros/cons as you see them.

Cons to going to the latest:

• need to upgrade before security patches are no longer published for the intermediate version
• tooling could get stuck. Let's say error prone never works with JDK13, and we stay stuck on JDK12. If error prone never gets fixed, and more than a year goes by, then we could start lagging behind and not receive security patches

Pros to going to latest:

• alignment to our own release cycle, we want to release every 3 or 6 months, that is well before the security patch cut-off
• being cut-off from security is maybe moot. We depend on users to upgrade their install. If they are not updating their TripleA every 3 or 6 months anyways, they are not getting the benefits of those patches
• simplify build, we only build to the latest. Instead of keeping the game working on both LTS and the latest, we'd only need to focus on latest (or at least the latest for which our tools still work).
• can take advantage of latest language features
• enticement for new contributors, using latest language is enticing

I was not previously thinking about the various pros, for me the most compelling was realizing that we are building against LTS and latest already, it seems we pay the cost of supporting latest without getting the benefit.

Let's keep working on this pro/con list and make a decision in the coming days.

[DanVanAtta]

Before we come to a decision, @ssoloff I'm really curious if you have any thoughts or a different perspective on this topic.

[ron-murhammer]

Other cons:

• Time it takes to upgrade build
• Time it takes to upgrade IDE/local environment
• Given we really actually release every 6-12 months, I'm concerned we could get stuck on a non-LTS for a while

[DanVanAtta]

Thanks for the extra points.

• build upgrade time: since we already build on LTS and the latest that works, are we not already paying this cost in full?
• Time it takes to upgrade IDE/local environment: this is fair. It is a one-time cost though?

Given we really actually release every 6-12 months, I'm concerned we could get stuck on a non-LTS for a while

• I think this is probably the biggest driver for the decision. An interesting consideration. Because we publish a bundled JDK, it could moot this point. For example, say a security issue is found and is patched. We'd have to be on the non-LTS for over a year to not see that patch. Error-prone is probably going to be the biggest reason why we can't upgrade. If we get into that state where we've waited over a year on a non-LTS and must upgrade, then we could decide to just drop error-prone to enable that release.

It's also worth noting, if there is a security patch that goes out to both non-LTS and LTS, we still need to make our users upgrade. So it'd have to be a scenario where we can't move forward, and our non-LTS is past support.

This last release cycle was an aberration? My impression is that the year long release is not how we want to operate. I've been tempted to say we should define a release cycle to bring more predictability. I vaguely recall that we've had conversations where the year long cycle is not something we want to repeat, that we want to go back to every 3 or 6 months.

[ron-murhammer]

• Build upgrade time - possibly though if we are going to be bundling the JDK then this is probably not necessary
• IDE - Well each time we pick a new JDK version then we have to do it but yeah 1 time cost each time

Generally before this release it was 6-12 months but usually closer to 6 I believe (FYI, its exactly 1 year ago today that we released the current stable). Without some kind of automated upgrader/patcher/etc, I don't think we'd ever push much below 6 months as its just too difficult for users to consume. I also think even ignoring that its hard given the number of contributions to justify releasing much more often than every 6 months or so.

[ssoloff]

It's also worth noting, if there is a security patch that goes out to both non-LTS and LTS, we still need to make our users upgrade. So it'd have to be a scenario where we can't move forward, and our non-LTS is past support.

That is the scenario I had in mind which would make me favor sticking to an LTS version. Consider the following theoretical example:

• Latest TripleA release on Java 16 (non-LTS).
• Java 17 (LTS) is released, EOLing Java 16. (At least through 2021, feature releases are always EOLed the same month as the next feature/LTS release.)
• TripleA can't immediately migrate to Java 17 due to an application (less of a problem) or dependency (more of a problem) incompatibility.
• In the meantime, a Java security issue is identified, and a hotfix is released for Java 17 and backported to Java 11 (LTS). No backport is done for Java 16 because it's EOL.

[RoiEXLab]

I don't have anything to add

[DanVanAtta]

Next question, if we stay with LTS, should we continue building the latest JDK (in addition to LTS)?

I'm thinking about the following scenario:

• we move to support JDK13 in build
• error prone, or other breaks
• we spend time fixing it or working around the problem
• by the time JDK17 is out, error prone has pushed a fix -> our effort are for nought when we could just waited

[ssoloff]

Next question, if we stay with LTS, should we continue building the latest JDK (in addition to LTS)?

@RoiEXLab, correct me if I'm wrong, but I believe the original purpose of adding the JDK 9 build in #2748 was twofold:

1. to track fixing existing Java 9+ build issues (e.g. illegal use of reflection, deprecated types, etc.), and
2. to prevent introducing new Java 9+ issues.

That is, the JDK 9 build was simply meant to give us a means to more casually migrate to a future JDK without having to fix everything en masse with a broken build (assuming a Travis switchover) or requiring all issues to be fixed in JDK 9 environments on local dev machines.

With that in mind, it would be nice to keep a bleeding-edge JDK parallel job to provide a heads-up for any migration issues without it actually affecting the stable build. That is, could the job be run in such a way that, if it fails, it doesn't fail the overall build? But, if it does fail, there still needs to be some kind of notification so the failures are not simply ignored. Similar to the script in place for tracking build failures on master, a GitHub issue could be opened/amended with a title/label of "JDK Next Blocker" or something.

[RoiEXLab]

@ssoloff That is correct.
When trying out github actions, I tried using JDK 13 which caused issues with error-prone at the bytecode level, so getting that fixed would be nice

[DanVanAtta]

If we don't migrate too quickly, I just found this:
https://docs.travis-ci.com/user/customizing-the-build/#rows-that-are-allowed-to-fail

You can define rows that are allowed to fail in the build matrix. Allowed failures are items in your build matrix that are allowed to fail without causing the entire build to fail.

[DanVanAtta]

To sum up, it sounds like LTS is being favored as we feel there will be better support in terms of patches and tooling. A counter-point to that is what we've seen on Travis where we need the Travis image to be updated to contain a latest, as a latest patch was not made available on JDK11. It seems we've scrambled to fix that, given we can install our own JDK, I don't think I see any problem with staying on LTS.

I'd like to discuss a last point and wrap up this conversation with really considering whether we should be building latest JDKs at all. I can see a number of reasons why we should not be:

• Travis build time is not free, the extra VM slows down builds. Build time is important for productivity. If for example, Travis decides to only give us 4 containers and not 5, the extra build slows us down by 7-10 minutes. A failing PR result coming back 20 minutes later compared to 10 minutes later is very significant.
• If we do not plan to migrate to a latest version, the value of building against a latest is just overall a bit questionable. We might be causing ourselves heartache over problems that are going to be fixed only for the next LTS. In another light, some problems might exist now, but when, two years down the road we start considering migrating to the next LTS, those problems might be fixed then. Worrying about problems on latest JDKs that we will not be migrating to anyways, might be causing us undo worry when we just need to wait for them to be fixed on the next LTS.
• We're already not building against the latest, just JDK12, it's concerning that already our desire to build the latest is not already being done. To further that, if any problems preventing us from building against the latest just needs time to be fixed, any efforts we spend now to fix builds against latest could be wasted.

[DanVanAtta]

@RoiEXLab I'm thinking we should drop the JDK12 build. We can then re-partition the other jobs and get better build speed. If we do not plan to migrate to JDK12 or any other JDK for at least 2 or 3 years, I don't think it matters one bit whether that build works now or not (and maintaining it is potentially just busy work).

[RoiEXLab]

I disagree, even if we don't plan to migrate to a newer JDK soon-ish, it's probably good to be runtime compatible with newer versions.
This was a problem in the past where users with java 9 were unable to start the engine because the inofficial eawt api was standardised and changed.
I mean obviously there is no guarantee that a simple build will catch all the regressions, but it's at least some safety.

I mean on Windows and Mac this issue isn't particularly pressing because we ship the JRE with it, but for Linux systems where java might be pre-installed it might be good to be forward compatible.

[RoiEXLab]

Besides that it looks like JDK 14 is getting some convenient features ("Helpful NPEs" for example, or switch expressions becoming standard (previously preview)) and some nice previews that might become standard in JDK 15 like "Text Blocks".

But I'd appreciate if our build would actually build using the latest JDK java 13 is out, so migrating does have value IMO for the reasons I just pointed out

[DanVanAtta]

it's probably good to be runtime compatible with newer versions.

Why? If we are not runtime compatible, who is to say the problem won't be solved later?

This was a problem in the past where users with java 9 were unable to start the engine because the inofficial eawt api was standardised and changed.

We are bundling JDK now, would this be a problem again?

I mean obviously there is no guarantee that a simple build will catch all the regressions, but it's at least some safety.

I wonder about the 'safety', if a new build is broken, does it really matter? It just means some dependency/tool that we are not using has not been fixed. Part of my argument is that it's busy-work. Error-prone note working for JDK13 is an examle, if we spent time on it, I'm quite sure it'd be pointless as eventually it'll be fixed for the next LTS.

but for Linux systems where java might be pre-installed it might be good to be forward compatible.

Then we should be running JDK13 on Travis and not 12 - yet we are not. If we are going to be forward-compatible, why not also then ship with JDK13?

Besides that it looks like JDK 14 is getting some convenient features

I'd like to pick up those features, but if we're sticking to LTS, then we are not picking them up.

#

I think in part, build time is REALLY valuable. I've been doing lots of builds of late. The time cost is very real, we need a very tangible benefit.

[DanVanAtta]

Another comment on:

I mean on Windows and Mac this issue isn't particularly pressing because we ship the JRE with it, but for Linux systems where java might be pre-installed it might be good to be forward compatible.

JDK8 -> JDK9 was a bit infamous as the release cycle was very slow and that pain was in party why Oracle decided to do releases more frequently. Such non-backward compatible changes are not as likely going forward.

I'd also be a bit surprised if linux standard install were moved to a non-LTS as well, particularly if there were a breaking change in it.

@RoiEXLab can you summarize the concrete benefits you see so we can perhaps sum this up and compare to the time cost of waiting for builds?

[RoiEXLab]

Hard to say, but I think there's a good compromise on this issue:
We could setup github actions to run the latest JDK on Pull Requests (or master, matter of taste) and therefore free up the time for travis.
How does that sound?

[DanVanAtta]

Do we know yet how to do a github action on PR open? Is the config for that different from what we have? The spotlessApply running on push is not ideal.

I suspect if we do the 'early JDK' build, we'd want to do JDK13 (and it'll break on error prone).

If we are breaking on error-prone, is there value with JDK13 builds, or will we incur overhead to disable error-prone for JDK13 builds? I mean, if we are checking latest, we should check latest. I'd be slightly surprised if someone runs a non-LTS that is end of life, (JDK12 is now or soon end of life, right?)

IF we have repeated failed builds, is that going to cause a red X on all builds? Ignoring failures is quite bad practice.

It's tempting to do a Travis-master only branch build with ignore failure on: https://github.com/triplea-game/triplea/issues/5486#issuecomment-556921184, but at the same time we are then ignoring a failure.

Maybe a travis JDK13 cron-build once a week could be good? But again, if its' broken and we ignore it, is there a point?

[RoiEXLab]

Do we know yet how to do a github action on PR open? Is the config for that different from what we have? The spotlessApply running on push is not ideal.

Yes, yes, agreed. The problem why I wasn't able to make it only run on Pull Requests is because if the build is triggered on a PR, the branch that's being checked out is some temporary merge branch without any link to the original branch, possibly without permission to push, whereas having this check per branch the permission is always there because the build is started from each fork individually.
Maybe there's a good way to solve this problem but I haven't figured out that one.
But having a simple PR build shouldn't be a challenge apart from build incompatibilities.

IF we have repeated failed builds, is that going to cause a red X on all builds? Ignoring failures is quite bad practice.

Yes, agreed.

Maybe a travis JDK13 cron-build once a week could be good? But again, if its' broken and we ignore it, is there a point?

Creating an issue on failure could help? Not sure though.

[DanVanAtta]

We'll be doing the latest build check mainly for the benefit of linux users. They represent under 5% of total users (based on DL counts), if not fewer. They are also more likely to be able to have/use a LTS if needed.

A non-LTS java can break due to:

• incompatible changes
• tooling issues

For tooling issues IMO we are wasting time and if we see a red-X in the meantime, then we are training ourselves to ignore a failure (and we're just adding process debt). If the problem is an incompatible change, we're assuming a bit that we can fix the problem on the current LTS. For example, let's say an API is deprecated in JDK12 and replaced by a new, then in JDK14 there is a hard-cutover.There would be no path on JDK11 to fix that.

I think we should perhaps consider for a few days in case some ideas come to us. My current opinion is that unless we have a realistic chance to target a deployment for a certain JDK version, then otherwise it's not worth building for it.

[DanVanAtta]

@RoiEXLab I think it might be time to whack the JDK12 build. What benefits are we losing if we remove it, and how do those benefits compare to (not) running JDK13 builds and other "true" latest JDKs? (noting that JDK12 is no longer the latest and should be end of life sooner)

[DanVanAtta]

Opened a PR, no further action items remain in this issue.

https://github.com/triplea-game/triplea/pull/5766