Quarkus: Native image does not inherit system time zone

[sgerr]

Native image does ignore system time zone, assumes GMT instead. Code snippet to verify:

@ApplicationScoped
public class Startup {

    public void startup(@Observes StartupEvent ev) {
        System.out.println("--- default time zone id is: " + ZoneId.systemDefault());
    }

}

Output (no matter system timezone is):

--- default time zone id is: GMT

[geoand]

@dmlloyd is this something you have seen before?

It sounds like a GraalVM bug, not a Quarkus one.

[dmlloyd]

No, I have never seen this. I agree this looks like a GraalVM bug. I wonder if it doesn't load the zone database by default to save space.

[sgerr]

Is there any workaround except of explicit specifying time zone every time it is needed?

[jaikiran]

I can't reproduce this. I get the right timezone. I'm on Graal 19.2.0.1 (I haven't yet tested on 19.2.1). What version of Graal VM are you on and which Java vendor and version?

Also, can you paste the exact command you use to generate the native image?

[jaikiran]

Tried on 19.2.1 of GraalVM. Can't reproduce there either - I get my system specific timezone as the value.

[sgerr]

Hello, @jaikiran

I'm building native image using maven with docker on Windows 10, running it on Oracle Linux 7 or even Fedora 30. To reproduce:

1. Create project

mvn io.quarkus:quarkus-maven-plugin:0.27.0:create -DprojectGroupId=com.sgerr -DprojectArtifactId=tztest -Dextensions="undertow" -Dpath=tz

1. Create class Starter, use code snippet above.
2. Build native image

mvnw package -Pnative -Dquarkus.native.container-build=true

Docker command is:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] docker run -v C:\work\sandbox\SandboxProjects\tztest\target\tztest-1.0-SNAPSHOT-native-image-source-jar:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.1 -J-Dsu
n.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.ne
tty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar tztest-1.0-SNAPSHOT-runner.jar -J-Djava.util.
concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-JNI -H:-UseServiceLoaderFeature -H:+StackTrace tztest-1.0-SNAPSHOT-runner

1. Copy and run it on Linux.

# date +%z && ./tztest -Dquarkus.http.port=9999
+0300
--- default time zone id is: GMT
2019-10-31 15:04:57,611 INFO  [io.quarkus] (main) tztest 1.0-SNAPSHOT (running on Quarkus 0.27.0) started in 0.016s. Listening on: http://0.0.0.0:9999
2019-10-31 15:04:57,611 INFO  [io.quarkus] (main) Profile prod activated.
2019-10-31 15:04:57,611 INFO  [io.quarkus] (main) Installed features: [cdi, resteasy, servlet]

[gsmet]

Stupid question but would the timezone be fixed at build time?

On Thu, Oct 31, 2019 at 4:09 PM sgerr [e-mail] wrote:

Hello, @jaikiran https://github.com/jaikiran

I'm building native image using maven with docker on Windows 10, running
it on Oracle Linux 7 or even Fedora 30. To reproduce:

1. Create project

mvn io.quarkus:quarkus-maven-plugin:0.27.0:create
-DprojectGroupId=com.sgerr -DprojectArtifactId=tztest
-Dextensions="undertow" -Dpath=tz

1. Create class Starter, use code snippet above.
2. Build native image

mvnw package -Pnative -Dquarkus.native.container-build=true

Docker command is:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] docker run -v C:\work\sandbox\SandboxProjects\tztest\target\tztest-1.0-SNAPSHOT-native-image-source-jar:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.1 -J-Dsu
n.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.ne
tty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar tztest-1.0-SNAPSHOT-runner.jar -J-Djava.util.
concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-JNI -H:-UseServiceLoaderFeature -H:+StackTrace tztest-1.0-SNAPSHOT-runner

1. Copy and run it on Linux.

date +%z && ./tztest -Dquarkus.http.port=9999

+0300
--- default time zone id is: GMT
2019-10-31 15:04:57,611 INFO [io.quarkus] (main) tztest 1.0-SNAPSHOT (running on Quarkus 0.27.0) started in 0.016s. Listening on: http://0.0.0.0:9999
2019-10-31 15:04:57,611 INFO [io.quarkus] (main) Profile prod activated.
2019-10-31 15:04:57,611 INFO [io.quarkus] (main) Installed features: [cdi, resteasy, servlet]

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/quarkusio/quarkus/issues/5016?email_source=notifications&email_token=AAJYOBMRTBIBDBKFOP4WLI3QRLYJNA5CNFSM4JGU4C32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECYEAFI#issuecomment-548421653,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJYOBNMXOM2SPSJGFTUN5DQRLYJNANCNFSM4JGU4C3Q
.

[dmlloyd]

I'm looking at the source code and it looks like this:

@AutomaticFeature
final class TimeZoneFeature implements Feature {
    static class Options {
        @Option(help = "When true, all time zones will be pre-initialized in the image.")//
        public static final HostedOptionKey<Boolean> IncludeAllTimeZones = new HostedOptionKey<>(false);
    }

    @Override
    public void afterRegistration(AfterRegistrationAccess access) {
        TimeZone defaultZone = TimeZone.getDefault();
        String[] supportedZoneIDs = Options.IncludeAllTimeZones.getValue() ? TimeZone.getAvailableIDs() : new String[]{"GMT", "UTC", defaultZone.getID()};
        Map<String, TimeZone> supportedZones = Arrays.stream(supportedZoneIDs)
                        .map(TimeZone::getTimeZone)
                        .collect(toMap(TimeZone::getID, tz -> tz, (tz1, tz2) -> tz1));
        ImageSingletons.add(TimeZoneSupport.class, new TimeZoneSupport(supportedZones, defaultZone));
    }
}

So it looks to me like it's only including, by default, GMT, UTC, and whatever the build host's time zone is. So if the run host has a different time zone, you won't get it and it'll fall back to UTC. This should be verifiable by building a native image with a different value set for the TZ env. variable and then running it to see if it falls back to UTC.

Setting -H:+IncludeAllTimeZones should "fix" the problem but might inflate the image significantly.

[sgerr]

Hello, @dmlloyd

Thank you for ypur reply. Please clarify, how to add -H:+IncludeAllTimeZones switch when maven build?

[sgerr]

Oh, yes. Including in pom properties

<properties>
...
        <quarkus.native.additional-build-args>-H:+IncludeAllTimeZones</quarkus.native.additional-build-args>
    </properties>

will do that.

Command now is:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] docker run -v C:\work\sandbox\SandboxProjects\tztest\target\tztest-1.0-SNAPSHOT-native-image-source-jar:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.1 -J-Dsu
n.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.ne
tty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -H:+IncludeAllTimeZones --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar tztest-1.0-SNAPSHOT-r
unner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-JNI -H:-UseServiceLoaderFeature -H:+StackTrace tztest-1.0-SNAPS
HOT-runner

But still no effect:

date +%z && ./tztest -Dquarkus.http.port=9999 +0300 --- default time zone id is: GMT 2019-11-01 06:13:34,096 INFO [io.quarkus] (main) tztest 1.0-SNAPSHOT (running on Quarkus 0.27.0) started in 0.019s. Listening on: http://0.0.0.0:9999 2019-11-01 06:13:34,097 INFO [io.quarkus] (main) Profile prod activated. 2019-11-01 06:13:34,097 INFO [io.quarkus] (main) Installed features: [cdi, resteasy, servlet] 2019-11-01 06:13:39,201 INFO [io.quarkus] (main) tztest stopped in 0.005s

Is it because Windows build?

[sgerr]

I just rebuild project on Fedora 30 (maven, opendjk 1.8, docker).

docker run -v /root/tztest/target/tztest-1.0-SNAPSHOT-native-image-source-jar:/project:z --user 0:0 --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.1 -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -H:+IncludeAllTimeZones --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar tztest-1.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-JNI --no-server -H:-UseServiceLoaderFeature -H:+StackTrace tztest-1.0-SNAPSHOT-runner

The same result:

./target/tztest-1.0-SNAPSHOT-runner --- default time zone id is: GMT 2019-11-01 07:33:45,352 INFO [io.quarkus] (main) tztest 1.0-SNAPSHOT (running on Quarkus 0.27.0) started in 0.020s. Listening on: http://0.0.0.0:8080 2019-11-01 07:33:45,353 INFO [io.quarkus] (main) Profile prod activated. 2019-11-01 07:33:45,353 INFO [io.quarkus] (main) Installed features: [cdi, resteasy, servlet]
Build host's time zone is neither GMT no UTC.

[dmlloyd]

It seems that the default time zone is not reset by GraalVM at native image start, so the time zone captured at build is retained. To fix it, there would have to be some startup hook which causes redetection of the default zone.

Furthermore there's a bug in that if you set the default time zone to null, you'll end up with lots of NPEs because the null handling logic is substituted away by GraalVM.

The class with the substitutions references an internal (secret) GraalVM bug ID GR-11844. I'm not sure what the contents of that bug ID are, but there are at least two major problems with their substitution class...

[sgerr]

so the time zone captured at build is retained.

Unfortunatelly, it does not. At build time zone was neither GMT no UTC too.

So, no workaround on that except to specify time zone explicitly at start time or invoke date +%z on *NIX system and collect output.

[dmlloyd]

Did you not say that you were building in a docker container? Are you setting the time zone env var in docker?

[sgerr]

Are you setting the time zone env var in docker?

No. I'm using standard maven build procedure as I mentioned above:

mvnw package -Pnative -Dquarkus.native.container-build=true

[dmlloyd]

I believe docker does not inherit the time zone from the host by default. See https://forums.docker.com/t/synchronize-timezone-from-host-to-container/39116/2

[sgerr]

May be. But it does not matter because the point of our interest is the TZ of host machine at runtime.

[geoand]

@sgerr what @dmlloyd is saying essentially is that if you do mvnw package -Pnative (and thus don't use Docker) you should see at runtime the TZ of the machine that built the native binary.

Due to the fact that you are using docker to build the binary, you don't see that behavior because Docker (seemingly) doesn't use the hosts default TZ.

[sgerr]

if you do mvnw package -Pnative (and thus don't use Docker) you should see at runtime the TZ of the machine that built the native binary.

@geoand, yes. But is it the behaviour you expected? Of course -- not. You are expecting to get host machine time zone at runtime. Moreover in some cases, for example storing timeseries to DB, you do not want to think about time zones at all because Hibernate is clever enought to do this work for you behind the scenes.

[geoand]

@sgerr so yes, of course the behavior is not expected.

What we are trying to do however is set the scene for what the actual problem can be and what is peripheral to the discussion.
@dmlloyd's analysis and your testing make it clear that the bug is in GraalVM, so I would propose you open an relevant issue in the project's issue tracker.

[sgerr]

@geoand yes, sure. Meanwhile we can use quarkus in JVM mode where it works as expected. Thank you.

[geoand]

Thanks!

[gastaldi]

Related to https://github.com/quarkusio/quarkus/issues/1842

[jaikiran]

I've raised a graal PR https://github.com/oracle/graal/pull/1819 to see if we can have timezone support enhanced to allow more fine grained control on how users or Quarkus can configure the available timezones in native-image.