Go: time: Location interprets wrong timezone (DST) with slim zoneinfo [1.15 backport]

Created on 22 Oct 2020 · 36Comments · Source: golang/go

What version of Go are you using (`go version`)?

$ go version
go version go1.15.3 linux/amd64

Does this issue reproduce with the latest release?

Yes, but it seems to be fixed in master with d83168eb385f7ae38710028fff5c84252267adbc.

What operating system and processor architecture are you using (`go env`)?

go env Output



$ go env

GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/root/.cache/go-build"

GOENV="/root/.config/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GOINSECURE=""

GOMODCACHE="/go/pkg/mod"

GONOPROXY=""

GONOSUMDB=""

GOOS="linux"

GOPATH="/go"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/usr/local/go"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"

GCCGO="gccgo"

AR="ar"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD=""

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build587886067=/tmp/go-build -gno-record-gcc-switches"

What did you do?

How to reproduce: https://gist.github.com/hlubek/f46a73bc9d150cf1f2af585b0849e3d9

Added a compiled Go binary to a very simple Alpine Docker image with installed tzdata package
Timezones (e.g. Europe/Berlin) uses wrong daylight saving information (e.g. CEST instead of CET for dates after October, 25th in 2020)
This seems to be caused by changes in the tzdata package in Alpine
It does not occur if the tzdata Alpine package is not installed and the embedded zoneinfo time/tzdata is used

What did you expect to see?

An error when loading the location with a non compatible zoneinfo

What did you see instead?

A wrongly interpreted date

Notes

This is already fixed with d83168eb385f7ae38710028fff5c84252267adbc but I wonder why there is no info about that issue (took me some time to figure it out) or a test for the code added to zoneinfo_read.go. This is quite a standard setup for packaging Go programs so I could think many people are affected by this. The fix turns out to only work if the cached zone for now() is the same as the last transition. This broke after the last clock change on October, 25th for CET/CEST.

CherryPickApproved release-blocker

Source

hlubek

👍1

Most helpful comment

I believe this may be a broader issue. This commit to zic changed the default format from fat to slim and that change was pushed out with the 2020b release. This means that the issue is likely to occur wherever 2020b or later timezone files are used (unless the package maintainer overrides the defaults when running zic). With daylight saving ending in many countries this weekend this issue may have a significant impact.

Edit: For those looking for a workaround here are a few options until a fix is released:

Use the ZONEINFO environmental variable to select a different zone file (e.g. export ZONEINFO=/usr/local/go/lib/time/zoneinfo.zip github ).
Include the tzdata package in your app (and do not install tzdata in the container - the package is only used if the time package cannot find tzdata files on the system).
Use a container built from scratch (in combination with one of the above options)
Recompile the relevant tz files in 'fat' mode (instructions)
Pin an earlier alpine version (i.e. alpine:3.8) that uses 2020a or earlier (note that version 3.8 is past its End of Support date).

MattBrittan on 22 Oct 2020

👍28

All 36 comments

Edit: For those looking for a workaround here are a few options until a fix is released:

Use the ZONEINFO environmental variable to select a different zone file (e.g. export ZONEINFO=/usr/local/go/lib/time/zoneinfo.zip github ).
Include the tzdata package in your app (and do not install tzdata in the container - the package is only used if the time package cannot find tzdata files on the system).
Use a container built from scratch (in combination with one of the above options)
Recompile the relevant tz files in 'fat' mode (instructions)
Pin an earlier alpine version (i.e. alpine:3.8) that uses 2020a or earlier (note that version 3.8 is past its End of Support date).

MattBrittan on 22 Oct 2020

👍28

/cc @ianlancetaylor

tklauser on 22 Oct 2020

When working on https://golang.org/cl/261363 (commit 5b509d993d3a3a212b4033815be8b7b439fac672) I noticed that some of the tests in package time failed due to the zic change mentioned in https://github.com/golang/go/issues/42138#issuecomment-714299002, see e.g. https://storage.googleapis.com/go-build-log/29cd3aeb/linux-amd64_58c05150.log.

I investigated this further and figured out that there is an issue with the way we cache location data, which lead to https://golang.org/cl/261877 (commit d83168eb385f7ae38710028fff5c84252267adbc). In addition, this should probably haven been reported as an issue to make it easier to discover, which I didn't do. Sorry about that.

I guess what we could do is backport the changes in src/time/zoneinfo_read.go from https://golang.org/cl/261877 to 1.15 but I'll defer to @ianlancetaylor to decide on that.

tklauser on 22 Oct 2020

Yes, I think this is likely due to the very recent change in the default tzdata install. We didn't notice until now because the change was so recent. Since fortunately @tklauser has already found and fixed the problem on tip, and since more and more people are going to be using slim tzdata installations, I agree that we should backport the fix.

This should be a straightforward backport for 1.15. Unfortunately it is more work for 1.14, which doesn't have the tzset function introduced in https://golang.org/cl/215539. So perhaps we will have to backport that also to 1.14.

I'll repurpose this issue for 1.15 and open a backport issue for 1.14.

ianlancetaylor on 23 Oct 2020

@gopherbot Please open backport for 1.14.

Without a backport people using Go 1.14 with a new tzdata installation will not get the correct timezone information.

ianlancetaylor on 23 Oct 2020

Backport issue(s) opened: #42155 (for 1.14).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

gopherbot on 23 Oct 2020

Change https://golang.org/cl/264301 mentions this issue: [release-branch.go1.15] time: support slim tzdata format

gopherbot on 23 Oct 2020

@hlubek It would be helpful if you could check that https://golang.org/cl/264301 fixes the problem for you. It is a patch against 1.15. Thanks.

ianlancetaylor on 23 Oct 2020

@hlubek It would be helpful if you could check that https://golang.org/cl/264301 fixes the problem for you. It is a patch against 1.15. Thanks.

Sure, will do!

hlubek on 23 Oct 2020

@hlubek I've tried using https://golang.org/cl/264301 and it didn't work as expected, unless I am missing something

dsantos on 26 Oct 2020

@dsantos I could reproduce that the patch fixes the issue reproduced in https://gist.github.com/hlubek/f46a73bc9d150cf1f2af585b0849e3d9 (Alpine with tzdata package) by using a patched Go toolchain to compile a binary for the image.

Could you provider more details on your issue?

hlubek on 26 Oct 2020

Picking up from your example, this is how I've tested it:

FROM golang:1.15 as builder

RUN git clone --branch release-branch.go1.15 https://go.googlesource.com/go /root/goroot &&\
    cd /root/goroot/src &&\
    git fetch https://go.googlesource.com/go refs/changes/01/264301/1 && git checkout FETCH_HEAD &&\
    ./make.bash

WORKDIR /app

COPY . .

RUN CGO_ENABLED=0 /root/goroot/bin/go build -o bin/test

# ---

FROM alpine:3.12

RUN apk update &&\
    apk --no-cache add tzdata &&\
    rm -rf /var/cache/apk/*

COPY --from=builder /app/bin/test /app/test

CMD ["/app/test"]

dsantos on 26 Oct 2020

@dsantos Thanks, I can see it's failing and I digged a little deeper here.

I took the test from https://go-review.googlesource.com/c/go/+/264301 and applied it to the golang image together with the patch and it failed. Before it was working for me. I think it's possible that the extend handling is not correct depending on the now where the cache zone is built (because we had the clock change to DST since early Sunday). And if I update the now seconds to the day before (sec -= 200000) the test will succeed.

hlubek on 26 Oct 2020

Change https://golang.org/cl/264939 mentions this issue: time: fix LoadLocationFromTZData with slim tzdata

gopherbot on 26 Oct 2020

I implemented a fix for tip that could be applied to the backport: https://go-review.googlesource.com/c/go/+/264939

hlubek on 26 Oct 2020

For anyone facing this issue, I switched to time zone data en Go using this: https://medium.com/@mhcbinder/using-local-time-in-a-golang-docker-container-built-from-scratch-2900af02fbaf.

This caused a lot of problems in our site since last week, using America/Mexico_City. A new deploy with docker, would not calculate the correct tz data and we displayed different dates on our site.

e3matheus on 27 Oct 2020

This caused us a few problems as we were using alpine:latest as our base image, which currently uses 2020c-r0 for tzdata. For anyone looking for a workaround, we pinned our version to alpine:3.8 and that seemed to do the trick (as it uses 2020a-r0).

rob-lowcock on 27 Oct 2020

👍1

We all run into this on 17th Oct. oh, btw, we switched to distroless docker
image in order to solve the problem.

On Tue, Oct 27, 2020 at 19:42 Rob Lowcock notifications@github.com wrote:

This caused us a few problems as we were using alpine:latest as our base
image, which currently uses 2020c-r0 for tzdata. For anyone looking for a
workaround, we pinned our version to alpine:3.8 and that seemed to do the
trick (as it uses 2020a-r0).

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/42138#issuecomment-717411178, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAE7EMDTHLRHDOUWVQDVWWTSM4A73ANCNFSM4S2YPIFA
.

butuzov on 27 Oct 2020

👍2

I am able to reproduce this with Alpine 3.12 but not Alpine 3.11. So you might not have to go back as far as 3.8. Can someone else confirm this?

magiconair on 28 Oct 2020

@magiconair I based the versions in my comment on the Alpine packages page which indicates that tzdata 2020c is current for all versions >3.8. I have just tested this (under docker) and can replicate the problem on versions 3.9-3.12 but 3.8 is OK.

MattBrittan on 28 Oct 2020

@MattBrittan Ah, we are using balenalib/intel-nuc-alpine:3.11 as base image and the test did not fail with the cached version of the container I had on my laptop. After removing the image and downloading it again it failed as you said for 3.9-3.12. ...

magiconair on 28 Oct 2020

Can confirm switching to Alpine 3.8 gives you 2020a tzdata which works fine

strideynet on 28 Oct 2020

If need be, I suspect (for go1.15.x) one might be able to build with -tags timetzdata to work around this too. If you embed a non-slim tz file during build and then it can be deployed anywhere... maybe?

glycerine on 29 Oct 2020

👍2

Approving per discussion in a release meeting. This is a serious issue without a workaround. This backport applies to both 1.15 (this issue) and 1.14 (#42155).

dmitshur on 29 Oct 2020

Closed by merging 414668cfbc41fd8cadf74e981849d1e05cc23b2e to release-branch.go1.15.

gopherbot on 29 Oct 2020

Change https://golang.org/cl/266299 mentions this issue: [release-branch.go1.15] time: fix LoadLocationFromTZData with slim tzdata

gopherbot on 29 Oct 2020

gopherbot why are you closing this issue?