Node: Incorrect timezone

[santsys]

I'm seeing variation in timezone inside of node. OS is reporting GMT-0700 (PDT), node is reporting GMT-0900 (Pacific Standard Time).

Node Version: v10.14.2
Platform: Linux (node running inside of docker container)

> date
Wed Jul 17 16:51:08 PDT 2019

> date +"%Z %z"
PDT -0700

> node
>(new Date()).toString();
'Wed Jul 17 2019 14:51:36 GMT-0900 (Pacific Standard Time)'

Node appears to be seeing PST as -0900 and not -0700. I figure I'm missing something, but I don't know what.

Any ideas would be appreciated! Thank you.

[bnoordhuis]

Can you retest with the latest v10.x release, v10.16.0? ICU, the library that handles i18n and tzdata, received several updates after v10.14.2.

[santsys]

Unfortunately I get the same result with v10.16.0.

> node --version
v10.16.0

> node
> (new Date()).toString()
'Thu Jul 18 2019 12:29:09 GMT-0900 (Pacific Standard Time)'

> date +"%Z %z"
PDT -0700

[Trott]

So, some odd things of note:

• There isn't anywhere on the planet right now that is on Pacific Standard Time. It is Pacific Daylight Time everywhere that uses Pacific Time.

• Even if there was a place that was currently on Pacific Standard Time, the offset should be -0800 and not -0900.

So that result is baffling. I'm unable to replicate it. Something strange is going on.

/ping @srl295

[srl295]

@santsys

node running inside of docker container
so… can you get me (from the linux container)

~- date~
~- echo $TZ~

• `ls -ld /etc/localtime'

is it a container I can access to debug?

[srl295]

I tried this, which may not match your use case.

$ docker run -it --rm node
Welcome to Node.js v12.6.0.
Type ".help" for more information.
> new Date().toString()
'Thu Jul 18 2019 22:21:06 GMT+0000 (Coordinated Universal Time)'

[santsys]

@srl295

Here is the output from ls -ld /etc/localtime on the host:

Jul 17 14:02 /etc/localtime -> ../usr/share/zoneinfo/US/Pacific

Also from the host system:

> date
Thu Jul 18 15:40:56 PDT 2019

> date +"%Z %z"
PDT -0700

> node --version
v6.7.0

> node
> (new Date()).toString();
'Thu Jul 18 2019 15:41:24 GMT-0700 (PDT)'

Unfortunately the docker image is private (but it's very simple), here is the base Dockerfile:

FROM alpine:latest

COPY . /src
WORKDIR /src

RUN apk --update add npm nodejs

RUN npm install
CMD node server.js

and built with

sudo docker build --no-cache -t *location/*name .

Docker inspect also shows that the /etc/localtime is mounted:

 {
    "Type": "bind",
    "Source": "/etc/localtime",
    "Destination": "/etc/localtime",
    "Mode": "ro",
    "RW": false,
    "Propagation": "rprivate"
}

I could understand if the container was spitting out UTC or some other time, but, it's spitting out an offset that doesn't even exist. So I'm just not sure what is going on. Happy to walk through or throw any additional commands at things... Thank you!

[srl295]

can you try to reproduce it with a minimum dockerfile and maybe a docker-compose? I suspect that ICU is getting confused by the mounted /etc/localtime. What's the host OS and version?

[srl295]

also try ls -ld /etc/localtime and od -t x1 < /etc/localtime | head on the container

[santsys]

The host is a custom Linux distribution, but it doesn't have issues with other containers.

For example, another container running node v8.9.3 prints the correct timezone. The /etc/localtime mappings are all exactly the same.

> node --version
v8.9.3

> node
> (new Date()).toString();
'Thu Jul 18 2019 17:15:15 GMT-0700 (PDT)'

So doing a little backtracking and rolling back to an alpine:3.8 docker image that rolls back to node v8.14.0, the correct timezone offset is used. Using alpine:3.9 and node v10.14.2 fails with the wrong timezone again. So it would seem that either there is some handling that is incorrect in the newer images with newer node implementations.

[srl295]

@santsys OK… but I need to figure out how to reproduce this.

The host is a custom Linux distribution, but it doesn't have issues with other containers.

Hmm… If you are mounting the /etc/localtime from your custom distribution into the container, how can ICU know how to work with it? I think I need a minimum example to test.

As I said I think ICU's getting confused by the localtime link. alpine doesn't seem to have /etc/localtime normally. … I'd actually recommend setting the TZ= environment variable rather than /etc/localtime.

node -e 'console.log(new Date().toLocaleString(), "|", new Date().toString())' might be helpful too, to compare the two code paths.

[santsys]

Alpine gets the correct timezone in all instances, it's the newer node distributions that are not picking up this correct timezone (or are incorrectly offsetting them to a PST offset that doesn't even exist). How is PST being calculated as GMT-0900, it should be GMT-0700 or GMT-0800 depending on daylight savings, no?

> node -e 'console.log(new Date().toLocaleString(), "|", new Date().toString())'
7/19/2019, 7:26:58 AM | Fri Jul 19 2019 07:26:58 GMT-0900 (Pacific Standard Time)

[srl295]

@santsys hi, i was still waiting to get the results of the following so I can evaluate this conclusively

also try ls -ld /etc/localtime and od -t x1 < /etc/localtime | head on the container

… or even, send me the localtime file.

Can you reproduce it without the mount? i.e. by manually copying the /etc/localtime file?

Overall, my general thought is that bind mounting /etc/localtime is not a good idea (though from a web search it seems to be sadly popular) as there isn't sufficient information in a portable format in that file. What ICU is trying to do is to read the _symlink_ such as:

lrwxrwxrwx. 1 root root 41 Jul 11 22:27 /etc/localtime -> ../usr/share/zoneinfo/America/Los_Angeles

… in order to extract the zone info. If this fails, it has to fall back to probing about the POSIX api to try to deduce the zone information, which apparently didn't work here.

[santsys]

I'm able to reproduce using the private docker image, i'm trying to reproduce that in something i can send that isn't private so you can see it directly. But that's a bit of a process (but i'm working on it).

The ls -ld /etc/localtime command from inside a container is:

-rw-r--r--    3 root     root          2819 Jul 10 14:44 /etc/localtime

however the od command doesn't run:

od -t x1 < /etc/localtime | head
the input device is not a TTY

---

OK, I think I have something public that will reproduce this, give this a try:

sudo docker run -it --mount type=bind,source=/etc/localtime,target=/etc/localtime,readonly santsys/node-tz-sample:1.0.0 node

Then inside of that container, you can run (new Date()).toString(); and it should output the incorrect PST offset ('Fri Jul 19 2019 08:17:18 GMT-0900 (Pacific Standard Time)').

[srl295]

@santsys thanks. From ICU's perspective, it doesn't try to read /etc/localtime, just to first see if it is a hard or soft link to something. For example, I tried a build inside alpine linux, and when I did:

ln -s /mnt/afs/0.0.0.0/var/db/usr/share/slice12/timezone/zoneinfo/America/Los_Angeles /etc/localtime

… then the zone information was correct. (no, that was not a valid path within the container)

If you are only running node, copying the zoneinfo as a symlink might work.

[srl295]

can you give a hint as to what flavor of distro your custom distro is? And, does it have a custom time zone? (I assume not)

[santsys]

can you give a hint as to what flavor of distro your custom distro is? And, does it have a custom time zone? (I assume not)

I'm able to reproduce on a standard Linux distro as well... Able to reproduce on CentOS Linux 7 (Core).

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[srl295]

It is interesting that your zone is US/Pacific instead of America/Los_Angeles, but that shouldn't be the problem ( https://github.com/eggert/tz/blob/master/backward#L126 )

[santsys]

On the new host that also has the issue:

lrwxrwxrwx.  1 root root   41 Feb 25  2016 localtime -> ../usr/share/zoneinfo/America/Los_Angeles



                    

[srl295]

ok- reproduced! with that image. I'm actually able to do a full ICU build in that container, so I'll debug this and let you know what I find.

But I also verified that setting TZ=US/Pacific produces the correct behavior (even with the anomalous /etc/localtime). So that's my recommendation.

[srl295]

OK, an ICU build says: <param name="tz.default">PST</param> , so year round standard time. Not sure about the offset. (I'm not a fan of DST but I'll leave that out of this here.)

I'm guessing that PST was as much as ICU was able to get out of the localtime file, and it went from there.

[santsys]

Still doesn't make sense why it would calculate it to an offset that isn't actually part of the timezone (to me anyways).

[srl295]

@santsys the /etc/localtime doesn't match any of the binary files in /usr/share/zoneinfo in that container. Usually it would be a symlink, or a hard link, or worst case a copy of one of them. Your /etc/localtime matches none of them.

/usr/share/zoneinfo/US # openssl md5 /etc/localtime
MD5(/etc/localtime)= 9323ff44ce205ce1a1bad7e2dca39939
/usr/share/zoneinfo/US # find /usr/share/zoneinfo/ -type f -exec openssl md5 {} \; | fgrep 9323ff44ce205ce1a1bad7e2dca39939
(no match)

so here's what POSIX is telling ICU your time zone is: ( from localtime() vs. gmtime() )

TZ=(null) std=PST dst=PDT daylight=1 offset=32400

daylight=1 means "daylight savings during june-ish" , and offset is 9 hours from UTC.

[santsys]

Is that some sort of new calculation in the newer versions of Node then? because older versions don't exhibit the same issues. I guess what i'm trying to understand is how to correct it as other containers/versions of node don't have an issue.

[srl295]

@santsys I'm not sure about older versions of node. The best way to correct this is to use the TZ variable. Bind mounting /etc/localtime from another distro is bad practice.

That said, after much debugging, I think I figured out the culprit. Since alpine uses the musl instead of glibc library, it's hitting a different spot in ICU4C's zone detection code (not the one i mentioned above). POSIX implementations are inconsistent as to whether the zone offset is applied in localtime. So anyway… there's another spot where an #ifdef MUSL (i.e. Alpine) needs to be made. The comment says that linux-glibc doesn't hit this spot because it exposes __timezone directly. Since MUSL refuses to provide a __MUSL__ macro there's no easy way to add an exception to the code site.

In short, since you haven't set a timezone in any of the usual ways, you've hit a block of code intended for Windows NT compatibility.

[santsys]

OK, very interesting. I'm not sure where exactly that leaves me. But i'll toss the information around and see if I can get some other information on how to proceed. I think i may just be stuck with using older versions of node in this system.

Thank you!

[srl295]

@santsys OK, can you comment on the following?

• just set TZ=US/Pacific or
• ln -s /usr/share/zoneinfo/US/Pacific /etc/localtime

[santsys]

Setting the TZ env variable seems to work fine.

> sudo docker run -it -e TZ=US/Pacific santsys/node-tz-sample:1.0.0 node
> (new Date()).toString()
'Fri Jul 19 2019 12:45:34 GMT-0700 (Pacific Daylight Time)'

I think what may end up needing to be done is to have the TZ environment added to our docker loading system. It appears that having both the mount and the environment variable allows for the correct timezone to be used. There will be a lot of application regression testing that needs to be done.

[srl295]

@santsys if you know what zone you want, you can do both of the bullet items. I would drop the bind mount completely, there's no guarantee that will actually work in any circumstance.

[srl295]

actually, come to think of it, if you bind mounted all of /usr/share/zoneinfo/ not just the one file, it might happen to work here. Then the binary file in /etc/localtime would match something in /usr/share/zoneinfo. I still don't like it because you're using a binary file from one distro against another distro's libc.

[srl295]

Sorry! I filed this whole bug in ICU and fogot to paste it in here: https://unicode-org.atlassian.net/browse/ICU-20724

[santsys]

@santsys if you know what zone you want, you can do both of the bullet items. I would drop the bind mount completely, there's no guarantee that will actually work in any circumstance.

It works in older versions fine, that's the part that's confusing. We will have to investigate properly setting the timezone for the host OS (alpine in this case) and getting that passed to node correctly. As Alpine sees the zone fine using the /etc/localtime mapping, node just translates it incorrectly.

[srl295]

Maybe an old v8 didn't use ICU for Date.toString()/tz info, i'm not sure.

[santsys]

actually, come to think of it, if you bind mounted all of /usr/share/zoneinfo/ not just the one file, it might happen to work here. Then the binary file in /etc/localtime would match something in /usr/share/zoneinfo. I still don't like it because you're using a binary file from one distro against another distro's libc.

And you are correct, mounting both paths seems to also work:

 sudo docker run -it --mount type=bind,source=/etc/localtime,target=/etc/localtime,readonly --mount type=bind,source=/usr/share/zoneinfo/,target=/usr/share/zoneinfo/,readonly santsys/node-tz-sample:1.0.0 node
> (new Date()).toString();
'Fri Jul 19 2019 13:06:55 GMT-0700 (Pacific Daylight Time)'

[srl295]

mounting both paths may be the path (ha) of least resistance for you.

I'm kind of inclined to close this bug, though it's been a fun conversation. This is probably not something to fix or workaround in node.js itself. Feel free to watch the ICU bug.

[santsys]

mounting both paths may be the path (ha) of least resistance for you.

I'm kind of inclined to close this bug, though it's been a fun conversation. This is probably not something to fix or workaround in node.js itself. Feel free to watch the ICU bug.

I agree... I think it's a "bug" but one probably not worth fixing inside of node (someone smarter than I may disagree and if they do I'll bump the ticket, or let them do so).

Thank you very much for the information/getting to the bottom of it!

[srl295]

I think in due time a fix from ICU should make its way into Node.

@bnoordhuis @Trott — thanks for jumping in too. Closing this, see workarounds above:

• (best) set TZ
• (ok-ish) bind mount /usr/share/zoneinfo also