Chakracore: Fix timezone-sensitive tests (disabled in Jenkins)

[dilijev]

There are a handful of tests which by their nature are timezone-sensitive (testing Date APIs). Because the execution of some Javascript Date APIs is dependent on the system time zone, the tests only work when run in the time zone in which the baselines were created (in our case, Pacific Time [UTC-8]).

The Jenkins machines are configured with UTC time and so we noticed that these tests would fail in any other timezone. To work around the problem for now, we disabled these tests in Jenkins.

It would be nice to add a way to enable these tests.

My thoughts:

• Some of these tests may not need to depend on the time zone. If possible, modify the tests to make them more general (as long as we keep exercising the same code paths).
• Add a flag to the test runner (rl.exe) and/or ch.exe which provides the ability to spoof the system timezone for tests, so that we can test the right code paths within ChakraCore which do depend on the system's reported time zone.

To locate the affected tests which were disabled in Jenkins, you can run:
git grep -n -C 3 "timezone-sensitive"

/cc @digitalinfinity @tcare @ianwjhalliday @kunalspathak FYI

Converted from VSO 5239718

[tcare]

This is something that we've been meaning to fix for a while. @ianwjhalliday encountered some pain over this when he was working remotely a while back :)

[ianwjhalliday]

@akroshg noticed it too on when he was working remote recently.

Afaik there is no way to change the timezone per process for easy spoofing. The system's timezone can be changed but that would affect new processes and not be a robust solution in the face of concurrent testing via rl.exe (it would work if all tests used the same timezone for spoofing, which they do currently, but it would lead to race condition bugs if new tests used other timezones). It's also not friendly to the user.

Spoofing if possible would be best done in the engine itself, in the date and time APIs.

[ianwjhalliday]

One test I ran into was using a Date to get a long string for use as a property name. It didn't depend on the date itself, nor the timezone, so I hardcoded the string that new Date(...) produced to make it work across timezones.

[dilijev]

@tcare Yep I had a bug open internally from when we set up Jenkins and I noticed this originally. I decided to move it here since it seems to be ChakraCore-specific. I should have added a link. Added that now.

@ianwjhalliday Good catch. Were you able to re-enable that test for Jenkins?

[dilijev]

@ianwjhalliday Yes, it seems there's no way to do it from the OS but in test mode we could have a flag on ch.exe override what the OS would say about the timezone and have the Date APIs use the provided timezone if possible. This effectively means we won't test the code path which uses the OS APIs to retrieve the timezone. (We could duplicate the tests and have one version spoof the timezone and one use the default behavior, but the one that doesn't spoof would still need to be disabled in test configurations not known to conform with the expected timezone.)

Having the user change their time zone isn't really a workable solution (too much of a hassle), and I don't think there's a way to automate a Jenkins task to do that for those test machines either. I guess I could use a powershell script as a build task which would set the timezone but that could disrupt other jobs if they depend on Jenkins machines being UTC. If a build failed Jenkins would kill the job right away thus offering no guaranteed way to clean up and reset the timezone.

This makes me wonder if any tests are locale-specific. I haven't noticed that problem yet but none of our test machines are outside of the US locale.

[ianwjhalliday]

@dilijev I don't remember which test it was, but looking at the grep results for timezone-sensitive it doesn't look like it was any of those. I might have fixed it before jenkins happened.

And yes, agreed, that is what I was saying about spoofing the timezone.

[jianchun]

Timezone-sensitive tests can spoof current timezone themselves and handle timezone differences internally instead of exposing and printing to output. Just run one API whose output contains timezone and spoof from it; do all comparisons within script with timezone as a variable; only print pass or failure details.

[dilijev]

This sounds reasonable to fix the tests, however, there's some concern that testing something other than "default" behavior reduces code coverage.

[obastemur]

We don't have timezone sensitivity problem on *nix. We could set the the timezone on CH for Windows process with a command line argument (i.e. ---set-timezone-for-testing...) or preferably we could do this on python (custom implementation).

[dilijev]

Also FWIW +1 to doing all comparisons in-script. Unless there's a need for baseline outputs (there are some actual cases where it is necessary), we should be doing all tests with in-script comparisons.

[dilijev]

@obastemur Definitely support doing such a custom implementation. Might be the impetus we need to kill off rl.exe and transition completely to the python-based test runner. /cc @tcare

[jianchun]

@dilijev That approach doesn't change default behavior. It tests the same default (time zone), just instead of printing it compares within script the expected result in that time zone. Should be the same code coverage (or better, as the tests able to run in different time-zone settings).

[dilijev]

@jianchun Good point. I thought you meant to do something like use Intl.DateTimeFormat with a specific timezone to format the string.

We should also test the toString behavior but we can have the test check the formatted pattern with a regex instead of matching exact text.

[dilijev]

/cc @aneeshdk