Runtime: TimeZoneInfo should have consistent Ids across Windows and Linux

[eerhardt]

Currently Windows has TimeZoneInfo.Id values like "Pacific Standard Time" and "Eastern Standard Time". But on Linux they are "America/Los_Angeles" and "America/New_York".

We should make these Ids consistent across platforms so the same code can run on both Windows and Linux. Currently if you call TimeZoneInfo.FindSystemTimeZoneById you need to pass in different strings between Windows and Linux.

[Clockwork-Muse]

Some of this is the intention of the System.Time package (among other things).

[mj1856]

I think the right thing to do here is to use CLDR mappings (either directly, or via ICU).

AFAIK, all Windows IDs can be mapped to TZDB Ids. This will let current code run on Linux without modification.

We could also map in the other direction, as _many_ TZDB Ids can be mapped back to Windows zones. Though users would have to consider that it might throw an InvalidTimeZoneException for a valid time zone if it's not mapable.

[mj1856]

@Clockwork-Muse - Since this particular bit is about platform support for the existing TimeZoneInfo class, it can't go in the System.Time package.

[karelz]

We need formal API proposal.
It will be entangled with all the other TimeZoneInfo issue - will likely be non-trivial API design and implementation.

[manigandham]

Some recent discussion here: https://github.com/dotnet/corefx/issues/11897

Seems like a small external library is best suited based on comments to keep up with changes without affecting system packages.

https://github.com/mj1856/TimeZoneConverter

[svrooij]

Not sure if this is the correct place, but maybe it is even a good idea to just support creating a TimeZoneInfo with the required fields. Just a public constructor maybe? Or a not read-only id....

I'm currently using Ews-api-core to talk to Office 365. And I cannot set the timezone to display correctly. That is because the TimeZoneInfo.Id gets serialised as XML into the soapheader incorrectly and Office 365 expects the Windows ID <typ:TimeZoneDefinition Id="W. Europe Standard Time"/> , but my application is running on Unix and this is serialised to <typ:TimeZoneDefinition Id="Europe/Amsterdam"/>.

[mj1856]

@svrooij - Does my TimeZoneConverter library help? If not, can you elaborate for the team as to why? Thanks.

[svrooij]

@mj1856 Thanks for the reply Matt, but that doesn't help. The problem is that the id's on unix are not in the format that Exchange wants. And Exchange uses the offset and the timezone id to set the timezone on appointments. And if the id isn't in the list it recorgnises it just picks a random one with the same offset (and then displays a notice to the user that the appointment is in a different timezone)

[mj1856]

Sure. So you call:

c# string tz = TZConvert.IanaToWindows("Europe/Amsterdam"); // Result: "W. Europe Standard Time"
Working .net fiddle here

You would do this in your code before you pass the time zone on to exchange.

[mj1856]

@svrooij - I found your report at sherlock1982/ews-managed-api#9. They would have to take the dependency on TimeZoneConverter with the above code to make this work seamlessly for you.

[jpenniman]

@eerhardt /@karelz, I'd like to take a stab at a proposal and implementation. This issue (and dotnet/corefx#11897) directly impacts at least 2 of our customers and more as we migrate customer to Core and Linux. Third party packages are a workaround, but doesn't change the fact that TimeZoneInfo isn't truly xplat/portable, since consuming code can't "build once, run anywhere".

Is there a formal API proposal template I should use? Who would I submit it to?

[eerhardt]

@jpenniman - thanks for your interest. Please check out the API review process doc for information on how to make an API proposal.

My original hope was that the existing APIs could be made to work with both sets of time zone IDs and not require new APIs. But thinking more about how this would actually work, it may not be possible, or ideal. So if you have ideas on new APIs that would help here, please feel free to submit them.

[tarekgh]

@jpenniman the challenge here is we are trying to avoid carrying the TZ data inside the framework. the reason is that will required to have some changes inside coreclr every time the TZ data get changed. so I would suggest looking at @mj1856 TimeZoneConverter and how to make it xplate and package can be used. consistently. we have other TZ things need to be done so I prefer to have a complete plan for everything together instead of picking some items here and there to do.

[jpenniman]

Thanks @eerhardt , I'll take a look.

@tarekgh, I agree, TZ data in the framework would be challenging and not ideal. However, it also makes the code not portable. We have customers with hundreds of services that will break when we move to Linux. I'd rather see if there's a way to fix TimeZoneInfo. Matt's TimeZoneConverter is a workaround, but has the same maintenance problem--now I have to rely on him to keep the package up-to-date instead of Microsoft. Using a workaround isn't just a couple lines of code in a service in my case... it's thousands of lines across hundreds of services, and in a few cases, data migrations. I'm not saying the TZ data has to be in the clr, just that, ideally, it would be nice if, the TimeZoneInfo api works the same xplat with the same values. IOW, if I call TimeZoneInfo.FindSystemTimeZoneById("Eastern Standard Time") on Linux, it "just works".

This is a major piece in several lift-and-shift efforts we have lined up. There's a lot of interest to move to core and Linux, especially among our AWS customers, because it reduces compute consumption costs by 60% and gives them all the managed features that are lost when running Windows on AWS.

I'll take a look at Matt's stuff as well as NodaTime and see if there's a happy medium--compatible API without TZ data in the clr.

Java ships a TZ database with JRE (a separate file, not embedded) and is updated separately from the JRE itself (using tzupdater). I'll take a look at how Apache is handling this in the OpenJDK. Maybe we can borrow some ideas from them, since Java already solved the platform differences.

[Clockwork-Muse]

What Oracle _recommends_ is to get updated timezones by updating the JRE/JDK, but provides the ability to update timezones via their tool. It's a binary file that's been loaded via, assumedly, the default implementation of a specialized Dictionary interface (essentially).

NodaTime essentially has its own interface, they just make you load the file manually, and use DI, if you want a different version.

Presumably you could have a nuget package that comprises a pre-built resource file (plus one dependency for the necessary interfaces, allowing people to do things like have network-based ones or whatever).

---

I say this, but updating timezone rules is a hard problem if you've ended up persisting any of them somewhere; scheduling apps can't be updated independently from their datastore, for example (which is why I mentioned a network-based database, although attempting to do this live is an even harder problem)

[tarekgh]

@jpenniman I fully agree with you that the framework should solve this problem. the only thing is we need just some time to bake the whole space and provide all missing functionality. One of the ideas we have but not fully investigated is to have TZ providers that can plug into TimeZoneInfo and can support providing data and possibly code for some of the special TZ calculations.

[mj1856]

So the key problem with regard to the xplat issue is that someone has to provide the mapping data from CLDR. It could be the framework, or a library like mine, or the operating system, or some other component on the operating system. But it is indeed data dependent, and there's no getting around that.

AFAIK, there do not exist OS-level APIs for conversion between IANA and Windows time zone IDs on Windows, Linux, or Mac. So that idea falls flat.

One possibility would be to use ICU for the conversions - if ICU is available. But then what to do when it is not?

One could also use an online service to perform the conversions, but again - what to do when offline or when the service is unavailable for whatever reason?

If we go down the path of having the framework itself perform the conversions, then the framework has to provide and maintain the data. It revs in an ad-hoc and unpredictable manner, and sometimes those changes are considered urgent by some. So keeping it in a separate component makes the most sense IMHO. The best we could do with plug-ability would be to reduce the API calls needed. It wouldn't change the nature of having to update the components.

[jpenniman]

I agree providing at least the database as a separate package would allow for easier updating rather than embedded into each platform runtime. Someone would need to keep it up-to-date -- perhaps an automated process that checks for a new version daily and builds a new package if there is. I wouldn't rely on a human to do it. @mj1856, even your TimeZoneConverter library is a version behind as of this post.

Assuming we can solve the data distribution how should the API behave? My initial thoughts are all the APIs on TimeZoneInfo work with the Windows version of the zone ids. For example, TimeZoneInfo.FindSystemTimeZoneById(), would only see "Eastern Standard Time" as valid and return an instance with that Id. That enables migrations, though would break any existing born on Linux/Mac projects. It's not much different from Java, in that the same database is used across all platforms--whether you're on Windows or Linux, you have to use "America/New York" in java. Microsoft conveniently already has that database and keeps it up-to-date ;)

Another idea is to combine them. So, GetSystemTimeZoneIds() would return the full set of Windows and IANA id's combine. Then, TimeZoneInfo.FindSystemTimeZoneById() would behave similar to Matt's TZConvert.GetTimeZoneInfo() method, returning the zone for any valid id regardless of platform.

In addition to standardizing TimeZoneInfo, perhaps a new set of timezone APIs could be part of the new System.Time package which work with the IANA standard, since that's what most other platforms and languages have adopted. These apis could offer conversion methods similar to Matt's TimeZoneConverter.

Thoughts?

[Clockwork-Muse]

...Any package distributed should be completely independent from whatever the system reports as its timezone, since there're ways to create your own (including on Windows); you're only looking _in that database_. Done correctly, there may be other time zone databases supplied, like for IATA, which people are unlikely to be using for their local systems.

The question then is how you get from the system time zone (assuming you actually care - many applications are going to be on a server and should absolutely not care) to a "proper" one (note you should be doing this even if the ids are the same, since otherwise the _rules_ might be different). At that point you'd want some sort of system-to-database-mapper, probably as another package (although this one should change far less frequently)

[jpenniman]

...Any package distributed should be completely independent from whatever the system reports as its timezone, since there're ways to create your own (including on Windows); you're only looking in that database.

I agree. This is how Sun did it and how Oracle continues to do it. The java apis only use their shipped database and not the system's, regardless of the system. My thought was, for compatibility/portability, the database contains both "Windows" timezone ids AND IANA timezone ids all in one (or at least appears to from the API side of it). That covers 100% of the current OS's that support coreclr. So, regardless of the system, if I call TimeZoneInfo.FindSystemTimeZoneById("Eastern Standard Time"), it will resolve, whether I'm on a Windows server, CentOS server, Mac Server, etc. Likewise, if I call TimeZoneInfo.FindSystemTimeZoneById("America/New York"), it will resolve on Windows, Linux, Mac, etc.

I like the idea of making that database plug-able, so devs could use any database they choose.

assuming you actually care - many applications are going to be on a server and should absolutely not care

I disagree. That's whole reason this issue and related ones were filed--the application _does_ care because it was initially built on Windows/Microsoft ecosystem and is being ported to, or integrating with new services hosted on Linux. Every calendaring service, including Exchange and Office 365 care very much about the timezone id. One of our customers has a business rule that time cards are due by Tuesday at noon, in the local timezone of the branch. Therefor, we have to store the timezone (and offset) of the branch.

We always use UTC server side. In that respect, yes, timezone doesn't matter. But business doesn't run on UTC, it runs in the local timezone. I agree that in the simple example, the server uses UTC and the client uses local time, but unfortunately, many scenarios are far more complex than that.

[Clockwork-Muse]

I disagree. That's whole reason this issue and related ones were filed--the application does care because it was initially built on Windows/Microsoft ecosystem and is being ported to, or integrating with new services hosted on Linux. Every calendaring service, including Exchange and Office 365 care very much about the timezone id. One of our customers has a business rule that time cards are due by Tuesday at noon, in the local timezone of the branch. Therefor, we have to store the timezone (and offset) of the branch.

Sorry, my point wasn't clear. What I meant was, in almost no case is the actual timezone of the server itself relevant, but the timezone of the entity under view. So yeah, storing and retrieving the timezone (and offset, because of rule changes) of the branch is what I'd expect to happen. If you meant "timezone database of the server", that would have made more sense

We always use UTC server side. In that respect, yes, timezone doesn't matter. But business doesn't run on UTC, it runs in the local timezone. I agree that in the simple example, the server uses UTC and the client uses local time, but unfortunately, many scenarios are far more complex than that.

Hm, personally I think applications should work correctly regardless of the current timezone of the machine they're running on (UTC or otherwise), but that gets tricky in client mode. Or more specifically,
consumer client mode, since most corporate client machines should be pulling "business unit timezone" from a central config server somewhere, if it's relevant (since it's likely any central processing would also have to take that same timezone into account, too).

[Clockwork-Muse]

Also, and this didn't hit me until later, we would absolutely NOT want to just pull the Windows zones in from the host box: that would be the fastest way to break an app running on both a Linux and Windows machine and using Windows zones. Consider what happens if the app is updated with the new package version, but the OS isn't, or vice versa. OS-level resources may not even be under programmer control, in some cases (ie, Functions as a Service). If we're doing a resource package, it's going to have to control _all_ its resources.

[mj1856]

@jpenniman - A version behind? If you mean WRT IANA version, no new time zone IDs have come out, and mappings are the same from CLDR.

[mj1856]

WRT the system's local time zone - that is a separate topic, out of scope for this discussion. The time zone data is indeed installed on the system, but we're not talking about what the time zone the system is set to (whether that's custom or standard).

WRT pulling in Windows zones from the host box - That is exactly how the TimeZoneInfo.FindSystemTimeZoneById API works, and has been working since it was created back in .NET 3.5. The Linux version is no different in that regard, it's just pulling in different data.

For that matter, .NET Core has already shipped with pulling Windows IDs on Windows, and IANA IDs on Linux/Mac. Changing that would be breaking existing code.

IMHO, there are really only two viable paths:

• Make TimeZoneInfo.FindSystemTimeZoneById take either form of identifier - essentially merging my library in. Somehow provide data updates. This takes care of the "make it just work" factor.

• Create some sort of "provider model" wherein one can supply the time zone data they care about instead of getting it from the OS.

I think both of these could co-exist, but I think this issue was primarily focused on the first one.

[exiled78]

For the record, I thought this might work for an app developed on windows, but deployed to AWS Lambda Function. It works, however, the catch code does not accommodate daylight savings. The windows code correctly gives the offset as +11, the catch code incorrectly gives +10. Note the tz.IsDaylightSavingTime() incorrectly returns false on linux.

                TimeZoneInfo tz;
                DateTimeOffset offset;
                try
                {
                    tz = TimeZoneInfo.FindSystemTimeZoneById("AUS Eastern Standard Time");
                    offset = dateTime.ToOffset(tz.GetUtcOffset(dateTime));
                }
                catch (TimeZoneNotFoundException)
                {
                    //Should work on linux
                    tz = TimeZoneInfo.FindSystemTimeZoneById("Australia/Sydney");
                    offset = dateTime.ToOffset(tz.GetUtcOffset(dateTime));
                }

[tarekgh]

the catch code incorrectly gives +10. Note the tz.IsDaylightSavingTime() incorrectly returns false on linux.

This looks a bug and needs to be investigated.

[Clockwork-Muse]

@exiled78 -
If you're developing on Windows, is WSL (Windows Subsystem for Linux) available? Granted, I don't know what AWS's test platform looks like, so I don't know if that's even possible, but that might allow for non-Windows zone ids. At minimum, though, I'd probably switch which is the "common" branch, since that function is going to be called far more often once deployed.

Keeping timezones in sync across deployed resources is... tricky. Sometimes I think I'd prefer to manually manage that as a resource dependency (because then, if it was off for a system, at least everything would be off the same way), instead of letting it bubble up from the hosting provider. That gets worse the more different types of systems get involved, since they may each have their own update tempos for the information.

[eerhardt]

@exiled78 - what version of Linux are you on? On my macOS High Sierra, it correctly returns +11. Is it possible the tzdata is out of date on that Linux machine?

Update: I also tried it on my Ubuntu 16.04 machine, and it also printed:

3/15/18 2:52:12 AM +11:00

[mj1856]

@eerhardt - TZDB data for Sydney hasn't changed in a long time, so I doubt it's that.

@exiled78 - Is it possible two different dates were being passed in the dateTime parameter, or that their .Kind properties were different due to how they were initialized? If not, what was the value that was passed? Was it something very old or far distant (like DateTime.MinValue or DateTime.MaxValue?)

For reference, note DST in Sydney is not the same at all as in the US
https://www.timeanddate.com/time/zone/australia/sydney

[jpenniman]

@mj1856, yes, I was referring to the new 2018 database being out and your library still on the last 2017 update. If there are no changes, then it's moot.

[jpenniman]

@mj1856, I agree on your two options:

IMHO, there are really only two viable paths:

• Make TimeZoneInfo.FindSystemTimeZoneById take either form of identifier - essentially merging my library in. Somehow provide data updates. This takes care of the "make it just work" factor.
  
  
  
  • Create some sort of "provider model" wherein one can supply the time zone data they care about instead of getting it from the OS.
  
  

I think both of these could co-exist, but I think this issue was primarily focused on the first one.

That solves TimeZoneInfo.FindSystemTimeZoneById and the various convert methods without breaking any existing code and allows portability, since you'll always get the "right thing" regardless of what's passed in.

That just leaves TimeZoneInfo.GetSystemTimeZoneIds(). I suppose it could just return what ever is on the system. We have customers with apps that return that list to populate timezone selection in the UI, but I suppose that could be re-written or the users could just deal with it. In a perfect world, for me, the experience of using the API would be exactly the same on any platform, which today it is not. Your second point of having a plug-able database solves that.

Greenfield projects specifically built for one platform or the other work just fine. The challenge I, and others, are faced with is migrating from one system to another, or creating cross-platform apps. My customers are currently faced with having to rewrite code because of this difference (and a lot of it).

[moodyan]

Any update on this? I'm having the same issue where it works on my local Windows machine but when I deploy it to AWS Lambda it no longer works. Either of the options mentioned by @mj1856 would be helpful!

[tarekgh]

@moodyan no updates yet but I am wondering doesn't @mj1856 library https://github.com/mj1856/TimeZoneConverter help in your case?

[moodyan]

Yes, shortly after commenting, I reread the comments and found that library, which definitely helps. Thanks!

[ericsampson]

@mj1856 I'm surprised this is still open given the ease of moving back and forth between Windows and Linux hosts in Azure, is this still going to get addressed without needing to pull in your non-platform TimeZoneConverter library? THanks!

[ericsampson]

I was just thinking as well, how would you use TimeZoneInfo.FindSystemTimeZoneById in an Azure Function where you have no control over what platform is running your code?

[Clockwork-Muse]

@ericsampson -
You have worse problems than just Linux VS Windows. You also have to worry about what version of the timezone rules are you using. As the hosts/instances get updated, you're going to get tearing as to what it sees as the "current" rule. This is an inherent race condition, and isn't really specific to a functions-as-a-service platform (although that exacerbates the issue) - it's happening because you're updating a running application.
For instance, this warning in java.time.zone.ZoneRulesProvider:

Many systems would like to update time-zone rules dynamically without stopping the JVM. When examined in detail, this is a complex problem. Providers may choose to handle dynamic updates, however the default provider does not.

And a similar statement for NodaTime.TimeZones.IDateTimeZoneSource:

The interface presumes that the available time zones are static; there is no mechanism for updating the list of available time zones.

Your biggest problem, though (tearing updates across multiple instances being a minor issue), is serialization: if you previously stored a future date with one set of rules, you now have to update it to match the new rules (and usually notify any affected customers). Note that you can't just store in UTC, because most uses of future dates are anticipated and used as local dates, within one timezone - meetings, doctor appointments, etc. Oh, and if this is in a database, and you were using built-in database functions to handle timezones there, you have another zone source that might be different.

---

Note, however, this is all only relevant when dealing with _future_ dates (....to some degree of tolerance, at least).

• For logging, the timezone is entirely irrelevant, as is UTC. You want an Instant (which are often serialized as UTC for historical/solution domain reasons).
• For past local dates, the rules should not be changing.

[cjbush]

For what it's worth, I appear to also be having @exiled78's problem when running on Linux containers. The following code:

DateTimeWithZone expirationDate;
try {
  expirationDate = new DateTimeWithZone(request.ExpirationDate.Value, TimeZoneInfo.FindSystemTimeZoneById("America/New_York")); //This is for Linux
}
catch (TimeZoneNotFoundException) {
  expirationDate = new DateTimeWithZone(request.ExpirationDate.Value, TimeZoneInfo.FindSystemTimeZoneById("Eastern Standard Time")); //This is for Windows
}
Console.WriteLine(expirationDate);

correctly produces UTC-5 on Windows, but produces UTC-4 on Linux. All DateTimeWithZone is doing under the hood is calling TimeZoneInfo.ConvertTime(dateTime, timeZone).

[tarekgh]

@cjbush could you please try to print the values of the created time zone (e.g. BaseUtcOffset and calling IsDaylightSavingTime on request.ExpirationDate.Value). Also, print the value of request.ExpirationDate.Value on Windows and Linux with the DateTimeKind to ensure you have exact same date/time object.

[cjbush]

@tarekgh I just checked and IsDaylightSavingTime was indeed coming back as true. And then I realized that's because the value for the date coming back for my request was the day after DST kicks in. So tl;dr I'm an idiot who doesn't know when daylight savings starts and the API works fine. I wonder if that was exiled78's original problem, since it occurred around the same time last year.

[CShepartd]

There was many proposals how to slove this problem. After marker it as 5.0 i preassume you picked one of them. Wich one?

[tarekgh]

There was many proposals how to slove this problem. After marker it as 5.0 i preassume you picked one of them. Wich one?

It is just moved to milestone 5.0 to look at it during then. It is not decided yet which direction we'll go with.

[mynkow]

Greetings from mid 2020.

Hala

[molekm]

Still waiting on something so crucial to be solved especially with Docker Linux/Windows.

[tarekgh]

@molekm can't you use the library https://github.com/mj1856/TimeZoneConverter?

[timmydo]

It would be nice if something like DOTNET_SYSTEM_GLOBALIZATION_APPLOCALICU (from https://docs.microsoft.com/en-us/dotnet/standard/globalization-localization/globalization-icu) existed for the tz data. TimeZoneConverter would only be a solution for people that can rewrite their code to point to the that library.

[udlose]

@molekm can't you use the library https://github.com/mj1856/TimeZoneConverter?

@tarekgh With all due respect, we shouldn't have to use an external library to do something so fundamental as working with TimeZones. I had hoped that the abstraction of "concern of which OS my app is running on" should be handled by .NET Core as that was one of the main goals of the .NET Core platform. Very disappointed that this issue has been opened for a little over 5 years now with little more than "...use an external library to do this" as a solution.

Can you please provide an update on where this is in progress? .NET 5.0 is slated for release in November 2020 which is less than 3 months away now.

[tarekgh]

@udlose I agree with you in principle. The issue here we were trying to avoid in the .NET is to carry Time zone data which will need to get updated when something change. This create a nightmare in servicing old versions of the .NET especially time zone is a core functionality. That is why we were planning to get into the point to depend on something that we can use and not require servicing the .NET.

In 5.0 we started to use ICU library for globalization support but this is huge work as globalization is kind affecting most of the basic functionality in the .NET. We are trying to stabilize the product with using ICU. When I suggested using TimeZoneConverter it was a temporary till the framework introduce the functionality.

We have the plan in 6.0 to look using ICU which we already integrated in 5.0 to do the TZ Ids conversions and possibly introduce more better interfaces for time zone in general. I hope this clarify why we delayed this work but I am emphasizing this is in our near future plan to get it done. Thanks for your understanding.

[ericsampson]

@tarekgh thanks for the update. It's not a small task to do 'well'. I'm happy to see the ICU changes coming in 5.0

[udlose]

@tarekgh Thank you for the update and I do appreciate the challenge you face. I am happy to see there is a plan for it in the near future and I'm looking forward to leveraging it. In the meantime, I will use the TimeZoneConverter you suggest.