Zig: Prefer depending on NtDll rather than kernel32 or other higher level DLLs

[daurnimator]

Proposal Clarification

---

"windows" as most people think of it is a stack of

• the nt kernel
• ntdll.dll
• kernel32.dll
• user32.dll

You don't have to use all elements of this stack, and there are situations where you want to avoid or are not able to use kernel32.dll and/or user32.dll.

The Zig standard library should not assume that kernel32.dll and user32.dll are unconditionally available.

[andrewrk]

there are situations where you want to avoid or are not able to use kernel32.dll and/or user32.dll.

What situations are these? Does anyone know?

This is an important proposal if we have some use cases to guide it.

[suirad]

Some background for anyone not familiar: ntdll.dll is the only constant that is required for a windows userland process, as its what does the syscalls. kernel32.dll mostly just wraps ntdll and provides what is colloquially known as Winapi. user32 is essentially the kernel32 of Windows gui stuff.
The windows userland stack ends up looking like:
winlibc (aka fopen()) -> kernel32 (aka CreateFileA()) -> ntdll (aka NtCreateFile()).

---

A couple examples of when some of these may not be available/wanted:

• In Windows Kernel/Driver development those kernel32.dll, ntdll.dll and user32.dll aren't available.
• In some position independent code that is loaded/unpacked/de-obfuscated from elsewhere

---

I would also like to propose having a way to tell what version of windows is being used. Something like builtin.windows_xp / builtin.windows_vista because each changes how some winapi are expressed.

An example of how this would be immediately useful is in creating std.mutex where windows xp and above has CriticalSections available to use, windows vista+ enables use of SRWLock and windows 8+ has WaitOnAddress available, which is the closest thing to a futex windows has. This would result in the mutex itself resolving which of these would be best to use at compile time rather than at runtime.

[jayschwa]

Windows XP

Microsoft stopped maintaining XP nearly 5 years ago. What is Zig's policy on supporting old operating systems?

[andrewrk]

What is Zig's policy on supporting old operating systems?

I don't think it's relevant to the language. But the standard library does not support Windows XP, since it's not supported by Microsoft.

However anyone is free of course to create a third party package for interfacing with Windows XP. And note that much of the standard library has no OS dependencies, and thanks to lazy analysis, you can therefore use the parts of the standard library that have no OS dependencies even when targeting an unsupported OS.

[PavelVozenilek]

There's one very specific use case, software hardened against tampering, where people "include" these standard DLL's (their code sections) inside the executable.

However, this (and driver development) is something very rare, and there's always present danger of feature creep, under-documentation, untested/untestable functionality.

[andrew-boyarshin]

Proposal

OS

• nt: kernel-mode/drivers
• winnative: native images (like _smss.exe_, _csrss.exe_, _autochk.exe_), user-mode with NTDLL only
• win32: user-mode Win32 subsystem images, with guaranteed existence of NTDLL, KERNEL32 and KERNELBASE (since Win10).
• windows => win32

libc

• msvc: Windows-only
• ros: ReactOS-based
• gnu: MinGW-w64
• wine (with wine-staging)

Compatibility

libc might impose additional dependencies.

• wine - msvcrt depends on KERNEL32, so supported only with win32 OS
• ros - full support
• gnu - not sure about nt and winnative

[andrewrk]

I would also like to propose having a way to tell what version of windows is being used. Something like builtin.windows_xp / builtin.windows_vista because each changes how some winapi are expressed.

This is #1907.

---

I'm accepting this proposal, and to provide further clarification of what such an acceptance looks like, it means:

When the standard library needs to interact with the Windows kernel, if there is ntdll API which provides the necessary components to perform the task correctly, then that is the preferred API to use. Generally, if another DLL such as kernel32 is wrapping ntdll functionality, Zig std lib should prefer NtDll directly.

Once we shave down the non-NtDll dependencies in the std lib, let's see what we have left and re-evaluate from there.

One strategy that can be used to find this information out is ProcMon. ProcMon can reveal when a higher level DLL is calling into a lower level one.

[daurnimator]

Generally, if another DLL such as kernel32 is wrapping ntdll functionality, Zig std lib should prefer NtDll directly.

To me this is like saying that on linux, we should prefer the raw syscall over libc functionality: should we be doing that too?

Also, I'm don't think all ntdll functions work correctly under wine; so that might make things harder for e.g. developers on linux to test.

[andrewrk]

To me this is like saying that on linux, we should prefer the raw syscall over libc functionality: should we be doing that too?

In some places this is true, for example with futexes. But generally, Zig users have a choice of which layer to target, by choosing whether to link libc. When linking libc, it makes sense to target the libc layer, for a few reasons:

• saving code size in some cases, since the C code must be linked anyway
• necessary for features such as glibc nss to work
• linux users might expect to be able to use LD_PRELOAD to override libc functions. This is how valgrind integrates with std.heap.c_allocator, for example.

However, not targeting the libc layer is also a primary use case, when not linking libc.

Also, I'm don't think all ntdll functions work correctly under wine; so that might make things harder for e.g. developers on linux to test.

The Zig project has already managed to get one bug fix upstreamed into Wine. Let's start with the optimistic attitude of improving the open source communities around us, before we give up and make compromises.

Now, if there are reasons to target kernel32 or other higher level DLLs, let's hear those reasons out. If they are compelling enough, then it probably makes sense to add an additional target configuration option, which is available to observe in std.builtin, and the std lib can decide which layer to prefer. But I'm not yet convinced this is desirable.

[adontz]

I am sorry, if I'll sound a bit harsh, but I have found a few false statements in this thread.

Linking to ntdll instead of kernel32 (as I see, now it is a weird mix) provides no benefits. ntdll is available in kernel mode and user mode, but exports different set of functions for different modes. For kernel mode, i.e. from driver, one will call ZwCreateFile and not NtCreateFile, so exporting user mode ntdll functions for means of driver development is simply a mistake. Such std will not be more useful, useful in more scenarios or something liek that. It's still user-mode only regular application case. Not a kernel driver, not early boot service, nothing like that.

kernel32 is always available to user mode application, it is loaded into process address space even if not linked at all. There are no reasons to avoid using kernel32, because it is always available. kernel32 is not a simple wrapper, function signatures are quite different and calls do not always map one to one. Those who link directly to ntdll will have to reimplement all kinds of argument conversion logic, or deal with all kinds of weird behavior.

Also, while Microsoft is champion of backwards compatibility (no irony, they are great), reading documentation gives us pretty straightforward directions

https://docs.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntcreatefile

Note Before using this function, please read Calling Internal APIs.

https://docs.microsoft.com/en-us/windows/win32/devnotes/calling-internal-apis

The functions and structures in Winternl.h are internal to the operating system and subject to change from one release of Windows to the next, and possibly even between service packs for each release. To maintain the compatibility of your application, you should use the equivalent public functions instead. Further information is available in the header file, Winternl.h, and the documentation for each function.

I do not know how can it be more clear that Nt-prefixed functions should not be used by regular user mode applications. Even if signatures match, parameter meaning or set of valid values may change. Drivers will use different (Zw-prefixed) functions anyway. So there are literally no reasons to avoid kernel32.

Also, paths are not compatible and it's not about maximum allowed length or some fancy prefix. "\Device\Harddisk0" is a valid NT path, but not a valid Win32 path with or without any prefix. There are many obvious and even more subtle differences between NT and Win32 behavior, so user mode std depending on NT calls is a really weird choice.

I can't remember a single language standard library (C, rust, D, .Net) to use these functions, and I've dissected many of them.

[andrew-boyarshin]

@adontz

1. Kernel mode can also call Nt* versions. Moreover, it is preferred to when data comes from user mode and cannot be trusted.
2. NTDLL is not used in kernel mode, because NTOSKRNL provides most of the NTDLL's API. Technically there is a common part (RTL) linked into both, difference part (for the function with Nt* name NTDLL usually does syscall, NTOSKRNL actually implements the function), plus a vast number of unique API's in both.
3. KERNEL32 and KERNELBASE are not present in Native Executable image scenario, although it's hard to deny that scenario is rare.
4. About DOS and NT paths: it depends on the function really. Some NTDLL functions expect DOS paths and convert them to NT paths internally before doing syscall, some expect already converted NT path.

Honestly, the proposal I've written might be good, but I've already decided against using Zig for low-level Windows development, so I don't really care if Zig stays user-mode only.

upd:

1. .NET depended on a lot of internal functions around the 3.0-3.5 timeframe. Later they migrated to stable APIs, or, where applicable, just started to bring the code from Windows into their own and adapting to their needs.

[daurnimator]

kernel32 is always available to user mode application, it is loaded into process address space even if not linked at all

Ah, appears they added that in XP for the windows subsystem. Most of the books on windows internals are from the Win 2000 era :(
Still, writing native subsystem programs is useful and interesting. midipix has a few examples.

kernel32 is not a simple wrapper, function signatures are quite different and calls do not always map one to one

That's sort of the issue: lots of kernel32 wrappers don't expose the full functionality of the NT API. From the basic stuff like missing flags, to different concepts of paths (nt paths are not null terminated, but length delimited resulting in end users needing tools like RegDelNull) to just plain missing functions (e.g. did you know nt supports fork... but kernel32 doesn't).

Also, while Microsoft is champion of backwards compatibility (no irony, they are great), reading documentation gives us pretty straightforward directions

https://docs.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntcreatefile

Note Before using this function, please read Calling Internal APIs.

https://docs.microsoft.com/en-us/windows/win32/devnotes/calling-internal-apis

The functions and structures in Winternl.h are internal to the operating system and subject to change from one release of Windows to the next, and possibly even between service packs for each release. To maintain the compatibility of your application, you should use the equivalent public functions instead. Further information is available in the header file, Winternl.h, and the documentation for each function.

I do not know how can it be more clear that Nt-prefixed functions should not be used by regular user mode applications. Even if signatures match, parameter meaning or set of valid values may change.

There have only been 2 notable function changes I know of since windows 2000, and they have coincided with major releases: despite Microsoft's statements, they are quite stable. What is unstable is the actual syscall numbers, which change even between monthly windows updates.

[andrewrk]

@adontz, I appreciate your input.

Linking to ntdll instead of kernel32 (as I see, now it is a weird mix) provides no benefits.

There is one very real benefit: the APIs are more powerful. For example, NtCreateFile has the ability to open a sub-directory path based on an open directory handle. The kernel32 APIs do not expose this. It has meaningful consequences in terms of avoiding file system race conditions.

When I first started working on Windows implementations of cross platform abstractions, I took the Microsoft documentation very seriously. But then I found out that pretty much every standard library uses SystemFunction036 from advapi32.dll for getting cryptographically secure random bytes, while Microsoft's docs says to use CryptGenRandom. The docs for CryptGenRandom now say

Important This API is deprecated. New and existing software should start using Cryptography Next Generation APIs. Microsoft may remove this API in future releases.

What I learned is that it's more important to consider ABIs rather than APIs.

Microsoft won't break RtlGenRandom and they won't break NtCreateFile. If they do, they have a lot more programs to worry about breaking than Zig binaries.

The bottom line here is that NtDll is lower-level ABIs to the kernel, and kernel32 is higher level code that wraps NtDll. This is clear when you look at calls in ProcMon. And frankly, Zig's open-source code that wraps NtDll is more robust and reliable than Microsoft's cruft inside kernel32.dll. I don't remember the specifics, but off the top of my head, there are functions in kernel32 that wrap NtDll functions that allocate heap memory, whereas the zig code that wraps NtDll functions does not. That can be the difference between deterministic latency or nondeterministic latency.

At this point I'm convinced that NtDll is an entirely appropriate abstraction layer for windows applications to target. My mind is open to counter-examples, but calls to NtDll functions have been in the zig std lib for quite some time now, and I'm not aware of a single issue it has caused. On the other hand, it has allowed us to unify the std.fs API across POSIX and Windows, since with NtCreateFile they both support operating on an open directory handle.

@andrew-boyarshin,

I've already decided against using Zig for low-level Windows development

Is there anything the Zig project can learn from your decision?

[andrew-boyarshin]

@andrewrk note that the following is purely about Windows kernel drivers, ReactOS kernel and NTDLL, user-mode Zig is cool, I play with it (but yet to use it in any _real_ project).

1. I'm not sure Zig would allow me to develop kernel driver code (standard library)
2. I'm not sure Zig would allow me to throw and handle SEH exceptions properly (ExceptionCode, user buffer probing, exception filters)
3. I'm not sure about calling conventions being compatible, but that might well be false. I've never tried Clang/LLVM for low-level code, and Zig is 2 layers where mistakes can exist. Clang+LLVM is 1.5 layer (Clang is very coupled with LLVM, so it's difficult to count them separately). I just feel MSVC is a safer (as in _error-proof_) bet (single layer monolith used by Windows team themselves).
4. I'm not sure I can rely on Zig features being safe (as in _secure_) and fast (as fast as they can theoretically be when implemented in C using kernel APIs to the fullest). I'm sure writing C-like Zig will be ok, but when it comes to coroutines and complete kernel asynchrony (can they survive preemption? how will they play with IRQL?)...

I feel like most of these issues can be eliminated by providing a good sample of e.g. FS/registry filter, and showcase what works and what does not.

upd:

1. Status codes. Zig has an awesome concept of static errors and obligatory explicit handling. Unfortunately it's impossible to use that since even MS Docs don't specify all possible NTSTATUS codes, not to mention non-documented functions. So, Zig offers no advantage in this respect.

[daurnimator]

I feel like most of these issues can be eliminated by providing a good sample of e.g. FS/registry filter, and showcase what works and what does not.

Is this something you'd like to take on @andrew-boyarshin ?

[andrew-boyarshin]

@daurnimator not in the foreseeable future. Certainly not in the next 5 months.

[adontz]

There is one very real benefit: the APIs are more powerful. For example, NtCreateFile has the ability to open a sub-directory path based on an open directory handle. The kernel32 APIs do not expose this. It has meaningful consequences in terms of avoiding file system race conditions.

Do you assume that directory cannot be deleted and/or moved while there is an open handle?

When I first started working on Windows implementations of cross platform abstractions, I took the Microsoft documentation very seriously. But then I found out that pretty much every standard library uses SystemFunction036 from advapi32.dll for getting cryptographically secure random bytes, while Microsoft's docs says to use CryptGenRandom.

I see no mention of cryptographic security of this function. But I do not expect some std.random to be cryptographically secure. I think reasonable expectations are that std.random should have nice distribution and be fast, but not crypto secure.

On the other hand, it has allowed us to unify the std.fs API across POSIX and Windows, since with NtCreateFile they both support operating on an open directory handle.

This is a strong argument. Really, I like it.

However, converting paths between namespaces is not an easy task.
You may convert "C:\Windows\System32" to some "\??\C:\Windows\System32" and go away with that, but what about the opposite direction? "\SystemRoot\System32\" may be "C:\Windows\system32", "D:\Windows\system32" and so forth up to "Z:\Windows\system32". Windows installations are automated in enterprise environment and you can install on any large enough properly formatted partition of any disk and assign any unused drive letter during setup you cannot just assume C:

https://blogs.msdn.microsoft.com/jeremykuhne/2016/05/02/dos-to-nt-a-paths-journey/
https://stackoverflow.com/questions/4445108/how-can-i-convert-a-native-nt-pathname-into-a-win32-path-name

As far as I know there is not official way of converting NT path to Win32 path.

[daurnimator]

As far as I know there is not official way of converting NT path to Win32 path.

Why would you need to do so? Note that some NT paths are impossible to represent as a win32 path.

@adontz you might find https://googleprojectzero.blogspot.com/2016/02/the-definitive-guide-on-win32-to-nt.html?m=1 interesting.

[adontz]

@daurnimator I know of RtlDosPathNameToRelativeNtPathName_U, it was never documented.

Why? To convert paths back. As far as I remember NtQueryInformationProcess returns NT path, but GetProcessImageFileNameW does not.

Note that some NT paths are impossible to represent as a win32 path.

Exactly my point. I don't know what is your experience, but I'd rather stay away of these vague path conversion rules.

I think our conversation smoothly shifted from Win32 vs NT, to documented vs undocumented.
There are a few attributes I want to focus, if you let me to.

1. Documented / Undocumented.
2. Supported / Deprecated (may affect security, compatibility).
3. Available in all OS versions / Available in some (maybe even not latest) OS versions.

I personally prefer documented, not deprecated APIs, with safe fallback logic for APIs not available on some OS versions. For instance, I hardly imagine any reasonable fallback for TaskDialog or HTTP Server API.

Windows 7 extended support just expired (January this year). I think Windows 8.1 is a safe bet.

[daurnimator]

Why? To convert paths back.

When would you do that? For display purposes?

[adontz]

Why? To convert paths back.
When would you do that? For display purposes?

1. Yes, display purposes are important. If you cannot find a file or have any other file related problem, logging or displaying \Device\ paths will make all users and IT support freak out.

2. Not only display purposes. NtQueryInformationProcess returns path like this

"\Device\HarddiskVolume5\Users\Adontz\Projects\VS2019\NtPathTest1\Debug\NtPathTest1.exe"

Now, imagine, I want to load a vendored dynamic library, or try to load a vendored dynamic library. It may be any closed source binary I want to use, or even precompiled binary distribution of open source. To avoid all kinds of errors I would like to give LoadLibraryW an absolute path. Of course it returns NULL and I get error 126 "The specified module could not be found.", because LoadLibraryW is Win32 and has no idea where is "\Device\HarddiskVolume5\".

[AlexKotik]

I can see one big downside about using ntdll based apis instead of kernel32 based: users that will compile stdlib code with ntdll apis will get all kind of antivirus false positives. As using ntdll apis is not common for user land programs, but quite common for different kinds of malware.