Rustup: Target rustup-init.exe to run on Microsoft Nano Server

[cdhunt]

Describe the problem you are trying to solve

I'm trying to build a Windows-based build container for Rust-lang. Initially, I tried to run rustup-init.exe in a Nano Server 1903 container and the process immediately dies. The verbose flag doesn't provide any extra information. The exit code is -1073741515.

This works fine in a Windows Server Core 1903 container, but the Nano image is 400MB and the Server Core image is 5GB.

Describe the solution you'd like

Being able to run the Rust toolchain in a Nano container would help in development in testing.

Notes

docker run --rm -it mcr.microsoft.com/powershell:nanoserver-1903
> iwr https://win.rustup.rs/ -outfile rustup-init.exe
> .\rustup-init.exe -y -v --default-toolchain nightly-2019-09-11

[kinnison]

At a guess, nano might be lacking the VC++ redist stuff though I appreciate I'm not a Windows person so can't be certain.

[cdhunt]

I copied in vcruntime140.dll and msvcp140.dll into the image and still no go. I'm also just throwing stuff against the wall to see if it sticks.

[mati865]

https://win.rustup.rs/ points to 32 bit executable but Nano can only run 64 bit binaries.
You can try using this: https://win.rustup.rs/x86_64

[cdhunt]

That executable does produce different results.

warning: installing msvc toolchain without its prerequisites
error: permission denied
info: caused by: The system cannot find the file specified. (os error 2)

[mati865]

I don't know how rustup Windows code works so I cannot help further but at least it got to the part where rustup is executed.

[cdhunt]

I recall someone working on a PowerShell version of rustup-init, but can't find it now.

[kinnison]

The incomplete rustup-init.ps1 work is in #1878

[cdhunt]

Thanks, @kinnison. I even commented on that. Not sure why I couldn't find it. I'll take a look at that again.

[kinnison]

@cdhunt Did you get any further with this?

[cdhunt]

Not really. I tried a number of methods for installing VS Build Tools. I couldn't find a package that would install link.exe and couldn't find a reliable hosted copy of the binary to manually add it.

[kinnison]

Oh dear. I wonder who might be able to assist. I have basically zero Windows knowledge :(

[cdhunt]

@jonathandturner working on https://github.com/nushell/nushell is probably a good contact. I don't know much about C build tools on Windows either.

[ljani]

Now even mcr.microsoft.com/windows/servercore:1909 seems to have problems running rustup-init.exe. docker build just hangs in the RUN command:

> docker build -t mytest .
Sending build context to Docker daemon  142.8kB
Step 1/4 : FROM mcr.microsoft.com/windows/servercore:1909
 ---> ec69effa48a7
Step 2/4 : RUN curl --output rustup-init.exe --show-error --silent --location "https://win.rustup.rs/x86_64" && rustup-init.exe -y --profile default --default-toolchain nightly && echo test
 ---> Running in 14a3b34da3f4
warning: installing msvc toolchain without its prerequisites
info: profile set to 'default'
info: syncing channel updates for 'nightly-x86_64-pc-windows-msvc'
info: latest update on 2019-11-15, rust version 1.41.0-nightly (82cf3a448 2019-11-14)
warning: Force-skipping unavailable component 'clippy-x86_64-pc-windows-msvc'
info: downloading component 'cargo'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: downloading component 'rustc-dev'
info: downloading component 'rustfmt'
info: installing component 'cargo'
info: installing component 'rust-docs'
info: installing component 'rust-std'
info: installing component 'rustc'
info: installing component 'rustc-dev'
info: installing component 'rustfmt'
info: default toolchain set to 'nightly'

  nightly installed - rustc 1.41.0-nightly (82cf3a448 2019-11-14)


Rust is installed now. Great!

To get started you need Cargo's bin directory (%USERPROFILE%.cargo\bin) in your
PATH
environment variable. Future applications will automatically have the
correct environment, but you may need to restart your current shell.
test
...Hung...

[kinnison]

The default profile includes rust-docs which is never fast on Windows -- you might find it's sat there dealing with the roughly twenty-thousand files for that. Otherwise I'm not sure -- can you get procmon or some other tracing tool onto the docker to see what it's up to?

[ljani]

@kinnison: It might be the reason. rustup-init.exe does exit as I've included && echo test at the end of the RUN command and it is printed to stdout. However, if rustup-init spawns some other processes, I'm not sure how Docker handles them.

I wonder how one would run procmon during build and will it even run as there is no graphical desktop environment.

[kinnison]

There shouldn't be any process spawning from rustup-init.exe and if there were it'd be very short-lived to set up some hard links. I'm afraid I'm not sure how to debug Windows stuff, so I can't really help there :( If there's no non-graphical procmon type thing then I haven't a clue.

[rbtcollins]

This is the more modern version of procmon that I was going to move onto if we couldn't figure stuff out - https://docs.microsoft.com/en-us/windows/win32/etw/about-event-tracing - it may be needed here.

I'll note that figuring this out will require more insight that running CLI tools and guesswork: I suggest figuring out how to get a process tree, how to inspect active processes file handles, and how to trace file activity (all of which should be fairly straight forward even with a container system in play).

Once you've got that figured out, come back to the puzzle of what rust is doing.

[rbtcollins]

But that said, procmon from the host may well be able to observe whats going on - I haven't looked into the docker container implementation yet for Windows; on Linux however containers are transparent to root and kernel level tools.

[ljani]

Regarding servercore: I ran procmon.exe on the host. I see lots of writes like this one:

15.24.17,3032204    dockerd.exe 3336    WriteFile   C:\ProgramData\docker\tmp\hcs306627461\Files\Users\ContainerAdministrator\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\share\doc\rust\html\core\core_arch\arm\neon\fn.vrsqrte_f32.html SUCCESS Offset: 4, Length: 601, Priority: Normal

The writes will stop after a good 15 minutes or so (Ryzen 3700X. Adata XPG SX8200 Pro NVMe disk).

It seems rustup-init.exe is either writing docs or the writes are only flushed at this point. There are no containers running at this point. However the build container is still there in Exited state, so I think it's the latter:

a2c48f1be016        95cf78b5a953        "powershell -Command…"   8 minutes ago       Exited (0) 8 minutes ago                       youthful_newton

The container will disappear after the writes are flushed.

As a workaround --profile minimal will install much faster.

[kinnison]

It's interesting to see that it's all coming under dockerd, I wonder if that's defeating Defender's rustup detection heuristics

[rbtcollins]

Almost certainly. To confirm, disable defender temporarily and see if performance skyrockets :).

I wonder if docker is acting as a file server here - some sort of userspace NFS thing?

[ljani]

To confirm, disable defender temporarily and see if performance skyrockets :).

Indeed. It took less than a minute now that I disabled Defender.

[rbtcollins]

So I think that the underlying bug here could be assigned to many places:

• we could own it, but if we do that then we will be chasing every
  pseudo-virtualisation layer that comes along masking IO signatures
• defender could own it, but I'm not sure how defender would sensibly tell
  that docker is doing IO on behalf of rustup - admittedly I don't know the
  plumbing involved here at this point, so perhaps it is solvable
• docker could own it - it is causing the loss of the heuristic that sets
  the av scan policy to on-read rather than on-write, which appears to be the
  underlying performance issue

I think it might make sense to default to a minimal profile on nano server,
as having docs there will never make sense. But that will only fix some of
the bahviour, because there is still lots of files being scanned beyond the
docs, so it is going to be a docker/MS optimisation problem I think.

Before we try to get attention from those communities getting it filed in
windows feedback hub could be useful (with a full trace). Link the report
id here.

-Rob

On Thu, 27 Feb 2020 at 07:38, Jani [e-mail] wrote:

To confirm, disable defender temporarily and see if performance skyrockets
:).

Indeed. It too a less than a minute now that I disabled Defender.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rust-lang/rustup/issues/1994?email_source=notifications&email_token=AADZ7XQFHUQIB7AEXXEF2MTRE2ZLJA5CNFSM4IWX3PKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENBMCRQ#issuecomment-591577414,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADZ7XRKO7QU4OBRL2GQUJ3RE2ZLJANCNFSM4IWX3PKA
.

[ljani]

Please note that my comments were specifically about Windows Server Core, not Nano Server, as I incorrectly claimed that Server Core did not work anymore either.

The reported bug still stands on Nano Server:

PS Host> docker run --rm -it mcr.microsoft.com/windows/nanoserver:1909
C:\>curl --output rustup-init.exe --show-error --silent --location "https://win.rustup.rs/x86_64" && rustup-init.exe -y --profile minimal --default-toolchain nightly
warning: installing msvc toolchain without its prerequisites
error: permission denied
error: caused by: The system cannot find the file specified. (os error 2)

EDIT: Interesting enough, nanoserver does not have any of the basic DLLs rustup-init.exe tries to load:

Procmon:

22.21.27,0127536    vmwp.exe    15372   QueryDirectory  C:\ProgramData\docker\windowsfilter\87ae77e6c9265475822761f1dd83c2fe92b89fca3fa778771d2c33eb8495116a\Files\Windows\System32\KERNEL32.DLL    NO MORE FILES   Filter: KERNEL32.DLL
22.21.27,0136983    vmwp.exe    15372   QueryDirectory  C:\ProgramData\docker\windowsfilter\87ae77e6c9265475822761f1dd83c2fe92b89fca3fa778771d2c33eb8495116a\Files\Windows\System32\ADVAPI32.dll    NO MORE FILES   Filter: ADVAPI32.dll
22.21.27,0138940    vmwp.exe    15372   QueryDirectory  C:\ProgramData\docker\windowsfilter\87ae77e6c9265475822761f1dd83c2fe92b89fca3fa778771d2c33eb8495116a\Files\Windows\System32\ole32.dll   NO MORE FILES   Filter: ole32.dll
22.21.27,0144020    vmwp.exe    15372   QueryDirectory  C:\ProgramData\docker\windowsfilter\87ae77e6c9265475822761f1dd83c2fe92b89fca3fa778771d2c33eb8495116a\Files\Windows\System32\KERNEL32.dll    NO MORE FILES   Filter: KERNEL32.dll
22.21.27,0145490    vmwp.exe    15372   QueryDirectory  C:\ProgramData\docker\windowsfilter\87ae77e6c9265475822761f1dd83c2fe92b89fca3fa778771d2c33eb8495116a\Files\Windows\System32\SHELL32.dll NO MORE FILES   Filter: SHELL32.dll
22.21.27,0146342    vmwp.exe    15372   QueryDirectory  C:\ProgramData\docker\windowsfilter\87ae77e6c9265475822761f1dd83c2fe92b89fca3fa778771d2c33eb8495116a\Files\Windows\System32\USER32.dll  NO MORE FILES   Filter: USER32.dll

Shell:

C:\>dir C:\Windows\system32\kernel*
 Volume in drive C has no label.
 Volume Serial Number is D65B-11BF

 Directory of C:\Windows\system32

02/16/2020  09:19 PM           282,544 kernel32legacy.dll
02/16/2020  09:19 PM         2,766,088 KernelBase.dll
03/19/2019  05:42 AM           236,544 KernelTraceControl.dll
               3 File(s)      3,285,176 bytes
               0 Dir(s)  21,301,809,152 bytes free

EDIT2: Ah:

Reverse Forwarders are an optional package on Nano Server and not installed by default – see the Nano Server deployment guide for steps to include this package. Included in the Reverse Forwarder package are shims for

• advapi32.dll
• comctl32.dll
• comdlg32.dll
• gdi32.dll
• kernel32.dll
• ole32.dll
• psapi.dll
• shell32.dll
• shlwapi.dll
• user32.dll
• version.dll
• winmm.dll

https://cloudblogs.microsoft.com/windowsserver/2015/11/16/moving-to-nano-server-the-new-deployment-option-in-windows-server-2016/

[rbtcollins]

Ok. So I think the core point here is we need a really clear understanding about what is happening before we seek attention from other communities :). Ideally we have a summary post that wraps it all up with a bow, and a clear conclusion, listing what we can sensibly do in the rustup context, and what things belong outside it.

[workingjubilee]

@rustbot label: +O-windows +O-containers