Rocket: Issue with cmd on Windows 7

[guangie88]

Hi, thanks very much for your work! Rocket is really easy to get started for doing web service.

Recently I have encountered an issue while exposing some HTTP endpoint to perform a std::process::Command call, and this only happens on Windows 7 x64 / Windows Server 2008 x64. The exit code, stdout and stderr from the Command output are fine until a certain number of invokes from the HTTP client, and subsequently, the exit code becomes consistently erroneous.

The above issue does not occur on Windows 8.1 x64, Windows 10 x64 and also not on Linux distribution builds. As the executable was built statically on Windows, the executable built on Windows 7 could be run on other Windows versions without issue, but strangely not on Windows 7. I have also tried building on Windows 10 and run on Windows 7, but the issue still occurs.

I have created a simple example using Rocket + Command (see link below) to demonstrate the issue. The program will expose the endpoint and let the self contained HTTP client execute the endpoint for 50 times to perform echo command. In my case, the issue occurs at around the 32nd iteration and onwards on Windows 7, where the exit code is no longer 0 and stdout is no longer correct.

For comparison, I have also created another example using Rouille (another link below) for comparison. The build works on all Windows versions.

I am not sure if it is really an issue with Rocket, or it could be an issue with Windows 7 instead. My original post at Rust forum: https://users.rust-lang.org/t/strange-crash-in-std-process-command-on-windows-7/10834 has more details and about my attempts at trying to find the cause of the issue.

Additional Details

• Rocket Version: 0.2.6
• Operating System: Windows 7 SP1 x64 / Windows Server 2008 R2 x64 (other OSs seem to work fine, including Ubuntu)
• Build Used: rustc + Cargo nightly, MSVC Build Tools 2015, static link flags added (i.e. rustflags = ["-Z", "unstable-options", "-C", "target-feature=+crt-static"])

Rocket + Command (issue)

https://github.com/guangie88/rocket_cmd_bug

Should observe the issue by building and running without any arguments. The stdout log should display erroneous exit code at around 32nd iteration and onwards, only on Windows 7 / Windows Server 2008 R2. Should work fine for all newer Windows versions and also on Linux builds.

Rouille + Command variant (for comparison)

https://github.com/guangie88/rouille_cmd_ok

Also can build and run without any arguments. The exit code and stdout in each entry should be 0 and 'hello' respectively.

[SergioBenitez]

This is interesting. I would be quite surprised if this has something to do with Rocket.

Have you tried simply spawning cmd in a loop, without any web framework? Just a standalone application. Something like:

fn main() {
    for i in 0..100000 {
        let out = Command::new("cmd").args(&["/C", "echo hello"]).output().unwrap();
        println!("Iteration {}: {}", i, out);
    }
}

If this eventually results in the same issue, then we can rule out Rocket.

Rocket and rouille use different underlying HTTP libraries. Namely, Rocket uses Hyper and rouille uses tiny_http. It's possible, though I still think highly unlikely, that something in Hyper is causing this issue. Have you tried the same experiment with bare-bones Hyper?

[guangie88]

Thanks for the reply! I have tried the cmd spawning example, and it works fine even up to 100000 iterations.

I have also tried out the experiments on both Hyper and tiny_http HTTP servers, as you have pointed out about the different underlying HTTP libraries that were used. Since Rocket is based on Hyper, I was initially expecting the issue to surface in the Hyper HTTP server set-up. However, the Hyper set-up printed the correct output without issue. The tiny_http set-up also printed the correct output.

Out of curiosity, I have also tried out Rocket with manual routing (based on the latest git commit a4292ba666fa60448e575b600d69f131795da080), but unfortunately it still exhibits the same issue as seen in the macro-based routing set-up (which was tested on Rocket 0.2.6 and 0.2.7).

As mentioned, all the set-ups are done in Windows 7 only, since the issue only appears in Windows 7 execution. Are there any other notable differences between the HTTP server done on Rocket and on bare-bones Hyper, or any other suggestions that you might have in mind?

The section below contains the set-up output for demonstration, as well as the links to the various set-ups.

Output of other HTTP frameworks

Normal output.

...
#30: Output { status: ExitStatus(ExitStatus(0)), stdout: "hello\r\n", stderr: "" }
#31: Output { status: ExitStatus(ExitStatus(0)), stdout: "hello\r\n", stderr: "" }
#32: Output { status: ExitStatus(ExitStatus(0)), stdout: "hello\r\n", stderr: "" }
#33: Output { status: ExitStatus(ExitStatus(0)), stdout: "hello\r\n", stderr: "" }
#34: Output { status: ExitStatus(ExitStatus(0)), stdout: "hello\r\n", stderr: "" }
#35: Output { status: ExitStatus(ExitStatus(0)), stdout: "hello\r\n", stderr: "" }
...

Output of Rocket (both for macro-based routing and manual routing)

Line 33 starts to output incorrectly, happens consistently on Windows 7 x64. The output is taken directly after the HTTP server executes the cmd.

...
#30: Output { status: ExitStatus(ExitStatus(0)), stdout: "hello\r\n", stderr: "" }
#31: Output { status: ExitStatus(ExitStatus(0)), stdout: "hello\r\n", stderr: "" }
#32: Output { status: ExitStatus(ExitStatus(0)), stdout: "hello\r\n", stderr: "" }
#33: Output { status: ExitStatus(ExitStatus(3221225794)), stdout: "", stderr: "" }
#34: Output { status: ExitStatus(ExitStatus(3221225794)), stdout: "", stderr: "" }
#35: Output { status: ExitStatus(ExitStatus(3221225794)), stdout: "", stderr: "" }
...

Application error pop-up when executing from lines 33 and onwards.

Set-Up links

Hyper

https://github.com/guangie88/hyper_cmd

tiny_http

https://github.com/guangie88/tiny_http_cmd

Rocket with manual routing

https://github.com/guangie88/rocket_manual_route_cmd

[SergioBenitez]

That's really surprising. Okay! :)

Can you try using Rocket from master? You should be able to just point your Cargo.toml to the github repository for both rocket and rocket_codegen.

[guangie88]

rocket_manual_route_cmd was already using the Rocket repository git master branch, and I have changed the original rocket_cmd_bug which uses the macro-based routing to also use the git master branch.

The issue is still present unfortunately, with Line 33 and onwards printing the incorrect output.

Thanks for your help!

Links:
https://github.com/guangie88/rocket_cmd_bug
https://github.com/guangie88/rocket_manual_route_cmd

[SergioBenitez]

I've spent a bit of time, and intend to spend more, investigating and eventually resolving this issue. Thus far, I've noted the following:

• I've reproduced the issue on Windows 7 SP1 with Visual C++ Build Tools 2015.
• I've confirmed that the issue does not occur with the hot-loop I posted above.
• The issue _may_ be caused by desktop heap exhaustion.

The lattermost point is what I found after doing a bit of research online. It coincides with your investigation in the initial forum post. If this is indeed the case, it should be possible to implement the workaround in Microsoft's hotfix.

If this workaround resolves the issue, then we can begin to think about why (or if?) Rocket is not freeing the resources used by the handler. That being said, a bit of heap profiling and resource monitoring should reveal whether Rocket is freeing resources appropriately, suggesting whether the workaround will have any impact or not.

[SergioBenitez]

Hoorah! After some more triage, I was able to determine that the cause of the issue was Rocket's use of the term-painter library. Under the hood, term-painter uses the term library to colorize terminal output. On Windows, term uses the win32 console API to colorize output. This means that a win32 console API call was being made every time a new color or style was printed to the console. At some point, something (I'm not sure just _what_ yet) goes wrong as a result of this, and execing any program thereafter fails.

Commit https://github.com/SergioBenitez/Rocket/commit/a6c4d053ade99bc396961e52ce7f3e53536e206f switches Rocket to yansi for output coloring. The crate only implements ANSI coloring, so no win32 API calls will be made on Windows. This may sound like a drawback; in reality, Rocket's use of term-painter was not appropriate, and so switching will bear no real loss and result in increased correctness.

@guangie88 I look forward to your confirmation that this resolves your issue. :)

[guangie88]

@SergioBenitez Yes the latest commit a6c4d05 has resolved the issue, thank you for your help! Works also for x86_64-pc-windows-gnu, which previously had the same issue.

Now there is no reason not to use Rocket for any Rust based HTTP server! :smile: On a side note, I see that yansi is also part of your work, keep up with all the good works! :+1:

If you happen to eventually find out the underlying issue about term-painter (which strangely only seems to affect Windows 7 / Windows Server 2008), I would be curious to know if you don't mind sharing.

Thanks again! :smiley: