Rocket: Memory leak when benchmarking hello_world example

[benwilber]

1. The version of Rocket you're using. Ensure it's the latest, if possible.

Latest master as of this bug report

$ git branch
* master
$ git rev-parse HEAD
3f87b16d759e91386371ef0eb61df9a372ea8fec

1. The operating system (distribution and version) where the issue occurs.

CentOS 7.3 on Digital Ocean 512MB Droplet.

$ uname -a
Linux rocket-test 3.10.0-514.21.1.el7.x86_64 #1 SMP Thu May 25 17:04:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

1. A brief description of the bug:

I'm testing benchmarks of the hello_world example against Rust nightly (6/20/2017) and noticing that resident memory grows and is never released until OOM occurs. I built a release binary of the hello_world example with:

$ rustup run nightly cargo build --release

I then run it with:

$ ROCKET_ENV=prod ROCKET_PORT=8000 target/release/hello_world
Warning: environment is 'production', but no `secret_key` is configured
🚀  Rocket has launched from http://0.0.0.0:8000

I then open another terminal session and watch memory usage with:

$ watch 'pidstat -r -C hello_world'

Every 2.0s: pidstat -r -C hello_world                                                                                         Tue Jun 20 18:57:10 2017

Linux 3.10.0-514.21.1.el7.x86_64 (rocket.sunspot.io)    06/20/2017  _x86_64_        (1 CPU)

06:57:10 PM   UID   PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
06:57:10 PM     0     10042  0.15      0.00   27452   1292   0.26  hello_world

Then from a different server I run Apache Bench with:

# 100 concurrent request, 1 million total
$ ab -c 100 -n 1000000 http://rocket-test.example.com:8000/

I observe that the RSS continues to grow rapidly and is never released until OOM (at ~512MB)

# After roughly 500k requests:
07:06:49 PM   UID   PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
07:06:49 PM     0     10578 13.83      0.00  234300 207992  41.56  hello_world

$ ROCKET_ENV=prod ROCKET_PORT=8000 target/release/hello_world
Warning: environment is 'production', but no `secret_key` is configured
🚀  Rocket has launched from http://0.0.0.0:8000
Killed

I tried stopping the test early (before OOM) and waiting for netstat to indicate all the TCP connections had cleared to see if the memory was released, but it wasn't. The memory stays resident until the app is restarted completely.

[benwilber]

I also tested the hello.rs example in Hyper directly and it doesn't exhibit the same problem:

$ rustup run nightly cargo run --release --example hello
   Compiling hyper v0.10.11 (file:///root/hyper)
    Finished release [optimized] target(s) in 9.39 secs
     Running `target/release/examples/hello`
Listening on http://0.0.0.0:8000

Then after a million requests:

Every 2.0s: pidstat -r -C hello                                                                                               Tue Jun 20 19:58:43 2017

Linux 3.10.0-514.21.1.el7.x86_64 (rocket.sunspot.io)    06/20/2017  _x86_64_        (1 CPU)

07:58:43 PM   UID   PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
07:58:43 PM     0      6170  0.72      0.00   22176   1136   0.23  hello

The RSS stays low/constant.

[Eijebong]

I just tried it myself and had no RSS growth at all with the rocket hello_world example

[SergioBenitez]

@benwilber You'd need to test against Hyper 0.10. Hyper 0.11 is effectively a complete rewrite.

[benwilber]

@SergioBenitez I tested against Hyper v0.10.11, which is the version used by Rocket master

[SergioBenitez]

Apologies. I misread the version number.

I've confirmed the issue on my side. This is a regression from 0.2. Investigating the cause of the issue now.

[SergioBenitez]

I've determined the cause of the issue. Working on a fix now.

[benwilber]

@SergioBenitez I'll be happy to test the fix when it's ready

[SergioBenitez]

@benwilber Please confirm the fix in https://github.com/SergioBenitez/Rocket/commit/3c4cb27d55faa8885421bfedcec0ef3501702793.

[benwilber]

@SergioBenitez Yeah that fixed it.

[SergioBenitez]

Excellent. Thank you for reporting this!