Rack: Large file upload consumes 100% cpu for a very long time

Created on 24 May 2016 · 15Comments · Source: rack/rack

Rack 2.0.0.rc1 with Rails 5.0.0.rc1 and Puma 3.4.0

The problem is that there is a loop in Rack::Multipart::Parser.parse (https://github.com/rack/rack/blob/master/lib/rack/multipart/parser.rb#L70). On each iteration it reads at most Rack::Multipart::Parser::BUFSIZE bytes which is 16384. But for the 200Mb file this means 12000 iterations and this is very slow.

My dirty hack:

# config/initializers/rack.rb
Rack::Multipart::Parser.const_set('BUFSIZE', 10_000_000)

Maybe there should be a config option to override default Rack::Multipart::Parser::BUFSIZE. Or maybe a better solution?

Source

olegantonyan

👍11

Most helpful comment

You can create a middleware and just insert it before Rack, such as:

class MultipartBufferSetter

  def initialize(app)
    @app = app
  end

  def call(env)
    env.merge!(Rack::RACK_MULTIPART_BUFFER_SIZE => 100*1024*1024)
    @app.call(env)
  end

end

and then:

config.middleware.insert_before "Rack::Runtime", "MultipartBufferSetter"

darfux on 28 Oct 2016

👍11

All 15 comments

Since no one has commented, you can pass header 'rack.multipart.buffer_size' with your response in order to specify the size of buffer instead of doing such workarounds.

miroslavLalev on 2 Jul 2016

👍3

Thank you @miroslavLalev !

olegantonyan on 2 Jul 2016

@miroslavLalev, @olegantonyan Can you please explain how to do it? Where to put it?

vizo on 16 Sep 2016

@vizo I didn't succeed to send the header, I just did the dirty hack of @olegantonyan and it's finally working fine.

disastrous-charly on 22 Sep 2016

Actually, i didn't work for me either, and I couldn't find why in rack's source code. I've just implemented clunked upload, so this option no longer required.

olegantonyan on 22 Sep 2016

Any news with this issue?

mahnunchik on 14 Oct 2016

@mahnunchik The "dirty" (but very useful) hack of @olegantonyan is working great. I just have a warning when starting the server.

Chunked upload is a good alternative (easy to do with https://github.com/tors/jquery-fileupload-rails ) but if you provide a public API, users will also have to implement the chunked upload. (lost my hairs trying to do it with cURL)

So that's why personnaly I go with the dirty hack.

disastrous-charly on 14 Oct 2016

You can create a middleware and just insert it before Rack, such as:

class MultipartBufferSetter

  def initialize(app)
    @app = app
  end

  def call(env)
    env.merge!(Rack::RACK_MULTIPART_BUFFER_SIZE => 100*1024*1024)
    @app.call(env)
  end

end

and then:

config.middleware.insert_before "Rack::Runtime", "MultipartBufferSetter"

darfux on 28 Oct 2016

👍11

@darfux Thanks, it works! ;)

vizo on 29 Oct 2016

You need to set rack.multipart.buffer_size to your Rack env before Rack::Request#params is called for the first time (or just #params in Rails), because that's when parsing the multipart body happens. I tested that the following script correctly sets the buffer size:

gem "rack-test_app"
require "rack/test_app"

app = proc do |env|
  request = Rack::Request.new(env)
  request.env["rack.multipart.buffer_size"] = 100
  request.params

  [200, {}, []]
end

test_app = Rack::TestApp.wrap(app)
test_app.post("/", multipart: {file: File.open(__FILE__)})

janko on 12 Apr 2017

👍2

Yeah, finded it!!! Ty!

winkser on 6 May 2017

If anyone's interested in conclusively addressing this, a CPU+memory usage benchmark for different buffer sizes (16kB, 1MB, 4MB, 16MB, etc) and file sizes (1kB, 1MB, 10MB, 100MB) would clearly indicate a sweet-spot buffer size.

Silly to have everyone needing to figure out a workaround themselves or insert this awkward runtime middleware. Please do investigate 😊