Koa: Add a multipart example

[Raynos]

@jonathanong is removing the multipart body parser from connect and the koajs/body-parser modules doesn't support it either.

It would be cool to have a new example of doing multipart, especially since there was some recent negative feedback on how connect did it with the temp files.

[tj]

haha yeah the "convenience" aspect of layering it in as middleware didn't work out too well, it would be nice to have some nice generator-friendly api, contrived example:

var parts = parse(this);
var part;

while (part = yield parts.pop()) {
  // do stuff with part
}

[Raynos]

Edit: there are a bunch of edge cases this doesn't handle. Like the end of the list of parts case and others.

I think the following might work with multiparty

var Form = require("multiparty").Form

function parse(req) {
  var form = new Form()
  var started = false

  return { pop: function (cb) {
    if (!started) {
      started = true  
      form.parse(req)
    }

    form.once("part", cb)
  } }
}

In this case part would be a ReadableStream. Also minor parse(this) vs parse(this.req) api difference.

[tj]

yeah, that's the contrived part :D haha, im sure we could figure something reasonable out, I definitely think ideally stream-like things are more or less just iterators and backpressure is implied by the lack of reading

[jonathanong]

i'll take a stab at it. i don't really like pop() though since i feel like you're getting the first part not the last part. we also got to handle errors and fields otherwise it's just going to hang

@Raynos that won't work because the part needs to be the second argument of the callback. also, when the stream is done, pop() will just hang since no more part events will be emitted.

[tj]

sorry yeah not .pop() haha, .shift() or .part(), derp

[Raynos]

I have an example of how I handled multipart ( https://github.com/Raynos/http-framework/blob/master/examples/multipart/server.js#L25 ) using callbacks.

Here's an idiomatic generators api:

function handleRequest(req, res) {
  var form = yield MultiPartyForm(req, res, {
    // handlePart is called in parallel.
    handlePart: function* (part) {
      // part is a readable stream
      // imagine yield friendly pipe.
      yield pipe(part, fs.createWriteStream("wherever"))
      return { location: "wherever", filename: part.filename }
    }
  })

  // from <input name='title' />
  var titleField = form.fields.title

  // from <input type='image' name='image' />
  var fileLocation = form.files.image.location
  // perge file or whatever.
  yield fs.unlink.bind(null, fileLocation)
  res.end("uploaded " + form.files.image.filename + " to " + fileLocation)
}

The cojs/busboy api is ok but if you yield in the while loop then its slow for multi file upload if that's a common use case.

[jonathanong]

why would it be slower? don't files upload sequentially?

[Raynos]

@jonathanong they do upload sequentially but if your do something complex asynchronously like writing it to disk and then running some conversion algorithms on it you would block all the other parts from being handled whilst running your conversions algorithms.

For the while loop not to be a blocker you would have to do EVERYTHING with the part in a streaming fashion using multi pipe. i.e. you musn't do any yield statements after part.emit("end")

Also any time difference between part.emit("end") and dest.emit("finish") is also wasted time that's not parallelized.

Remember your talking to the guy who says middleware is "too slow" :D

[tj]

if you yielded something that writes it to a location then yeah it would be slower, but you could parallelize that portion and keep the part reading sync. if you stuff the part write streams (yieldable ones) into an array and then yield arr after the loop you'd get the block-until-all-writes-are-complete behaviour

[tj]

contrived lame api:

var form = parse(this);
var writes = [];
var part;

while (part = yield form.part()) {
  writes.push(write(part, 'somewhere'));
}

yield writes;

[tj]

the typical fork/join coro pattern would be like:

var form = parse(this);
var writes = [];
var part;

while (part = form.part()) {
  writes.push(fork write(part, 'somewhere'));
}

join writes;

[Raynos]

@visionmedia that doesn't work.

You won't get the next part until you have consumed the first part. This is because the request body is sequential you have to consume the entire first file before the second file is in the request stream buffer.

I tried that api and it broke on larger examples where parts don't fit in a single data chunk

[Raynos]

even handlePart in my example wont run in true parallel. You still need to get part.emit("end") to happen before the next part becomes available.

[jonathanong]

it can work if write looks something like this:

function write(stream, dest, done) {
  var writer = fs.createWriteStream(dest)
  stream.pipe(writer)

  var ended = false
  writer.once('finish', function () {
    ended = true
    if (done) done()
  })

  return function (fn) {
    if (ended) fn()
    else done = fn
}

[Raynos]

It would work but it is surprising. If I'm going generator heavy I would want to use yield X to mean I am doing IO

Having a write() function that does IO but is not yielded upon might be confusing.

[tj]

yeah it's kinda misleading to have both immediate and non-immediate yieldable things, that's part of why I feel like generators don't really make enough of a difference to really "solve" a problem concurrency-wise, especially compared to more robust stuff like Go but I want to try, way too much legacy node stuff to ditch for now

[Raynos]

@visionmedia you can emulate go channels ( https://github.com/rads/csp.js https://github.com/olahol/node-csp ) there are various attempts at this.

[tj]

kinda, best we'll get in js for a long time at least, still tons of hackery and edge-cases with generators, and marginally better error handling

[Raynos]

That's not true. The error handling is leaps and bounds better. You can use generators to propagate errors upwards like they would with exceptions.

What does "truely better error handling" look like?

[tj]

coros/goroutines/erlang procs, all provide much better isolation with separate stacks, we still don't come close to that with generators, it is way better than traditional node code though

[Raynos]

@visionmedia by real isolation you mean something like domain.dispose(); // just keep the server running would work? i.e. you can run each req/res in a coroutine and just kill that coroutine and keep the server running without any side effects ?

[tj]

all those other solutions (and erlang more so) give you different stacks, so if anything anywhere explodes it can just trickle back up, even with generators we can do our best to have that sort of behaviour but it'll still never be perfect. Underlying node libs are still going to be broken or be missing "error" handlers etc. Makes me sad that the node-fiber guys had issues with v8's stack usage it would have been really awesome to fork node and try out a more go-like variant

[Globik]

Блять, а как вы будете проверять у многих файлов mimetype? Бля, цикл виснет и все нахуй.

[Globik]

Когда по одному файл загружаешь заебись, а когда сразу то виснет сука и в обратку ошибку не шлет.

[juliangruber]

@Globik english please