Node-restify: Issues uploading pdf file via streams

[Sigurthorb]

I am working on a server that manages pdf files. This server needs to be able to upload, download and store pdfs.

I tried getting help on mIRC and stack overflow without any response, This is my last resort

Client download has been implemented without any issues.

The client upload is the problem:
Headers are being piped into my file

What I have Tried

1. Removing bodyParser for upload endpoint

When I included the bodyparser the stream was not writing any data into the file. My endpoint is defined before the bodyParser and queryParser

2. Using streams

Because of the possible size of files I do not want to transfer the file in its entirety and want to use streams. When I use streams, the header of the request is also pushed to the stream. This is not acceptable and requires me to ignore the first chunk that arrives in the response (see code examples below).

The header being pushed to the file stream:

----------------------------487602989105343035284942
Content-Disposition: form-data; name="file"; 
filename="myfileuploadtest.pdf"
Content-Type: application/pdf

My current code

This is my current client code for upload

var req = request.post({url: `http://127.0.0.1:${PORT}/pdf/file`, headers:
{filename: "myfileuploadtest.pdf"}}, (err, resp, body) => {
    if(err){
        console.log("ERROR", err.message);
    } else {
        console.log("body: ", body);
    }
});

var form = req.form();
form.append( "file", fs.createReadStream("myfileuploadtest.pdf"));

This is my current code for upload to the server

let doneIgnoreFirst = false;
req.on("data", (chunk) => {
    if(!doneIgnoreFirst){
        doneIgnoreFirst = true;
    } else {
        writeStream.write(chunk);
    }
});

//dont know if this event is actually fired
req.on("error", (err) => {
    console.log("Error while receiving PDF");
    res.send(500);
});

req.on("end", () => {
    writeStream.end(() => { 
        res.send(500);
    });
});

If not for my issue with the headers being included in the stream, I would have used something like this:

req.pipe(writeStream);
writeStream.on("finish", () => { res.send(200); });
writeStream.on("error", () => { res.send(500); });

I am doing something wrong

There is something that I am missing. I suspect that the restify bodyParser is the fault here. The headers are not parsed from the stream which introduces the issue that I am having and requires me to skip the first chunk.

This is not a solution for this issue. Depending on the request sent, parts of the pdf can be included in the first chunk. this breaks the pdf because not all of its data is included in the file on the server. This makes the endpoint very unstable which I do not want.

Any help is greatly appreciated!

Versions

• restify: 4.3.0
• request: 2.81.0

[retrohacker]

Hey @Sigurthorb,

Thank you for opening an issue with such detailed notes, and for including a repro case! Flagging this as a potential bug and/or question, and will circle back later today to take a look :heart:

[retrohacker]

Oh my :disappointed:,

By "later today" I apparently meant in 7 months... So terribly sorry @Sigurthorb, were you able to resolve this?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed as stale because it has not had recent activity.