Got: responseType of buffer is always encoded as utf8

Created on 3 Dec 2019 · 4Comments · Source: sindresorhus/got

Describe the bug

Node.js version: 13.2.0
OS & version: FreeBSD 12.0

With version 10 of got (10.01 at the time of writing this), when I set the responseType to buffer in my options, it unconditionally encodes the body for the buffer as utf8 due to the code using Buffer.from(body) and not accepting an encoding (possibly also because of getStream not being passed an encoding either). This is a problem if one wants to get a raw set of bytes from a URL, such as an image. An example would be a PNG, the first byte of the buffer should be 0x89, but instead I get 3 bytes of 0xEF 0xBF 0xBD (the Unicode replacement character in UTF-8). The only workaround I found was to use the default responseType with an encoding of binary and manually convert to a Buffer with Buffer.from(response.body, 'binary').

Actual behavior

A UTF-8 encoded buffer, which for images such as PNGs results in a malformed file as described above.

Expected behavior

To be able to get raw bytes from a URL in order to avoid making the file malformed.

Code to reproduce

async function test()
{
    const testUrl = 'http://kirby.cyberbotx.com/WrongEgg.png';

    // Incorrect response
    let options = {
        method: 'GET',
        responseType: 'buffer'
    };
    let response = await got(testUrl, options);
    let header = response.body.slice(0, 16);
    console.log('Incorrect header: ', header);
    // Buffer <EF, BF, BD, 50, 4E, 47, 0D, 0A, 1A, 0A, 00, 00, 00, 0D, 49, 48>

    // Correct response
    options = {
        encoding: 'binary',
        method: 'GET',
        responseType: 'default'
    };
    response = await got(testUrl, options);
    header = Buffer.from(response.body, 'binary').slice(0, 16);
    console.log('Correct header: ', header);
    // Buffer <89, 50, 4E, 47, 0D, 0A, 1A, 0A, 00, 00, 00, 0D, 49, 48, 44, 52>
}
test();