Http-proxy-middleware: [ Question ] How do you manipulate proxy response

[iroy2000]

Hi, I'm using Express 4,

In normal way, I can do the following

    app.get('/v1/users/:username', function(request, response, next) {
        var username = request.params.username;
        findUserByUsername(username, function(error, user) {
            if (error) return next(error);
            return response.render('user', user);
        });
    });

But how do I execute custom logic if I'm using proxy, let's say I want to manipulate some of the data before response to the user? Is there a good pattern to do that with this middleware ?

app.use('/api', proxy({target: 'http://www.example.org', changeOrigin: true}));

[chimurai]

I don't have the answer unfortunately.

You are not the first one the ask this question. (https://github.com/chimurai/http-proxy-middleware/issues/91#issue-165297848)

Happy to add this one to the http-proxy-middleware recipes.

Maybe this example might help you:
https://github.com/nodejitsu/node-http-proxy/blob/master/examples/middleware/modifyResponse-middleware.js

Can you post this question on StackOverflow (with backlink)? Hopefully your question will get an answer there.

Thanks!

[iroy2000]

I tried to use https://github.com/langjt/node-http-proxy-json

And it seems to solve my problem. Note that the library is still fairly new, but it's definitely take out some pain points from many people that has the same issues.

Let me just quote my code here for people who is interested, Hope it helps :)

@chimurai You can even add that to your recipe if you think it is good enough.

import express from 'express';
import proxy from 'http-proxy-middleware';
import config from 'config';
import modifyResponse from 'node-http-proxy-json';

const router = new express.Router();

const myProxy = proxy(config.get('some.json.api.Url'), {
  changeOrigin: true,
  logLevel: 'debug',
  pathRewrite: (path, req) => path.replace('/api', '/_ajax'),
  onProxyRes(proxyRes, req, res) {

    delete proxyRes.headers['content-length'];

    modifyResponse(res, proxyRes.headers['content-encoding'], function (body) {
      if (body) {
        // modify some information
        body.version = 2;
        body.props = {
          "nestedProps": true
        };
      }
      return body;
    });
  },
});


router.use('/api/example', myProxy);

export default router;

[silverprize]

@iroy2000 Thanks for your approach. But I'd like to add custom header from body's property.
I tried and faced "Can't set headers after they are sent.".
I suggest to use multi proxy module.
express-http-proxy provide async proxy response.
express-http-proxy is inconvenient than http-proxy-middleware thus keep http-proxy-middleware as main proxy.
Few apis that need to modify header or body are called via express-http-proxy.

[cokeknight]

here is my answer,

onProxyRes :function(proxyRes, req, res){
      var _write = res.write;
      var output;
      var body = "";
      proxyRes.on('data', function(data) {
          data = data.toString('utf-8');
          body += data;
      });
      res.write = function (data) {
        try{
          eval("output="+body)
          output = mock.mock(output)
          _write.call(res,JSON.stringify(output));
        } catch (err) {}
      }
    }

add onProxyRes option on the http-proxy-middleware
use the data event on the proxyRes to get the output
then modify the output in res.write

[enl31]

@cokeknight thank you, can you help me more? How do you change header in this approach?
I want to have custom header containgn length of (json) body in response like this:

  res.write = data => {
    data = data.toString('utf-8');
    body += data;
    try{
      body = JSON.parse( body);
       res.set({
        'Content-Type': 'Application/json',
        'count': body.meta.count
      });
      _write.call(res,JSON.stringify(body.data));
    } catch (err) {}
  }

[zaaack]

Not working for async case, I need return some async data, but it just return nothing.

[benjamin658]

@enl31 You can set content-length by this way:

proxyRes.headers['content-length'] = data.length;

[westmark]

This was my final solution (inspired by @kokarn 's solution over at https://github.com/nodejitsu/node-http-proxy/issues/796

I'm using this in a webpack dev server setup by create-react-app, hence the comment further down.

```
onProxyRes: ( proxyRes, req, res ) => {
const _writeHead = res.writeHead;
let _writeHeadArgs;
const _end = res.end;
let body = '';

proxyRes.on( 'data', ( data ) => {
  data = data.toString( 'utf-8' );
  body += data;
} );

// Defer writeHead
res.writeHead = ( ...writeHeadArgs ) => {
  _writeHeadArgs = writeHeadArgs;
};

// Defer all writes
res.write = () => {};

res.end = ( ...endArgs ) => {
  // Do everything in the end. Means we wont be streaming the response but who cares in a dev env..
  const output = body
    .replace( /stufftoreplace/g, '/leweb' );

  if ( proxyRes.headers && proxyRes.headers[ 'content-length' ] ) {
    res.setHeader( 'content-length', output.length );
  }

  // This disables chunked encoding
  res.setHeader( 'transfer-encoding', '' );

  // Disable cache for all http as well
  res.setHeader( 'cache-control', 'no-cache' );

  _writeHead.apply( res, _writeHeadArgs );

  if ( body.length ) {
     // Write everything via end()
    _end.apply( res, [ output ] );
  } else {
    _end.apply( res, endArgs );
  }
};

}

[youngzhaosignifyd]

If proxyRes is gzipped, I will need to unzip it before pass it to res. Is there a better way to work around this?

Here is my code

onProxyRes: (proxyRes, req, res) => {
    const end = res.end;
    const writeHead = res.writeHead;
    let writeHeadArgs;
    let body; 
    let buffer = new Buffer('');

    // Concat and unzip proxy response
    proxyRes
      .on('data', (chunk) => {
        buffer = Buffer.concat([buffer, chunk]);
      })
      .on('end', () => {
        body = zlib.gunzipSync(buffer).toString('utf8');
      });

    // Defer write and writeHead
    res.write = () => {};
    res.writeHead = (...args) => { writeHeadArgs = args; };

    // Update user response at the end
    res.end = () => {
      const output = manipulateBody(body); // some function to manipulate body
      res.setHeader('content-length', output.length);
      res.setHeader('content-encoding', '');
      writeHead.apply(res, writeHeadArgs);

      end.apply(res, [output]);
    };
  }

[nickgrealy]

I'm currently using @cokeknight 's solution (https://github.com/chimurai/http-proxy-middleware/issues/97#issuecomment-268180448), and I'm finding my JSON responses are being truncated after 8192 characters.

Is anyone else finding this?

N.B. It could be (probably is) because I'm not using the following code snippet... can someone explain what the intention is, and why you wouldn't just do _write.call(res, body);?

eval("output="+body)
output = mock.mock(output)
_write.call(res, JSON.stringify(output));

(e.g. assign body reference to variable output (why use eval?), then wrap it in a mock?)

[ortonomy]

Allright, I also used : https://github.com/langjt/node-http-proxy-json

and it solved my problem. I recommend it over the manual re-writing of res.write methods - it the same thing as the solutions above, but it's captured in a library. 😅

[sesm]

Perhaps the core problem is that response proxy gets from original source is not passed through existing express middleware chain?
Express already has an ecosystem of middleware, but for some reason we have to duplicate it's functionality in onProxyRes. This doesn't seem right.

[TitasGailius]

I've been trying to set a cookie which is extracted from the response body but headers are already sent after the response is read.
Any ideas on how could I defer sending headers until I read the response?

This is one of the ways I tried with
https://github.com/langjt/node-http-proxy-json

{
    onProxyRes: (proxyRes, req, res) => {
        modifyResponse(res, proxyRes, body => {
            res.cookies.set('foo', 'bar')
        })
    }
}

[sergeymakoveev]

@youngzhaosignifyd @westmark
Fix it

res.setHeader('content-length', output.length);

to

res.setHeader('content-length', Buffer.byteLength(output));

[garajo]

I needed to res.redirect(). Thanks to the solutions above, here is the variance. Binding with res prevents undefined this errors.

onProxyRes: function (proxyRes, req, res) {
  const _writeHead = res.writeHead.bind(res);

  ...

  proxyRes.on('end', () => {
    const data = JSON.parse(body)
    res.writeHead(301, { Location: req.headers.referer })
    res.end()

  })
})

[sergeymakoveev]

@garajo
try _writeHead(301, 'ok', { Location: req.headers.referer })
instead res.writeHead(301, { Location: req.headers.referer })

[sergeymakoveev]

Guys, you can use my universal helper for a proxy and modify the response:
https://gist.github.com/sergeymakoveev/cc453586a59896c7605c22382d181bb7#file-webpack-response-modify__-babel-js

[garajo]

@sergeymakoveev I think you meant this, res.writeHead(301, 'ok', { Location: req.headers.referer }), which worked. Thank you! p.s. I don't need to modify my response

[NoraGithub]

Is there an official solution? Asking for help.

[NoraGithub]

@sergeymakoveev Thanks for your code. I read it. If the framework doesn't call res.writeHead would result in some problems, like I use in my koa server not webpack_dev_server. It just use res.setHeader.

[Njerschow]

I have tried the solutions above and unfortunately I am getting a junked body. Here is my code:

onProxyRes : function(proxyRes, req, res) {
            var body = "";
            proxyRes.on('data', function(data) {
                data = data.toString( 'utf-8' );
                body += data;
                console.log(body.toString('utf8'));
            });
},

and here is my output:

�iD����,�Y Ս����� &�GHUN뮬�2�d2V�%m�\�ݵu�m��erfĬ;u�
6���e���#a��w�L�}9C �9E"a2𑥔C\;�!�zc�Z�@l$<� ��t�)XA�r�QiL<B�Y�_���1Kze�$�f@�c�uJ�E��EJ�q��y).���c�/#�eۓm}�V3���y�t}�d��N-�p�)L�/��o�-�_�媚�uEDȽ����)�-�8 V����>R���6�ζ��4\��M�M��Ǿ��W���LJ�3|�u�J�=Z���G��ƏP-��+��E'�KLT�ޘ������줠��k=�{��B��G�������A
7�3�r��<�)�'8��Z��>^IU5[;���F��8��Y�ܘM1���k9�nj�S�Y�jQ]����v��Ma�`

I am expecting a json response, but everything I've tried doesn't seem to be working.

Thank you for response in advance.

[Njerschow]

I solved my own problem, the response from the server was gzipped, so I simply set the "accept-encoding" header like so:
'accept-encoding': 'gzip;q=0,deflate,sdch'

[xingshijie]

The easy method

onProxyRes: function (proxyRes, req, res) {
    const _pipe = proxyRes.pipe
    // can get resp body from proxyRes pipe

    var body = new Buffer('test1234');

    proxyRes.pipe = function (res) {
      res.write(body)
      res.end()
    }
  }

[ViggoV]

Can anyone tell me what type proxyRes is? Documentation features an example or two but I'm desperately looking for an overview/api reference of how it actually works..

[ViggoV]

Just had a minor epiphany and double checked.. proxyRes is a Node stream.Readable instance.. Might be valuable to include in the docs :)

[mrt123]

My solution:

const zlib = require('zlib');
...
onProxyRes: (proxyRes, req, res) => {
    let originalBody = new Buffer('');
    proxyRes.on('data', function (data) {
        originalBody = Buffer.concat([originalBody, data]);
    });
    proxyRes.on('end', function () {
        const bodyString = zlib.gunzipSync(originalBody).toString('utf8')
        const objectToModify = JSON.parse(bodyString)
        objectToModify.modification = 'Mickey Mouse'
        res.end(JSON.stringify(objectToModify));
    });
},
selfHandleResponse: true . // necessary to avoid res.end being called automatically
...

[eschaefer]

@mrt123 's response got us 90% there. But if you need to gzip your response before sending it back out, here's how to do that:

 onProxyRes: (proxyRes, req, res) => {
  let originalBody = Buffer.from([]);
  proxyRes.on('data', data => {
    originalBody = Buffer.concat([originalBody, data]);
  });

  proxyRes.on('end', () => {
    const bodyString = zlib.gunzipSync(originalBody).toString('utf8');
    const newBody = doSomeReplacementStuff(bodyString);

    res.set({
      'content-type': 'text/html; charset=utf-8',
      'content-encoding': 'gzip'
    });
    res.write(zlib.gzipSync(newBody));
    res.end();
  });
},
selfHandleResponse: true,

[dvelasquez]

I'm looking to do something similar, but transforming the encoding from latin1 to utf-8. I also need to preserve the original headers, but I'm getting unformatted data.

[sachinmour]

I think this is the correct way to do it according to the official documentation modify-response

app.use('/api', proxy({
  target: 'http://www.example.org',
  changeOrigin: true,
  selfHandleResponse: true, // so that the onProxyRes takes care of sending the response
  onProxyRes: function(proxyRes, req, res) {
    var body = new Buffer('');
    proxyRes.on('data', function(data) {
      body = Buffer.concat([body, data]);
    });
    proxyRes.on('end', function() {
      body = body.toString();
      console.log("res from proxied server:", body);
      res.end("my response to cli");
    });
  }
}));

[devinrhode2]

Not sure why, but when I use @sachinmour's code the "res from proxied server:" has encoding errors, and I mostly see diamonds with question marks inside. Furthermore, when I try to load a page, it gets downloaded, and viewing the file I can see the same encoding errors that I see in terminal or chrome console :(

[devinrhode2]

Lo-and-behold, the previous example from @eschaefer does not have these encoding errors. I guess I needed gzip...!

[andyseonet]

Sorry for adding to an old thread but this may save someone hours and potentially days in searching for a solution to removing headers from an http-proxy response.

The way of using selfHandleResponse seems to be a little bit of a blunt instrument for me and it didn't take into consideration streaming and forces the nodejs to buffer the full response before sending and for large files this could be a problem.

I found by looking at the library that it would be possible to remove headers for example by looking for the proxyResp events without specifying the selfHandleResponse option, the important thing to note was to ensure that the headers are in the correct order as browsers can be very picky about the ordering of the headers.

It is indeed possible to simply to iterate and copy the headers across by object name however this will loose the header order and browsers will start to half load pages or just fail for some content types.

Here's a simplified version of proxyRes event listener with the logic i have used - this will allow the proxy to stream as it should and remove headers for example if you don't want to disclose the apache or tomcat version or whatever header you wish to hide.

I'm not sure to the extent that it will handle any complex domain rewrites etc that you might have setup and you should test and handle them but it's worth it to keep streaming imho.

let headersToRemove = ['X-Forwarded-For','Apache-Version',........];

proxy.on('proxyRes', function (proxyRes, req, res) {
  if(!res.headersSent){
    proxyRes.headers = {};
    for(let x = 0; x < proxyRes.rawHeaders.length; x+=2) {
      if(headersToRemove .indexOf(proxyRes.rawHeaders[x]) == -1) { 
        proxyRes.headers[proxyRes.rawHeaders[x]] = proxyRes.rawHeaders[x+1];
      }
    }
  }
});

Hope it helps someone.

Cheers

[quanru]

how do i modify response with condition?

[lmcsu]

I needed to just modify html content, none of the examples above worked for me properly.
So here's my solution, maybe it will help someone

selfHandleResponse: true,
onProxyRes (proxyRes, req, res) {
    const bodyChunks = [];
    proxyRes.on('data', (chunk) => {
        bodyChunks.push(chunk);
    });
    proxyRes.on('end', () => {
        const body = Buffer.concat(bodyChunks);

        // forwarding source status
        res.status(proxyRes.statusCode);

        // forwarding source headers
        Object.keys(proxyRes.headers).forEach((key) => {
            res.append(key, proxyRes.headers[key]);
        });

        // modifying html content
        if (proxyRes.headers['content-type'] && proxyRes.headers['content-type'].includes('text/html')) {
            let html = body.toString();

            // do whatever you want
            html = html.replace('foo', 'bar');

            res.send(new Buffer.from(html));
        } else {
            res.send(body);
        }

        res.end();
    });
},

[mindplay-dk]

I solved my own problem, the response from the server was gzipped, so I simply set the "accept-encoding" header like so:
'accept-encoding': 'gzip;q=0,deflate,sdch'

This instructs the server to use a different type of compression - you were just lucky, since the server you were trying only supported gzip and not deflate or sdch.

What you want is {'accept-encoding': 'identity'}, which instructs the server not to compress.

Even then, some servers may compress regardless, and you will need @eschaefer's solution to decompress. (source)

It would be nice if this package provided a solution to this common problem - compression handled and all.

[aedenmurray]

I solved my own problem, the response from the server was gzipped, so I simply set the "accept-encoding" header like so:
'accept-encoding': 'gzip;q=0,deflate,sdch'

This instructs the server to use a _different_ type of compression - you were just lucky, since the server you were trying only supported gzip and not deflate or sdch.

What you want is {'accept-encoding': 'identity'}, which instructs the server not to compress.

Even then, some servers may compress regardless, and you will need @eschaefer's solution to decompress. (source)

It would be nice if this package provided a solution to this common problem - compression handled and all.

Dude you saved my job

[Zver64]

I have tried all solutions above but I keep getting the original response from the server.... In other words res.write(.. or res.end(... doesn't do anything for me...
This is my last attempt

onProxyRes(proxyRes, req, res) {
  console.log('req url ', req.url);
  if (urlRegexp.test(req.url)) {

    if (req.url.includes(queryName)) {

      proxyRes.on('end', function () {
        const bodyString = originalBody.toString()
        const objectToModify = JSON.parse(bodyString)
        console.log('the data: ', objectToModify);
        objectToModify.modification = 'Mickey Mouse';
        console.log('the data modified: ', objectToModify); // this fires up in my console with the right data. But I still get the original response
        res.end(JSON.stringify(objectToModify));
      });
    }
  }
};