Browser-sync: Proxying so only one request is forwarded to the WebServer
Issue details
I have 2 web browsers synchronized with Browser Sync and a WebServer connected to a DataBase. I would like to know if there is any option, plugin or solution of any kind so only the requests of the first browser are forwarded to the WebServer. BrowserSync would then wait this request to complete and serve the same response to the 2 browsers.
This would fulfill my needs:
- having only one session;
- don't duplicate the entries (the database wouldn't allow it);
- (as a bonus) compare the requests;
Thanks for your great work BTW!
Please specify which version of Browsersync, node and npm you're running
- Browsersync [ 2.24.7 ]
- Node [ v8.11.3 ]
- Npm [ 5.6.0 ]
Affected platforms
- [x] linux
- [x] windows
- [ ] OS X
- [ ] freebsd
- [ ] solaris
- [ ] other _(please specify which)_
Browsersync use-case
- [x] API
- [ ] Gulp
- [x] Grunt
- [x] CLI
If CLI, please paste the entire command below
{cli command here}
for all other use-cases, (gulp, grunt etc), please show us exactly how you're using Browsersync
{Browsersync init code here}
fflorent
All 22 comments
Also very curious about this @fflorent as we have a similar use case and browsersync would be the perfect fit if it can do this as it is already part of our dev dependencies.
LewisRogers
on 10 Oct 2018
You can plug-in the router from Express, and then use any middlewares - I made an example here
shakyShane
on 13 Oct 2018
@shakyShane Are you really sure that only one request is forwarded to the server when there are two browsers? I would expect that the two browsers forward one request each instead, even with a middleware.
fflorent
on 14 Oct 2018
@fflorent ahh! so sorry, I totally mis-understood your original post - I think I may of mixed it up with another :)
What you're suggesting would be an amazing feature for sure - I'm just not sure if would even be possible (to do it well)
shakyShane
on 14 Oct 2018
@shakyShane Thanks for your feedback!
What do you think of this?
- The requests of the first browser are sent and cached with a key being the checksum of concatenation of the method (POST, GET, …), the path and the body;
- The user may filter the data from the body and the path passed to the hash function in order to skip or fix some useless data (like timestamps);
- The following requests matching the keys are replied with the response of the first request either:
- directly when the first request is complete;
- when the first request is done otherwise;
- The user may also transform the response;
- Each cache entry may be deleted after a certain number of use (once per replicating browsers) and after a certain time;
Also I wonder if defining a primary browser and replicating browsers wouldn't help simplify / stabilize the feature.
fflorent
on 14 Oct 2018
@shakyShane (also: would you mind reopening the issue unless you are quite sure it is impossible to implement? :-))
fflorent
on 14 Oct 2018
@fflorent I've implement the idea of a 'controller' in the past, it just never made it to stable for various reasons.
It worked well for limititing things like who could broadcast 'sync' events etc, but I'm not entirely sure it would work in this case.
If your issue is trying to prevent CRUD operations on a DB, I believe the best option there is to just disable all sync features - the idea of limiting a form submission (for example) to a single browser sounds incredibly error-prone.
shakyShane
on 14 Oct 2018
@fflorent having said all of that, I'd happily show you or anyone else where or how to begin thinking about implementing such a feature if you wanted to contribute
shakyShane
on 14 Oct 2018
having said all of that, I'd happily show you or anyone else where or how to begin thinking about implementing such a feature if you wanted to contribute
@shakyShane Yeah, I would be happy to give it a try :).
fflorent
on 14 Oct 2018
@fflorent I had a similar idea for a solution to this feature. Would be v.happy to talk this through / help you out if you wanted a hand to get this one going? @shakyShane a little insight into where to start on this would be great too 🙂
LewisRogers
on 15 Oct 2018
I hacked a little bit these days and managed to make something that almost works (at least for GET requests but not for POST or PUT ones for some reasons)
This hack works in our context, we pass JSON data when we submit data.
How it works:
- we add a middleware to intercept requests that may be cached or already in progress for other browsers;
- if the request is not known of the cache, we let the request go to the server and we store a promise of its response;
- when the request is responded (proxyRes), we read it's data and resolve the promise stored in the cache;
Warning: this code is uggly, that's a POC ;).
// Native
const path = require('path');
const crypto = require('crypto');
const url = require('url');
// Contrib
const bodyParser = require('body-parser');
const express = require('express');
const proxy = require('http-proxy-middleware');
const browserSync = require("browser-sync");
function hashRequest(req) {
const hash = crypto.createHash('md5');
hash.update(req.method);
const {pathname} = url.parse(req.url);
hash.update(pathname);
// Something we don't want but some users may do
// hash.update(req.query);
hash.update(JSON.stringify(req.body));
return hash.digest('hex');
}
const recordedRequests = new Map();
recordedRequests.cache = function (key, ...args) {
// Automatically clear cache after 10s so we don't leak memory
setTimeout(() => {
this.delete(key);
}, 1e4);
return this.set(key, ...args);
}
async function middleware(req, res, next) {
const cacheKey = hashRequest(req);
console.log("cacheKey = %s (url = %s)", cacheKey, req.url);
const cachedRequestPromise = recordedRequests.get(cacheKey);
if (cachedRequestPromise) {
const cachedRequest = await cachedRequestPromise;
for (let [headerName, headerValue] of Object.entries(cachedRequest.headers)) {
res.setHeader(headerName, headerValue);
}
res
.status(cachedRequest.status)
.end(cachedRequest.content);
} else {
const cachedRequest = new Promise(resolve => {
// work like a "defer" promise, we store the `resolve` function so we can call it when we receive the response (in proxyRes)
req._resolveCachedRequest = resolve;
});
recordedRequests.cache(cacheKey, cachedRequest);
next();
}
}
const app = express();
app.use(bodyParser.json()); // Probably overkill but this is a simple way to have the request body accessible. I may get rid of it
app.use('/api', middleware);
browserSync({
https: {
key: '/path/to/key.pem',
cert: '/path/to/cert.crt'
},
host: 'localhost',
middleware: [app],
proxy: {
target: 'https://localhost:8080',
proxyRes: function (proxyRes, req) {
if (req._resolveCachedRequest) {
// gather response data in `body` and resolve Promise so the secondary browsers can respond
let body = new Buffer('');
proxyRes.on('data', data => {
body = Buffer.concat([body, data]);
});
proxyRes.on('end', () => {
req._resolveCachedRequest({
status: proxyRes.statusCode,
headers: proxyRes.headers,
content: body
});
});
}
}
}
});
Some thoughts:
- I wonder if hashing in md5 is a good idea for generating the cache key or if it's too expansive. We may would rather store a concatenation of the strings (methods, url and body)
- It's too specific to our context, the idea is that we can make it generic enough at a later point so this is either a generic plugin or be part of the browser-sync project.
TODO:
- [ ] make it work with PUT / POST / DELETE;
- [ ] stabilize it;
- [ ] enhance performance (I feel a little degradation);
What do you think of the idea? (poke @LewisRogers)
fflorent
on 17 Oct 2018
Hey @fflorent this looks awesome!
We participate in the Open Source Friday initiative where I work and would love to get this to the rest of the team on Friday to have a hack on. Particularly to have a look at how we could support the other HTTP methods (Feel free to join us if your available 14:00-16:00 GMT?).
Do you have a branch with this on anywhere?
I also agree with your comment around the specificity of this feature, maybe it should be some sort of pluggable middleware? But a problem I guess we can solve once we have a working stable feature.
LewisRogers
on 17 Oct 2018
Great!
(Feel free to join us if your available 14:00-16:00 GMT?)
Sorry, I am not.
Do you have a branch with this on anywhere?
No, sorry.
I also agree with your comment around the specificity of this feature, maybe it should be some sort of pluggable middleware? But a problem I guess we can solve once we have a working stable feature.
Yes!
Florent
fflorent
on 19 Oct 2018
Alright, I managed to make the POC work (in our context, probably worth to have feedback). The problem was the use of body-parser (which was useless, I managed to get the body using another trick):
// Native
const path = require('path');
const url = require('url');
// Contrib
const express = require('express');
const proxy = require('http-proxy-middleware');
const browserSync = require("browser-sync");
const recordedRequests = new Map();
const rawBodyKey = Symbol('rawBody');
function getCacheKey(req) {
return `method="${req.method}",pathname="${url.parse(req.url).pathname}",rawBodyKey="${req[rawBodyKey].toString('hex')}"`;
}
recordedRequests.cache = function (key, ...args) {
setTimeout(() => {
this.delete(key);
}, 10000);
return this.set(key, ...args);
}
async function middleware(req, res, next) {
const cacheKey = getCacheKey(req);
const cachedRequestPromise = recordedRequests.get(cacheKey);
if (cachedRequestPromise) {
const cachedRequest = await cachedRequestPromise;
for (let [headerName, headerValue] of Object.entries(cachedRequest.headers || {})) {
res.setHeader(headerName, headerValue);
}
res
.status(cachedRequest.status)
.end(cachedRequest.content);
} else {
const cachedRequest = new Promise((resolve, reject) => {
req._resolveCachedRequest = resolve;
});
recordedRequests.cache(cacheKey, cachedRequest);
next();
}
}
const app = express();
app.use('/api', function (req, res, next) {
req[rawBodyKey] = new Buffer([]);
if (req.trailers) {
return next();
}
req.on('data', data => {
req[rawBodyKey] = Buffer.concat([req[rawBodyKey], data]);
});
req.on('end', () => {
next();
});
}, middleware);
browserSync({
https: {
key: '/path/to/key.pem',
cert: '/path/to/cert.crt'
},
host: 'localhost',
middleware: [app],
proxy: {
target: 'https://localhost:8080',
ws: false,
proxyRes: function (proxyRes, req) {
if (req._resolveCachedRequest) {
let body = new Buffer('');
proxyRes.on('data', data => {
body = Buffer.concat([body, data]);
});
proxyRes.on('end', () => {
req._resolveCachedRequest({
status: proxyRes.statusCode,
headers: proxyRes.headers,
content: body
});
});
}
}
}
});
I am not sure if that would work well with files. Also there is no need for a master or a slave with this POC.
Thoughts @LewisRogers @shakyShane?
fflorent
on 24 Oct 2018
Hey @fflorent looks good and cant wait to try. I've been away last week but will 100% have a go this week.
LewisRogers
on 29 Oct 2018
Hey @LewisRogers, I think I was too optimistic, I thought I had read the request body in order to build a unique key for the cache, but in fact it is harder than it seems.
At least, it seems to be quite stable for my project (still there are some cases where this would be requested).
If you get a solution before me, please share, I would be really grateful :).
fflorent
on 29 Oct 2018
Hey @fflorent, So I've been looking into this over the past few days and understand what you mean now by optimism.
I've followed your example and have not been able to get it working for my use case.
To provide some background, I wanted this feature to be able to do some visual regression / xb testing automation work. The problem is that the system currently under test has a very unique way of managing sessions (due to the nature of the business we operate in, it has to be that way).
I'm thinking of another way to achieve this (Probably using Browsersync as a dependency) but instead of mirroring requests and responses, doing something with the MutationObserver DOM API to try and literally just draw back each mutation in the DOM across different devices using a similar caching mechanism as you suggested above.
I'm not sure exactly what you wanted this feature for, but if it is similar to mine, once I get some sort of POC working I can loop you in?
Take it easy.
LewisRogers
on 31 Oct 2018
Hello @LewisRogers,
I managed to make the POC working now, taking into account the body in the cache key. The trick is to use a proxyReq in order to use node-http-proxy (internally used by Browser-Sync) and body-parser. It almost work perfectly (still some issues to address):
// Native
const _ = require('lodash');
const bodyParser = require('body-parser');
const path = require('path');
const url = require('url');
// Contrib
const express = require('express');
const proxy = require('http-proxy-middleware');
const browserSync = require("browser-sync");
const ALLOWED_HEADER_KEYS = […];
const DISMISSED_QUERY_KEYS = […];
const recordedRequests = new Map();
const rawBodyKey = Symbol('rawBody');
function getCacheKey(req) {
let key = `method="${req.method}",pathname="${url.parse(req.url).pathname}"`;
if (req.body instanceof Buffer) {
key += `,body="${req.body.slice(100).toString('hex')}"`;
}
key += ALLOWED_HEADER_KEYS.reduce((str, headerName) => {
const headerValue = req.get(headerName);
if (headerValue) {
// TODO escape headerValue
str += `,header-${headerName}=${headerValue}`;
}
return str;
}, '');
key += Array.from(Object.entries(_.omit(req.query, DISMISSED_QUERY_KEYS)))
.reduce((str, [key, value]) => {
return `${str},query-${key}=${value}`;
}, '');
console.log("key = ", key);
return key;
}
recordedRequests.cache = function (key, ...args) {
setTimeout(() => {
this.delete(key);
}, 10000);
return this.set(key, ...args);
}
async function middleware(req, res, next) {
const cacheKey = getCacheKey(req);
const cachedRequestPromise = recordedRequests.get(cacheKey);
if (cachedRequestPromise) {
const cachedRequest = await cachedRequestPromise;
for (let [headerName, headerValue] of Object.entries(cachedRequest.headers || {})) {
res.setHeader(headerName, headerValue);
}
res
.status(cachedRequest.status)
.end(cachedRequest.content);
} else {
const cachedRequest = new Promise((resolve, reject) => {
req._resolveCachedRequest = resolve;
});
recordedRequests.cache(cacheKey, cachedRequest);
next();
}
}
const app = express();
app.use('/api', bodyParser.raw({type: '*/*'}), middleware);
browserSync({
https: {
key: '/path/to/key.pem',
cert: '/path/to/cert.crt'
},
host: 'localhost',
middleware: [app],
proxy: {
target: 'https://localhost:8080',
ws: false,
proxyReq: [function (proxyReq, req) {
if (req.body instanceof Buffer) {
proxyReq.write(req.body);
}
}],
proxyRes: [function (proxyRes, req) {
if (req._resolveCachedRequest) {
let body = new Buffer('');
proxyRes.on('data', data => {
body = Buffer.concat([body, data]);
});
proxyRes.on('end', () => {
req._resolveCachedRequest({
status: proxyRes.statusCode,
headers: proxyRes.headers,
content: body
});
});
}
}]
}
});
The problem is that the system currently under test has a very unique way of managing sessions (due to the nature of the business we operate in, it has to be that way).
I see. Is the session key transmitted by HTTP though?
I'm thinking of another way to achieve this (Probably using Browsersync as a dependency) but instead of mirroring requests and responses, doing something with the MutationObserver DOM API to try and literally just draw back each mutation in the DOM across different devices using a similar caching mechanism as you suggested above.
I see. In our case, we would like to see DOM differences, as we rely on a Framework (ExtJS) which can have a different behavior and thus generate a different DOM depending on the browser. Browser-Sync would help us a lot qualifying our application.
once I get some sort of POC working I can loop you in?
Sure!
Florent
fflorent
on 2 Nov 2018
@LewisRogers I have made a plugin based on this POC. I am about to open-source this. Stay tuned!
fflorent
on 13 Nov 2018
Looking forward to seeing it in action @fflorent 🙂
LewisRogers
on 13 Nov 2018
Maybe disable ghosting?
soryy708
on 18 Dec 2018
@soryy708 how would it help?
fflorent
on 19 Dec 2018
Related issues
tonyoconnell
·
3Comments
hgl
·
3Comments
demisx
·
4Comments
danielverejan
·
3Comments
ericmorand
·
3Comments
Most helpful comment
I hacked a little bit these days and managed to make something that almost works (at least for
GETrequests but not forPOSTorPUTones for some reasons)This hack works in our context, we pass JSON data when we submit data.
How it works:
Warning: this code is uggly, that's a POC ;).
Some thoughts:
TODO:
What do you think of the idea? (poke @LewisRogers)