Node: HTTP Client and automatic request retry

[papakai]

• Version: v6.10.0
• Platform: debian
• Subsystem:

I tried to figure out, how the node http client behaves in case of a premature connection close?
Why I am asking on that is covered here: https://www.ietf.org/rfc/rfc2616.txt, section "Client Behavior if Server Prematurely Closes Connection" @ Page 50.

I have not found anything covered in node.js source code which would automatically retry a request dependent or indepent of the expect header usage (which would be great). Does anyone know if there's a case where the node http client library would automatically retry a request? Has anyone else deeper insights?

TIA

[mscdex]

There is no automatic retrying of requests in node core.

[thelostone-mc]

@mscdex Would that be something which could be considered ?
As of now when I face this issue I end up using a npm module like promise-retry !

Lot of folks who just start out on node usually start with the usecase of crawling websites -> hit a block when the server closes the connection / socket hangup and get annoyed.

One could argue that it's not a required feature but it would be something that would be nice to have via an optional parameter or so!

[Trott]

As of now when I face this issue I end up using a npm module

I think that's appropriate.

[Trott]

Would that be something which could be considered ?

It might depend on the implementation. If it's a small amount of code that seems unlikely to result in support problems (bugs to fix, performance issues, etc.), it's likely to get more support.

@nodejs/http

[mscdex]

IMHO this kind of feature is best left to userland. You shouldn't even need to install a module for this, it's easily implemented manually (then again I'm not a proponent of those extremely small npm modules).

[thelostone-mc]

@mscdex True that implementing it is pretty straightforward but the npm modules are well tested and if ones feeling lazy, that would be the way to go.
But I was talking from a newbies perspective. With node get popular, lot of folks who just started using node ask me this. My response would be to write it yourself ( as you suggested ) and if they weren't up for that , you have npm.

Like I said, it would be a luxury rather than a necessity to have this built into node.

[mscdex]

@adityaanandmc The same can be said for many other features implemented in userland modules like request, for example: automatically following redirects, submitting urlencoded and multipart forms, automatic buffering of responses, proxy support, cookie jars, automatic gzip handling, etc.

Adding such features to core does not make much sense to me and an 'automatic retry' feature falls into this same feature bucket.

[thelostone-mc]

@mscdex Haha that is probably true but my point of view was entirely based on what newbies face while using request !! Nothing more.

retry is something in my opinion which everyone who uses request would have to handle. I may be wrong but in all my usecases where I've used request , handling retires has always been a constant.

In an ideal world , all those addon(s) would probably be adopted into the core. :P

[mscdex]

@adityaanandmc Then submit a PR to request that adds a retry config option.

[papakai]

If you add this retry functionality, I would prefer to have a config option to also disable it as this is something we don't want to have activated by default or what we want to have disabled :)
Having that deactivated by default should then also be downward compatible.

[papakai]

question: in case a server prematurely closes connection, is that case then also covered through "end" event of server response? or have I explicitely to subscribe the "close" event to return any response to the client?

[jasnell]

Automatic retry is not likely something that can or should be added to core. It's also problematic in the protocol itself as it can only work with safe and idempotent methods like GET and HEAD. HTTP/2 includes mechanisms to make it easier but even then, it's better left to user land.

[benjamingr]

To be fair, I don't think retry should even be part of the HTTP stack - there are modules for retrying a function independently of requests.

[papakai]

Hey guys,

I really need your help here. I am trying to figure out on how to react in http request on a server close.
Currently we always need to wait for the timeout to be fired. Server also shuts down after the timeout and not earlier even though close() was already called.

var req = require('http').request(someconfig, function(res) {
  var response = "";
  res.setEncoding('utf8');
  res.on('data', function(chunk) {
    response += chunk;
  });
  res.on('end', function() {
    // now we return that response to client
    response = "";
  });
});
  //250s (to be sure to get an API response)
  req.on('socket', function(socket) {
    socket.setTimeout(250000, function() {
      req.abort();
    });
  });
  //e.message = 'problem with request: ' + e.message;
  req.on('error', function() {
    //here we return a common error message to client
  });
  req.write(somedata);
  req.end();

I tried out different things, but couldn't find the solution. Can anyone assist here how to react there on server connection 'close' without having to wait until timeout is getting fired?

Here a simple server I wrote for that purpose:

'use strict';

var sleep = require('sleep');
var app = require('express')();
var DEFAULT_HOST = '0.0.0.0';
var DEFAULT_PORT = 8083;

//this is covered through timeout
app.use('/call/prematurelyclose1', function(req, res) {
  console.log('/call/prematurelyclose1 accessed');
  sleep.sleep(310); // sleep for ten seconds
  console.log('send empty response after 310s');
  res.end();//timeout is getting fired earlier!
});

//this leads to empty response
app.use('/call/prematurelyclose2', function(req, res) {
  console.log('/call/prematurelyclose2 accessed');
  sleep.sleep(10); // sleep for ten seconds
  console.log('send empty response after 10s');
  res.end();
});

//this is covered through timeout
app.use('/call/prematurelyclose3', function() {
  console.log('/call/prematurelyclose3 accessed');
  sleep.sleep(310); // sleep for ten seconds
  console.log('server close after 310s');
  server.close();
});

//this is covered through timeout, but really need to wait (client + server) for that timeout? 
app.use('/call/prematurelyclose4', function() {
  console.log('/call/prematurelyclose4 accessed');
  sleep.sleep(10); // sleep for ten seconds
  console.log('server close after 10s');
  server.close();
});

var server = require('http').createServer(app).listen(DEFAULT_PORT, DEFAULT_HOST);
server.on('listening', function() {
  console.log("server running: (" + DEFAULT_HOST + ":" + DEFAULT_PORT + ").");
});

module.exports = app;

As you may recognize, I am not sure how to exactly trigger the server to prematurely close the connection. So I wrote these 4 test cases. Above I am basically speaking about test case 4 where I have no clue on how to react on that server close, it simply waits until socket timeout.

[Trott]

Hi @pabigot @papakai! If I'm understanding correctly, at this point, I'd say this isn't a bug report in Node.js core or a feature request anymore. It seems like a support request. Would you mind reposting over in https://github.com/nodejs/help? You can also try the #Node.js channel on Freenode IRC. And, of course, there are lots of unofficial places to get support like StackOverflow.

Thanks!

[pabigot]

@Trott I think you meant @papakai as I have no idea what this is about.

[Trott]

Yup, sorry.

[papakai]

Yes, sorry. Didn't know about https://github.com/nodejs/help. Yesterday I got pointed on that resource too in another ticket. Thanks anyway to all in here :)