Elasticsearch-js: Access Through Proxy

[kelaban]

Having trouble connecting through a corporate proxy to a host.
the http module in node by can connect through a proxy by specifying

{
    host: 'proxy.example.com',
    path: 'http://example.elasticsearch.com'
}

however specifying this in the elasticsearch hosts configurations don't work as expected.

[spalger]

Perhaps the issue is that hosts in the elasticsearch.js client default to port 9200 (elasticsearch's default port).

If you turn on 'trace' logging you will get pretty detailed info about what is going on which would help me debug the issue.

var client = new elasticsearch.Client({ host: ..., log: 'trace' });

Code I used to replicate the issue: https://gist.github.com/spenceralger/b16b8b30ca3eba774d1e

[kelaban]

yes, I've tried this, perhaps I should have mentioned this in the initial post. At first I did not specify port so it was trying something like http://proxy.example.com:9200/http://example.elasticsearch.com

After that I changed configs to:

{
    host: 'proxy.example.com',
    port: 80,
    path: 'http://example.elasticsearch.com'
}

which yielded
http://proxy.example.com:80/http://example.elasticsearch.com

this is the incorrect behavior. it should be sent as

GET http://example.elasticsearch.com HTTP/1.1
Host: http://proxy.example.com:80

i.e. a request to path and setting host header.

[spalger]

Well elasticsearch.js uses node's http module under the covers, and the host and path are specified separately. Formatting the actual text of the HTTP response is of course out of my control...

[kelaban]

I think the issue here is that the path is being concatenated to the host param

[spalger]

@kelaban As I said previously, I'm not concatenating the host and path params. They are specified separately here. Perhaps the issue is that the path is prepended with a forward slash?

_Please_ provide an error message from your proxy, or log output from the client, to receive further help.

[kelaban]

From the trace with hostname and proxy name replaced. The host configuration is what is shown above.

Elasticsearch TRACE: 2014-05-14T17:29:38Z
  -> POST http://proxy.example.com:80/http://host.example.com:9200/_bulk

the proxy returns a 403 access denied response.

However

curl -x http://proxy.example.com:80 http://host.example.com:9200 works as expected. As well as using the above configurations with http.request

Also this little node example works as expected

var http = require('http');

http.request({
  method: 'GET',
  hostname: 'proxy.example.com',
  port: '80',
  path: 'http://example.host.com:9200'
}, function (res) {
  var data = '';
  res.on('data', function(chunk) { data += chunk; } );
  res.on('end', function() { console.log(data); } );
  res.on('error', function(err) { console.log(err); } );
}).end();

Edit: I also tested this with a method: 'POST' http.request and it worked as expected, so it is not an issue with the verb going through the proxy

[spalger]

Wow, so leading slashes on paths are optional... Didn't expect that.

Can you check out this gist and try using the customized HttpConnector class? It will strip out leading slashes if they have been added.

[kelaban]

That does the trick! Interesting, does the leading slash get prepended somewhere in the client?

[spalger]

It does. In browser's it is very important to have the leading slash, which is why I force it. I would have never guessed that it would have any impact in non-browser environments.

I will consider the impact of removing this from the client but in the meantime I imagine the custom HttpConnector will work for you.

[minervas]

@spalger
I think I'm having the same problem only with the msearch function.

Elasticsearch INFO: 2015-07-01T00:23:53Z
  Adding connection to http://proxy.example.com:proxyport/http://host.example.com:9200/
Elasticsearch DEBUG: 2015-07-01T00:23:53Z
  starting request { method: 'POST',
    bulkBody: true,
    path: '/_msearch',
    body:
     [ { index: 'the_index' },
       { query: [Object] } ],
    query: { search_type: 'count' } }

Only I am just getting a timeout.
Unfortunately the gist you mentioned is down. Could you repost it?

[neilha]

Hi. I am having the same problem. Would it be possible to make the gist withe the customized httpconnector available again? I get 404 when I try to access the link. Thanks.

[ryan65]

Hi

I opened another issue on this https://github.com/elastic/elasticsearch-js/issues/402
If you modify host.js not to add leading slash then it works.

[clakech]

Maybe this code can help someone.

Add an env variable to reach your proxy HTTP_PROXY = http://127.0.0.1:5000/

const elasticsearch = require('elasticsearch');
const proxy = require('proxy-agent');

const client = new elasticsearch.Client({
  host: 'my.host.com',
  createNodeAgent: () => proxy(process.env.HTTP_PROXY)
});

client.search(query);

Au revoir :kissing_closed_eyes:

[imran-uk]

@clakech Thanks very much, that worked for us too.

I had to code it slightly differently to cater for the PROXY env var being missing - in whichcase a direct connection is made:

'use strict';

const elastic = require('elasticsearch'),
      proxy = require('proxy-agent'),
      config = require('../../config/config.js');

let nodeAgent;

if(process.env.MY_ELASTIC_PROXY) {
  nodeAgent = () => proxy(process.env.MY_ELASTIC_PROXY);
}

const client = new elastic.Client({
  host:               config.elastic.host,
  log:                config.elastic.log,
  suggestCompression: config.elastic.compression,
  apiVersion:         config.elastic.apiVersion,
  createNodeAgent:    nodeAgent
});