Elasticsearch-py: New ConnectionError [Errno 111] Connection Refused using Docker

Created on 22 Jan 2018 · 18Comments · Source: elastic/elasticsearch-py

I have elasticsearch running in one docker container and elasticsearch-dsl installed in another docker on the same network.

I am trying to establish a connection between the two but am getting an error. Elasticsearch is up and running.

I entered the docker with elasticsearch-dsl installed.

from elasticsearch import Elasticsearch
connections.create_connections(hosts=['localhost:9200'])
client = Elasticsearch()
client.ping()

I get an error here, copied below. Is there a configuration step I'm missing?

Thanks

HEAD http://localhost:9200/ [status:N/A request:0.000s]
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.5/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/usr/local/lib/python3.5/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/elasticsearch/connection/http_urllib3.py", line 149, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=request_headers, *kw)
File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.5/site-packages/urllib3/util/retry.py", line 333, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.5/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 357, in _make_request
conn.request(method, url, *httplib_request_kw)
File "/usr/local/lib/python3.5/http/client.py", line 1107, in request
self._send_request(method, url, body, headers)
File "/usr/local/lib/python3.5/http/client.py", line 1152, in _send_request
self.endheaders(body)
File "/usr/local/lib/python3.5/http/client.py", line 1103, in endheaders
self._send_output(message_body)
File "/usr/local/lib/python3.5/http/client.py", line 934, in _send_output
self.send(msg)
File "/usr/local/lib/python3.5/http/client.py", line 877, in send
self.connect()
File "/usr/local/lib/python3.5/site-packages/urllib3/connection.py", line 166, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.5/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused
HEAD http://localhost:9200/ [status:N/A request:0.000s]
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.5/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/usr/local/lib/python3.5/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Source

laubnyc

Most helpful comment

You have 2 containers on the same network.

Are you using Docker-compose?

Localhost is probably the wrong hostname to use.

When Docker networks containers you can communicate between containers by using the name of the containers.

So say you named your elasticsearch container elasticsearch you would then want to establish a connection to ES using the name elasticsearch instead of localhost.

By using localhost the container running -dsl-py will attempt to connect to WS running within the same container. But it will fail cause ES is available in a different container.

The reason I asked about compose is because it makes it easy to name your services and the name of each service also doubles as a DNS name to reach said service.

fxdgear on 22 Jan 2018

👍23 ❤11 🚀9

All 18 comments

You have 2 containers on the same network.

Are you using Docker-compose?

Localhost is probably the wrong hostname to use.

When Docker networks containers you can communicate between containers by using the name of the containers.

So say you named your elasticsearch container elasticsearch you would then want to establish a connection to ES using the name elasticsearch instead of localhost.

By using localhost the container running -dsl-py will attempt to connect to WS running within the same container. But it will fail cause ES is available in a different container.

The reason I asked about compose is because it makes it easy to name your services and the name of each service also doubles as a DNS name to reach said service.

fxdgear on 22 Jan 2018

👍23 ❤11 🚀9

@fxdgear thanks for the reply

Yes, my container with the elasticsearch image is called 'drv_elastic' and I am using a docker-compose.yml.

I just tried connections.create_connections(hosts=['drv_elastic'])

This returned the same error when I tried to ping. Should I take another approach.

Here are relevant parts of the docker-compose.yml:
[drv_viz_web is where elasticsearch-dsl is installed]

DRV VIZ

drv_viz_web:
  restart: always
  hostname: drv_viz_web
  build: ./drv_viz/web
  expose:
    - "8003"
  volumes:
    - /usr/src/app
    - /usr/src/app/static
  depends_on:
    - drv_viz_postgres
    - drv_base_web
    #- drv_elastic [Note: I've tried with and without this line]
  environment:
    - LOCAL_TESTING=false
    - DEBUG=0
    # a bunch of sensitive info
  command: /usr/src/app/startup.sh
  networks:
    - deploy_network

DRV ELASTIC ###############

drv_elastic: 
    restart: on-failure:1 
    hostname: drv_elastic 
    build: ./drv_forest/drv_elastic 
    environment: 
      - cluster.name=drv-elastic-cluster 
      - bootstrap.memory_lock=true 
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g -Des.enforce.bootstrap.checks=true" 
    ulimits: 
      memlock: 
        soft: -2 
        hard: -2 
    mem_limit: 2g 
    volumes: 
      - drvstore_elastic_es:/usr/share/elasticsearch/data 
    ports: 
      - 9200:9200 
    networks: 
      - deploy_network

laubnyc on 23 Jan 2018

@laubnyc sorry delay :(

So I'm looking at your drv_elastic service, a couple of notes:

you are using your own build of ES. So I don't know what exactly is going on with your custom ES image. This could or could not be an issue. There's just not enough info to tell
you are exposing 9200 externally to the containers, so you should be able to go to localhost:9200 in your browser and access ES. You should get a default response back from ES.
If step 2 fails, then we know that there might be a problem with the ES container and i'd recommend looking at the logs to verify that ES is in fact up and running.
If possible i'd recommend that you use the official docker images that Elastic provides.
- docker.elastic.co/elasticsearch/elasticsearch:6.1.1 (with x-pack)
  
  or
- docker.elastic.co/elasticsearch/elasticsearch-oss:6.1.1 (without x-pack)
  
  you can feel free to change the version number to which ever version you need.

fxdgear on 23 Jan 2018

Here is a very very simple docker-compose file that illustrates the connection between an ES container and another container:

test.yml:

version: "3.3"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.1.1
    ports:
      - "9200:9200"
    networks:
      - esnet
    environment:
      - bootstrap.memory_lock=false

  client:
    image: appropriate/curl:latest
    networks:
      - esnet
    command: "curl http://elasticsearch:9200"
networks:
  esnet:

Now to test first in one shell run:

docker-compose -f test.yml up elasticsearch

watch the output and wait till it stops.

Now in a new shell run:

docker-compose -f test.yml up client

you should see an output like this:

client_1         |   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
client_1         |                                  Dload  Upload   Total   Spent    Left  Speed
100   433  100   433    0     0    433      0  0:00:01 --:--:--  0:00:01 39363
client_1         | {
client_1         |   "name" : "JzqdMOC",
client_1         |   "cluster_name" : "docker-cluster",
client_1         |   "cluster_uuid" : "WETuEUARSiSgT4OtSPsUcQ",
client_1         |   "version" : {
client_1         |     "number" : "6.1.1",
client_1         |     "build_hash" : "bd92e7f",
client_1         |     "build_date" : "2017-12-17T20:23:25.338Z",
client_1         |     "build_snapshot" : false,
client_1         |     "lucene_version" : "7.1.0",
client_1         |     "minimum_wire_compatibility_version" : "5.6.0",
client_1         |     "minimum_index_compatibility_version" : "5.0.0"
client_1         |   },
client_1         |   "tagline" : "You Know, for Search"
client_1         | }
pythonclient_client_1 exited with code 0

fxdgear on 23 Jan 2018

@fxdgear: I ran into a similar problem as @laubnyc, and asked a question on https://stackoverflow.com/questions/48711455/create-dockerized-elasticsearch-index-using-a-python-script-running-in-docker . If I do your exact curl edit to my docker-compose yml I get the response fine from the elasticsearch service, but still have the ConnectionRefusedError: [Errno 111] Connection refused with my python script.

davidefiocco on 9 Feb 2018

👍1

@davidefiocco it looks like from your pastebin link on stack overflow your script is trying to connect to localhost and not elasticsearch.

Creating elasticsearch ... done HEAD http://localhost:9200/test [status:N/A request:0.002s]

Please make sure your script is configured to connect to elasticsearch.

If you can create a simple gh repo with everything you’re trying to do and share here I can review it and give you some feedback.

fxdgear on 9 Feb 2018

Also, there is a synchronisation bug in there, because the indexer.py script doesn't make sure that elasticsearch really is up before it tries to query it.

mihaitodor on 9 Feb 2018

👍2

@fxdgear @mihaitodor Thanks both! Mihai nailed it, here's code that connects fine to elasticsearch without the [Errno 111]: https://github.com/davidefiocco/dockerized-elasticsearch-indexer

davidefiocco on 10 Feb 2018

👍1

@davidefiocco @mihaitodor

ahh yes that would make sense.

Another option in your indexer script is to put the sleep in a while loop:

import time
connected = False

while not connected:
    try:
        es.info()
        connected = True
    except ConnectionError:
        print("Elasticsearch not available yet, trying again in 2s...)
        time.sleep(2)

edit:
or even better:
https://github.com/elastic/elasticsearch-py/blob/master/elasticsearch/helpers/test.py#L22-L28

fxdgear on 12 Feb 2018

👍5 🎉1

@fxdgear I'd prefer to use exponential backoff, though :) Maybe something like @backoff.on_exception from this package: https://pypi.python.org/pypi/backoff in combination with the client.cluster.health(wait_for_status='yellow') trick that you pointed out.

mihaitodor on 13 Feb 2018

Yeah. For sure.

fxdgear on 13 Feb 2018

I know this issue was closed a while ago but I wanted to post a solution that I found to work in case someone has the same problem in the future. (Note: I'm not using docker-compose) I was using the Python API for Elasticsearch with two separate containers (one for my flask application, and one for Elasticsearch). What I found to work was the following:

docker run --name elasticsearch -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -v esdata:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:6.3.2

docker run --name <some_name> --rm --link elasticsearch:elasticsearch --env ELASTICSEARCH_URL=http://elasticsearch:9200 <repo_name>

I simply had to link the containers. Hope it helps someone!

nromano7 on 22 Feb 2019

@nromano7 That doesn't save you from the synchronisation bug, unless your code does retries.

mihaitodor on 22 Feb 2019

@mihaitodor can you help me check that bug, I have found many ways but cannot fix it:

File "models.py", line 178, in search_meo
search = es.search(index = table_name, doc_type=table_name, body = body)
File "/home/foodforfamilyf3/.virtualenvs/deploy/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 73, in _wrapped
return func(args, params=params, *kwargs)
File "/home/foodforfamilyf3/.virtualenvs/deploy/local/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 632, in search
doc_type, '_search'), params=params, body=body)
File "/home/foodforfamilyf3/.virtualenvs/deploy/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 312, in perform_reques
t
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/home/foodforfamilyf3/.virtualenvs/deploy/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 124, in
perform_request
raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(: Failed to e
stablish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError( at 0x7f613c880b90>: Failed to establish a new connection: [Errno 111] Connection refused)

ngocAnh1999 on 14 May 2019

@ngocAnh1999 Have a look at this question on Stackoverflow and see if you can replicate the setup in the question. Then see this answer below on how to make sure elasticsearch is up and running before trying to connect to it. Hope that works for you.

mihaitodor on 16 May 2019

👍3

Thanks @mihaitodor , so why i run it in localhost , my program can connect with host elasticsearch , but at host online it can't ?

ngocAnh1999 on 16 May 2019

Sorry, I don't think I can help with that. You'll probably have to talk with the person who set up the online version and ask how to connect to it.

mihaitodor on 16 May 2019

Add sleep and try . You need to wait until the daemon starts..
I faced a similar problem as connection refused while running docker commands ..
Once i added 'sleep 15' after docker run the problem got resolved