Elasticsearch-py: urllib socket.timeout, no indication as to why, timeout=0.1

[geudrik]

I'm seeing the infrequent exception being raised with a strange timeout value, related to connection pooling. The exception below is the entire exception - zero reference to the code that I'm writing. The only redacted info is the host IP, nothing else has been altered.

GET http://ip.ip.ip.ip:9200/_nodes/_all/http [status:N/A request:0.101s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/http_urllib3.py", line 149, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=request_headers, **kw)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 333, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 389, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='ip.ip.ip.ip', port=9200): Read timed out. (read timeout=0.1)

My ES instance is being created as follows

        self.es = elasticsearch.Elasticsearch(eshost,
                                              sniff_on_start=True,
                                              sniff_on_connection_fail=True,
                                              sniffer_timeout=60)

When I'm inserting data, I'm using

        try:
            bulk(self.es, self.to_ingest, chunk_size=1000, request_timeout=15)
        except ConnectionTimeout as e:
            try:
                bulk(self.es, self.to_ingest, chunk_size=500, request_timeout=40)

Am I missing something painfully obvious here? It has been "one of those weekends" 😛

[geudrik]

This exception isn't thrown until my worker has been running for several hours, I should add. Per the doc suggestion, I'm creating a single instance within my worker and re-using it.

[geudrik]

I've done more testing, and this issue actually appears to be (still) random, stemming from my use of sniffing.

The setup that I'm using is a single master/ingest combo, and two data nodes. The master/ingest is a container, and the two data nodes are physical boxes.

Removing sniffing, these exceptions disappear. I suppose we can consider this solved... but... 🤒

[fxdgear]

I'm trying to recreate and have trouble doing so. This is weird behavior.

I've been able to verify that sniffing works:

1. on connection
2. on error

I started 3 containers clustered together.
Created a 4th container with the client library and instantiated a connection using just one host.
Then I verified I could connect and see the _cat/nodes. then I periodically would kill the one that the python client was connecting to and tried to connect to it again. I would receive a connection error and a notice that the node was on a 60s timeout (had failed 1 times in a row)

But when I tried the request again I was able to successfully hit the cluster agin using a different node this time and seeing that there were now 2 nodes in the cluster.

have you been able to verify that there's been no cluster issues during your ingestion process? (maybe your master/ingest crashes/restarts iteself).

[vovavovavovavova]

another possible issue and strange behaivour with socket timeout and python-exceptions.

es.index(something, request_timeout = 0.1)

chain of exceptions if i get timeout:

socket.timeout -->

urllib3.exceptions.ConnecdtionTimeoutError/urllib3.exceptions.ReadTimeOutError -->

elasticsearch.exceptions.ConnectionTimeout.

If i put this in try/except pass scope, i get errors with chain: "#socket.timeout-->#urllib.exceptions.*" . Further execution after except block is stopped.
Finally i have timeout error even if i put all in try/except block.
Double try/except block have the same result as single try/except.
I tried: empty except block, except Exception, except elasticsearch.exceptions.ConnectionTimeout. with the same result.

So where am i wrong or timeout can't be excepted in try? I haven't troubles with catching other es exceptions.

[daveisfera]

I'm having the same issue and would love to find a way to be able to catch this exception and make sure it's handled correctly

[magorbalassy]

I have the same issue as well and while in some cluster i never get the error, the majority of the es clusters produce the symptoms.