Elastalert: [Errno 111] Connection refused in executing elastalert-create-index

[ShaneC-C-Y]

Hi everyone,

I install ElasticSearch, Logstash and Kibana on AWS EC2 node. And now I want to put Elastalert on the same node. I follow the setup in this article: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04

Here is the error message when I running elastalert-create-index

ubuntu@ip-172-31-10-92:~/elastalert$ elastalert-create-index 
/usr/local/lib/python2.7/dist-packages/elasticsearch-2.3.0-py2.7.egg/elasticsearch/connection/http_requests.py:59: UserWarning: Connecting to https://ec2-52-42-32-212.us-west-2.compute.amazonaws.com:9200 using SSL with verify_certs=False is insecure.
New index name? (Default elastalert_status) 
Name of existing index to copy? (Default None) 
Traceback (most recent call last):
  File "/usr/local/bin/elastalert-create-index", line 9, in <module>
    load_entry_point('elastalert==0.0.92', 'console_scripts', 'elastalert-create-index')()
  File "/usr/local/lib/python2.7/dist-packages/elastalert-0.0.92-py2.7.egg/elastalert/create_index.py", line 107, in main
    if es_index.exists(index):
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/indices.py", line 226, in exists
  File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 329, in perform_request
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_requests.py", line 79, in perform_request
elasticsearch.exceptions.ConnectionError: ConnectionError(HTTPSConnectionPool(host='ec2-52-42-32-212.us-west-2.compute.amazonaws.com', port=9200): Max retries exceeded with url: /elastalert_status (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1349232250>: Failed to establish a new connection: [Errno 111] Connection refused',))) caused by: ConnectionError(HTTPSConnectionPool(host='ec2-52-42-32-212.us-west-2.compute.amazonaws.com', port=9200): Max retries exceeded with url: /elastalert_status (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f1349232250>: Failed to establish a new connection: [Errno 111] Connection refused',)))

The config.yaml file I have is

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: example_rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: "ec2-52-42-32-212.us-west-2.compute.amazonaws.com"

# The elasticsearch port
es_port: 9200

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with SSL to elasticsearch
use_ssl: True

# GET request with body is the default option for Elasticsearch. 
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

es_username: "-----my Nginx username-----"
es_password: "-----my Nginx password------"

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

Here I am not very sure what is the es_username and password. And I know I have setup Ngnix user and there is an auth_basic restricted access there, so I put the username and password here.

It is the file /etc/nginx/sites-available/default for Nginx

server {
#       listen 80 default_server;
#       listen [::]:80 default_server ipv6only=on;
        listen 80;

#       root /usr/share/nginx/html;
#       index index.html index.htm;

        # Make site accessible from http://localhost/
        server_name ec2-52-42-32-212.us-west-2.compute.amazonaws.com
#       server_name ${hostname};

        auth_basic "Restricted Access";
        auth_basic_user_file /etc/nginx/htpasswd.users;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
#               try_files $uri $uri/ =404;
                # Uncomment to enable naxsi on this location
                # include /etc/nginx/naxsi.rules
                proxy_pass http://localhost:5601;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_set_header Host $host;
                proxy_cache_bypass $http_upgrade;
        }
}

Thanks in advanced!

[Qmando]

It looks like nginx is listening on port 80, not 9200. If the username/password were wrong, I would expect an http 403 response, not connection refused.

[ShaneC-C-Y]

Hi Qmando,
Thanks for your prompt reply and I think you're right!!!

This is what I get now after I change to port from 9200 to 80 in my config.yaml

ubuntu@ip-172-31-10-92:~/elastalert$ elastalert-create-index 
/usr/local/lib/python2.7/dist-packages/elasticsearch-2.3.0-py2.7.egg/elasticsearch/connection/http_requests.py:59: UserWarning: Connecting to https://ec2-52-42-32-212.us-west-2.compute.amazonaws.com:80 using SSL with verify_certs=False is insecure.
New index name? (Default elastalert_status) 
Name of existing index to copy? (Default None) 
/usr/local/lib/python2.7/dist-packages/urllib3-1.16-py2.7.egg/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/urllib3-1.16-py2.7.egg/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/urllib3-1.16-py2.7.egg/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/urllib3-1.16-py2.7.egg/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/urllib3-1.16-py2.7.egg/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Traceback (most recent call last):
  File "/usr/local/bin/elastalert-create-index", line 9, in <module>
    load_entry_point('elastalert==0.0.92', 'console_scripts', 'elastalert-create-index')()
  File "/usr/local/lib/python2.7/dist-packages/elastalert-0.0.92-py2.7.egg/elastalert/create_index.py", line 107, in main
    if es_index.exists(index):
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/indices.py", line 226, in exists
  File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 329, in perform_request
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_requests.py", line 73, in perform_request
elasticsearch.exceptions.SSLError: ConnectionError([Errno 1] _ssl.c:510: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol) caused by: SSLError([Errno 1] _ssl.c:510: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol)

[Qmando]

It looks like you are not using SSL in NGINX. Remove use_ssl: True

[ShaneC-C-Y]

Thank you a lot!!
But now there is another common issue, I found lots of people have this problem.

ubuntu@ip-172-31-10-92:~/elastalert$ elastalert-create-index 
New index name? (Default elastalert_status) 
Name of existing index to copy? (Default None) 
Traceback (most recent call last):
  File "/usr/local/bin/elastalert-create-index", line 9, in <module>
    load_entry_point('elastalert==0.0.92', 'console_scripts', 'elastalert-create-index')()
  File "/usr/local/lib/python2.7/dist-packages/elastalert-0.0.92-py2.7.egg/elastalert/create_index.py", line 107, in main
    if es_index.exists(index):
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 69, in _wrapped
  File "build/bdist.linux-x86_64/egg/elasticsearch/client/indices.py", line 226, in exists
  File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 329, in perform_request
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_requests.py", line 84, in perform_request
  File "build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py", line 108, in _raise_error
elasticsearch.exceptions.RequestError: TransportError(400, u'')

I have already started elasticsearch, and it has logs there.

[Qmando]

Are you sure that Elasticsearch is running correctly?
does curl --user username:password localhost:80/_cat/ output

=^.^=

?

[ShaneC-C-Y]

I can see the Kibana printing out the latest log. The log is grabbed from client node by Logstash and store in elasticsearch.

This is the result when I ran curl

ubuntu@ip-172-31-10-92:~$ curl localhost:9200
{
  "name" : "Helio",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.3.3",
    "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
    "build_timestamp" : "2016-05-17T15:40:04Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

and

ubuntu@ip-172-31-10-92:~$ curl localhost:80
<script>var hashRoute = '/app/kibana';
var defaultRoute = '/app/kibana';

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>

ubuntu@ip-172-31-10-92:~$ curl localhost:9200/_cat/
=^.^=
/_cat/allocation
/_cat/shards
/_cat/shards/{index}
/_cat/master
/_cat/nodes
/_cat/indices
/_cat/indices/{index}
/_cat/segments
/_cat/segments/{index}
/_cat/count
/_cat/count/{index}
/_cat/recovery
/_cat/recovery/{index}
/_cat/health
/_cat/pending_tasks
/_cat/aliases
/_cat/aliases/{alias}
/_cat/thread_pool
/_cat/plugins
/_cat/fielddata
/_cat/fielddata/{fields}
/_cat/nodeattrs
/_cat/repositories
/_cat/snapshots/{repository}

So I think my elasticsearch is still on port 9200?

[Qmando]

Nginx is not pointing to Elasticsearch then, it's pointing to Kibana. If curl works without --user, remove es_username and es_password, and change es_port to 9200, and change es_host to localhost.

[ShaneC-C-Y]

Really appreciate your help, I got it working!!!!