Datadog-agent: Disabling default auto-discovery?

Created on 15 Nov 2018  路  9Comments  路  Source: DataDog/datadog-agent

I've noticed that a subset of the built-in DataDog plugins will auto-discover by default:

 $  find /etc/datadog-agent -name auto_conf.yaml
/etc/datadog-agent/conf.d/kubernetes_state.d/auto_conf.yaml
/etc/datadog-agent/conf.d/consul.d/auto_conf.yaml
/etc/datadog-agent/conf.d/couchbase.d/auto_conf.yaml
/etc/datadog-agent/conf.d/mcache.d/auto_conf.yaml
/etc/datadog-agent/conf.d/coredns.d/auto_conf.yaml
/etc/datadog-agent/conf.d/couch.d/auto_conf.yaml
/etc/datadog-agent/conf.d/redisdb.d/auto_conf.yaml
/etc/datadog-agent/conf.d/kyototycoon.d/auto_conf.yaml
/etc/datadog-agent/conf.d/riak.d/auto_conf.yaml
/etc/datadog-agent/conf.d/etcd.d/auto_conf.yaml
/etc/datadog-agent/conf.d/kube_dns.d/auto_conf.yaml
/etc/datadog-agent/conf.d/elastic.d/auto_conf.yaml
/etc/datadog-agent/conf.d/tomcat.d/auto_conf.yaml

I'm curious if there is some way to disable these individually? One way would be just to delete the relevant auto_conf.yaml of course, but when using the datadog docker image, this simple act requires rebuilding the image - this seems a bit heavy for such a simple change. I'm wondering if it's possible to do it with an Envar or parameter or something?

For what it's worth, the reason I ask this is because we run _a lot_ of redis containers across our test environments, and all of these are being unnecessarily monitored having a significant impact on memory usage by the agent.

I'd prefer to have auto discovery disabled by default, and enable it adhoc by adding the relevant annotations to the kubernetes deployments we have for redis. This is exactly what we do for Apache, Nginx and MySQL, all of which do not default to auto-discovery.

componenautodiscovery kinquestion
Source
picture mf-lit
👍1

Most helpful comment

@Simwar
unfortunately we are seeing 2 checks when I run the agent status, one from the Elasticsearch nodes directly (they are working fine) and the other one created by autodiscovery (we want to disable this one)

    elastic (1.16.0)
    ----------------
      Instance ID: elastic:1c6287d5484862c1 [ERROR]
      Configuration Source: file:/etc/datadog-agent/conf.d/elastic.d/auto_conf.yaml
      Total Runs: 61,577
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 61,577
      Average Execution Time : 11ms
      Last Execution Date : 2020-05-05 17:04:29.000000 UTC
      Last Successful Execution Date : Never
      Error: 401 Client Error: Unauthorized for url: http://10.22.134.247:9200/_nodes/_local/stats?all=true
      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 713, in run
          self.check(instance)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 71, in check
          stats_data = self._get_data(stats_url, config)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 206, in _get_data
          resp.raise_for_status()
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
          raise HTTPError(http_error_msg, response=self)
      requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://10.22.134.247:9200/_nodes/_local/stats?all=true
      Warning: Error while trying to get Elasticsearch version from http://10.22.134.247:9200 401 Client Error: Unauthorized for url: http://10.22.134.247:9200/

      Instance ID: elastic:57efe4b14a0e59aa [ERROR]
      Configuration Source: file:/etc/datadog-agent/conf.d/elastic.d/auto_conf.yaml
      Total Runs: 61,577
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 61,577
      Average Execution Time : 13ms
      Last Execution Date : 2020-05-05 17:04:21.000000 UTC
      Last Successful Execution Date : Never
      Error: 401 Client Error: Unauthorized for url: http://10.22.145.152:9200/_nodes/_local/stats?all=true
      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 713, in run
          self.check(instance)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 71, in check
          stats_data = self._get_data(stats_url, config)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 206, in _get_data
          resp.raise_for_status()
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
          raise HTTPError(http_error_msg, response=self)
      requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://10.22.145.152:9200/_nodes/_local/stats?all=true
      Warning: Error while trying to get Elasticsearch version from http://10.22.145.152:9200 401 Client Error: Unauthorized for url: http://10.22.145.152:9200/

      Instance ID: elastic:cfa3e954f5b14ea7 [ERROR]
      Configuration Source: file:/etc/datadog-agent/conf.d/elastic.d/auto_conf.yaml
      Total Runs: 61,577
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 61,577
      Average Execution Time : 5ms
      Last Execution Date : 2020-05-05 17:04:22.000000 UTC
      Last Successful Execution Date : Never
      Error: 401 Client Error: Unauthorized for url: http://10.22.139.185:9200/_nodes/_local/stats?all=true
      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 713, in run
          self.check(instance)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 71, in check
          stats_data = self._get_data(stats_url, config)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 206, in _get_data
          resp.raise_for_status()
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
          raise HTTPError(http_error_msg, response=self)
      requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://10.22.139.185:9200/_nodes/_local/stats?all=true
      Warning: Error while trying to get Elasticsearch version from http://10.22.139.185:9200 401 Client Error: Unauthorized for url: http://10.22.139.185:9200/

      Instance ID: elastic:e657074115021e1f [OK]
      Configuration Source: kubelet:docker://975bda6e66d096960d0d0b61e2118bbc5e891cecd403d77735d79aa5968187a5
      Total Runs: 61,578
      Metric Samples: Last Run: 246, Total: 1,000,236
      Events: Last Run: 0, Total: 6
      Service Checks: Last Run: 2, Total: 123,156
      Average Execution Time : 229ms
      Last Execution Date : 2020-05-05 17:04:30.000000 UTC
      Last Successful Execution Date : 2020-05-05 17:04:30.000000 UTC
      metadata:
        version.major: 7
        version.minor: 6
        version.patch: 1
        version.raw: 7.6.1
        version.scheme: semver

      Instance ID: elastic:f9f729ab3a23f808 [OK]
      Configuration Source: kubelet:docker://278572c4201694bdc099cf899b46dd22b0d5549cf408a3039b59d1f0e410558a
      Total Runs: 61,577
      Metric Samples: Last Run: 246, Total: 1,000,236
      Events: Last Run: 0, Total: 4
      Service Checks: Last Run: 2, Total: 123,154
      Average Execution Time : 283ms
      Last Execution Date : 2020-05-05 17:04:29.000000 UTC
      Last Successful Execution Date : 2020-05-05 17:04:29.000000 UTC
      metadata:
        version.major: 7
        version.minor: 6
        version.patch: 1
        version.raw: 7.6.1
        version.scheme: semver

      Instance ID: elastic:fca02be9167b1e16 [OK]
      Configuration Source: kubelet:docker://855a03ea65aafbb480791d961e6c46f743bb5326cd50d0f576edba51e687b01a
      Total Runs: 61,577
      Metric Samples: Last Run: 246, Total: 1,000,236
      Events: Last Run: 0, Total: 4
      Service Checks: Last Run: 2, Total: 123,154
      Average Execution Time : 86ms
      Last Execution Date : 2020-05-05 17:04:21.000000 UTC
      Last Successful Execution Date : 2020-05-05 17:04:21.000000 UTC
      metadata:
        version.major: 7
        version.minor: 6
        version.patch: 1
        version.raw: 7.6.1
        version.scheme: semver
picture balusarakesh on 5 May 2020
👍5

All 9 comments

Hey @mf-lit thanks for reaching out,

The autodiscovery component is a quite critical piece of the agent and is even used to load "static" file configuration and thus cannot be disabled as of now.

However, there's a few tricks you can use to avoid your test redis containers from being picked up by the AD:

  • Using another image name than redis
  • Adding empty AD annotations/labels that'll override default AD file template (see https://docs.datadoghq.com/agent/autodiscovery/?tab=kubernetes#setting-up-check-templates)
mfpierre picture mfpierre on 15 Nov 2018

Thanks for getting back to me @mfpierre .

The empty annotations trick you suggested works perfectly. Thank you.

I must admit I think it's a mistake to have only some plugins auto-discover by default, but mostly pleased to have solved the immediate issue. I'll close this issue now.

mf-lit picture mf-lit on 21 Nov 2018

@mfpierre
We would like to disable elastic autodiscovery checks ONLY since we setup elastic checks on our elasticsearch pods through annotations. Can you please guide me how I can do that?

balusarakesh picture balusarakesh on 4 May 2020
👍1

Hey @balusarakesh,
The annotations should take precedence over the auto_conf.yaml files, creating only one check.
This should work out of the box.

Let us know if this is not the case!

Thanks

Simwar picture Simwar on 5 May 2020

@Simwar
unfortunately we are seeing 2 checks when I run the agent status, one from the Elasticsearch nodes directly (they are working fine) and the other one created by autodiscovery (we want to disable this one)

    elastic (1.16.0)
    ----------------
      Instance ID: elastic:1c6287d5484862c1 [ERROR]
      Configuration Source: file:/etc/datadog-agent/conf.d/elastic.d/auto_conf.yaml
      Total Runs: 61,577
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 61,577
      Average Execution Time : 11ms
      Last Execution Date : 2020-05-05 17:04:29.000000 UTC
      Last Successful Execution Date : Never
      Error: 401 Client Error: Unauthorized for url: http://10.22.134.247:9200/_nodes/_local/stats?all=true
      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 713, in run
          self.check(instance)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 71, in check
          stats_data = self._get_data(stats_url, config)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 206, in _get_data
          resp.raise_for_status()
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
          raise HTTPError(http_error_msg, response=self)
      requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://10.22.134.247:9200/_nodes/_local/stats?all=true
      Warning: Error while trying to get Elasticsearch version from http://10.22.134.247:9200 401 Client Error: Unauthorized for url: http://10.22.134.247:9200/

      Instance ID: elastic:57efe4b14a0e59aa [ERROR]
      Configuration Source: file:/etc/datadog-agent/conf.d/elastic.d/auto_conf.yaml
      Total Runs: 61,577
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 61,577
      Average Execution Time : 13ms
      Last Execution Date : 2020-05-05 17:04:21.000000 UTC
      Last Successful Execution Date : Never
      Error: 401 Client Error: Unauthorized for url: http://10.22.145.152:9200/_nodes/_local/stats?all=true
      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 713, in run
          self.check(instance)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 71, in check
          stats_data = self._get_data(stats_url, config)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 206, in _get_data
          resp.raise_for_status()
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
          raise HTTPError(http_error_msg, response=self)
      requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://10.22.145.152:9200/_nodes/_local/stats?all=true
      Warning: Error while trying to get Elasticsearch version from http://10.22.145.152:9200 401 Client Error: Unauthorized for url: http://10.22.145.152:9200/

      Instance ID: elastic:cfa3e954f5b14ea7 [ERROR]
      Configuration Source: file:/etc/datadog-agent/conf.d/elastic.d/auto_conf.yaml
      Total Runs: 61,577
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 61,577
      Average Execution Time : 5ms
      Last Execution Date : 2020-05-05 17:04:22.000000 UTC
      Last Successful Execution Date : Never
      Error: 401 Client Error: Unauthorized for url: http://10.22.139.185:9200/_nodes/_local/stats?all=true
      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 713, in run
          self.check(instance)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 71, in check
          stats_data = self._get_data(stats_url, config)
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 206, in _get_data
          resp.raise_for_status()
        File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
          raise HTTPError(http_error_msg, response=self)
      requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://10.22.139.185:9200/_nodes/_local/stats?all=true
      Warning: Error while trying to get Elasticsearch version from http://10.22.139.185:9200 401 Client Error: Unauthorized for url: http://10.22.139.185:9200/

      Instance ID: elastic:e657074115021e1f [OK]
      Configuration Source: kubelet:docker://975bda6e66d096960d0d0b61e2118bbc5e891cecd403d77735d79aa5968187a5
      Total Runs: 61,578
      Metric Samples: Last Run: 246, Total: 1,000,236
      Events: Last Run: 0, Total: 6
      Service Checks: Last Run: 2, Total: 123,156
      Average Execution Time : 229ms
      Last Execution Date : 2020-05-05 17:04:30.000000 UTC
      Last Successful Execution Date : 2020-05-05 17:04:30.000000 UTC
      metadata:
        version.major: 7
        version.minor: 6
        version.patch: 1
        version.raw: 7.6.1
        version.scheme: semver

      Instance ID: elastic:f9f729ab3a23f808 [OK]
      Configuration Source: kubelet:docker://278572c4201694bdc099cf899b46dd22b0d5549cf408a3039b59d1f0e410558a
      Total Runs: 61,577
      Metric Samples: Last Run: 246, Total: 1,000,236
      Events: Last Run: 0, Total: 4
      Service Checks: Last Run: 2, Total: 123,154
      Average Execution Time : 283ms
      Last Execution Date : 2020-05-05 17:04:29.000000 UTC
      Last Successful Execution Date : 2020-05-05 17:04:29.000000 UTC
      metadata:
        version.major: 7
        version.minor: 6
        version.patch: 1
        version.raw: 7.6.1
        version.scheme: semver

      Instance ID: elastic:fca02be9167b1e16 [OK]
      Configuration Source: kubelet:docker://855a03ea65aafbb480791d961e6c46f743bb5326cd50d0f576edba51e687b01a
      Total Runs: 61,577
      Metric Samples: Last Run: 246, Total: 1,000,236
      Events: Last Run: 0, Total: 4
      Service Checks: Last Run: 2, Total: 123,154
      Average Execution Time : 86ms
      Last Execution Date : 2020-05-05 17:04:21.000000 UTC
      Last Successful Execution Date : 2020-05-05 17:04:21.000000 UTC
      metadata:
        version.major: 7
        version.minor: 6
        version.patch: 1
        version.raw: 7.6.1
        version.scheme: semver
balusarakesh picture balusarakesh on 5 May 2020
👍5

similar situation over here but for kube_apiserver_metrics a separate config file was created at /etc/datadog-agent/conf.d/kube_apiserver_metrics.yaml but I still see both the autoconf check running and my custom check running

madsonic picture madsonic on 8 May 2020
👍3

similar situation over here but for kube_apiserver_metrics a separate config file was created at /etc/datadog-agent/conf.d/kube_apiserver_metrics.yaml but I still see both the autoconf check running and my custom check running

I have the same problem
My solution was that I mounted custom config to that directory. I use KOPS and kops are had default port 443, not 6443. In my opinion, would be great if the helm chart has ability to enable/disable auto discovery for some service or rewrite config from the box because it's a little bit noisy. You have to create your own configmap or secret by hand then modify the helm chart to mount the volume

kolikons picture kolikons on 13 Jul 2020

@Simwar Hit this myself today with both a redis check (activated through a cluster check) also brought along a autodiscovery. In this case it looks like it's preventing the metrics from being published. The link up top doesn't point anywhere useful any more, is there another work around?

casret picture casret on 8 Oct 2020

Same here with agent 7.23.1 annotation based configuration works ok but auto_conf.yaml integration wasn't disabled and still tries to connect to Elastic:
```
elastic (1.20.0)
----------------
Instance ID: elastic:739e7264ffc6c4f1 [ERROR]
Configuration Source: file:/etc/datadog-agent/conf.d/elastic.d/auto_conf.yaml
Total Runs: 5
Metric Samples: Last Run: 0, Total: 0
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 1, Total: 5
Average Execution Time : 7ms
Last Execution Date : 2020-12-05 16:59:12.000000 UTC
Last Successful Execution Date : Never
Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/opt/datadog-agent/embedded/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/opt/datadog-agent/embedded/lib/python3.8/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/opt/datadog-agent/embedded/lib/python3.8/http/client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
      resp = conn.urlopen(
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
      retries = retries.increment(
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
      raise six.reraise(type(error), error, _stacktrace)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
      raise value.with_traceback(tb)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
      httplib_response = self._make_request(
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
      six.raise_from(e, None)
    File "<string>", line 3, in raise_from
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
      httplib_response = conn.getresponse()
    File "/opt/datadog-agent/embedded/lib/python3.8/http/client.py", line 1347, in getresponse
      response.begin()
    File "/opt/datadog-agent/embedded/lib/python3.8/http/client.py", line 307, in begin
      version, status, reason = self._read_status()
    File "/opt/datadog-agent/embedded/lib/python3.8/http/client.py", line 276, in _read_status
      raise RemoteDisconnected("Remote end closed connection without"
  urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 828, in run
      self.check(instance)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 75, in check
      stats_data = self._get_data(stats_url, config)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 209, in _get_data
      resp = self.http.get(url)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py", line 283, in get
      return self._request('get', url, options)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py", line 332, in _request
      return getattr(requests, method)(url, **new_options)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/api.py", line 75, in get
      return request('get', url, params=params, **kwargs)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/api.py", line 60, in request
      return session.request(method=method, url=url, **kwargs)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/sessions.py", line 533, in request
      resp = self.send(prep, **send_kwargs)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/sessions.py", line 646, in send
      r = adapter.send(request, **kwargs)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
      raise ConnectionError(err, request=request)
  requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  Warning: Error while trying to get Elasticsearch version from http://10.17.0.100:9200 ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

  Instance ID: elastic:938061df44533c2b [OK]
  Configuration Source: kubelet:docker://a759101f5d57ed37fcd9e8ecc0cd4cd16aee7bacb60d40882a00df2ac2448c35
  Total Runs: 5
  Metric Samples: Last Run: 189, Total: 945
  Events: Last Run: 0, Total: 0
  Service Checks: Last Run: 2, Total: 10
  Average Execution Time : 86ms
  Last Execution Date : 2020-12-05 16:59:21.000000 UTC
  Last Successful Execution Date : 2020-12-05 16:59:21.000000 UTC
  metadata:
    version.major: 7
    version.minor: 9
    version.patch: 3
    version.raw: 7.9.3
    version.scheme: semver

  Instance ID: elastic:d3876b554cb30b31 [ERROR]
  Configuration Source: file:/etc/datadog-agent/conf.d/elastic.d/auto_conf.yaml
  Total Runs: 5
  Metric Samples: Last Run: 0, Total: 0
  Events: Last Run: 0, Total: 0
  Service Checks: Last Run: 1, Total: 5
  Average Execution Time : 5ms
  Last Execution Date : 2020-12-05 16:59:19.000000 UTC
  Last Successful Execution Date : Never
  Error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  Traceback (most recent call last):
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
      httplib_response = self._make_request(
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
      six.raise_from(e, None)
    File "<string>", line 3, in raise_from
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
      httplib_response = conn.getresponse()
    File "/opt/datadog-agent/embedded/lib/python3.8/http/client.py", line 1347, in getresponse
      response.begin()
    File "/opt/datadog-agent/embedded/lib/python3.8/http/client.py", line 307, in begin
      version, status, reason = self._read_status()
    File "/opt/datadog-agent/embedded/lib/python3.8/http/client.py", line 276, in _read_status
      raise RemoteDisconnected("Remote end closed connection without"
  http.client.RemoteDisconnected: Remote end closed connection without response

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
      resp = conn.urlopen(
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
      retries = retries.increment(
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
      raise six.reraise(type(error), error, _stacktrace)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
      raise value.with_traceback(tb)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
      httplib_response = self._make_request(
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
      six.raise_from(e, None)
    File "<string>", line 3, in raise_from
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
      httplib_response = conn.getresponse()
    File "/opt/datadog-agent/embedded/lib/python3.8/http/client.py", line 1347, in getresponse
      response.begin()
    File "/opt/datadog-agent/embedded/lib/python3.8/http/client.py", line 307, in begin
      version, status, reason = self._read_status()
    File "/opt/datadog-agent/embedded/lib/python3.8/http/client.py", line 276, in _read_status
      raise RemoteDisconnected("Remote end closed connection without"
  urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py", line 828, in run
      self.check(instance)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 75, in check
      stats_data = self._get_data(stats_url, config)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/elastic/elastic.py", line 209, in _get_data
      resp = self.http.get(url)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py", line 283, in get
      return self._request('get', url, options)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py", line 332, in _request
      return getattr(requests, method)(url, **new_options)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/api.py", line 75, in get
      return request('get', url, params=params, **kwargs)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/api.py", line 60, in request
      return session.request(method=method, url=url, **kwargs)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/sessions.py", line 533, in request
      resp = self.send(prep, **send_kwargs)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/sessions.py", line 646, in send
      r = adapter.send(request, **kwargs)
    File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
      raise ConnectionError(err, request=request)
  requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
  Warning: Error while trying to get Elasticsearch version from http://10.17.0.100:9200 ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

```

Halama picture Halama on 5 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

cvele picture cvele  路  3Comments
SleepyBrett picture SleepyBrett  路  3Comments
btsuhako picture btsuhako  路  3Comments
agosto-calvinbehling picture agosto-calvinbehling  路  3Comments
joshuabaird picture joshuabaird  路  3Comments