Victoriametrics: vmagent can't send metrics to remote data-source due to packet lost/latency
Describe the bug
vmagent can't send metrics to remote data-source due to packet lost/latency, this lead to consistent vm_persistentqueue_bytes_read_total increasing. Additionally there are not any logs in vmagent about this event.
To Reproduce
- vmagent that writes to remote data-source with ping around 300ms
- vmagent collecting metrics from node-exporter, blackbox-exporter, kube-state-metrics, etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet
Expected behavior
vmagent consistently sending metrics.
Version
vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3
Used command-line flags
default expect -remoteWrite.url -remoteWrite.username -remoteWrite.password
See the following docs for details:
Additional context
gorouting.pprof and cpu.pprof sent via [email protected]
ofen
All 13 comments
@ofen , could you share the contents of http://vmagent:8429/metrics page after vmagent stops sending data to remote storage?
valyala
on 4 Sep 2020
@ofen , could you share the contents of
http://vmagent:8429/metricspage after vmagent stops sending data to remote storage?
job="blackbox_kube-apiserver" (3/3 up)
state=up, endpoint=http://blackbox-exporter:9115/probe?target=https%3A%2F%2F10.0.9.212%3A6443%2Fhealthz, labels={instance="https://10.0.9.212:6443/healthz", job="blackbox_kube-apiserver", node="i01-kbm01", service="kube-apiserver"}, last_scrape=2.748s ago, scrape_duration=0.010s, error=""
state=up, endpoint=http://blackbox-exporter:9115/probe?target=https%3A%2F%2F10.0.9.213%3A6443%2Fhealthz, labels={instance="https://10.0.9.213:6443/healthz", job="blackbox_kube-apiserver", node="i01-kbm02", service="kube-apiserver"}, last_scrape=39.906s ago, scrape_duration=0.010s, error=""
state=up, endpoint=http://blackbox-exporter:9115/probe?target=https%3A%2F%2F10.0.9.214%3A6443%2Fhealthz, labels={instance="https://10.0.9.214:6443/healthz", job="blackbox_kube-apiserver", node="i01-kbm03", service="kube-apiserver"}, last_scrape=135.177s ago, scrape_duration=0.008s, error=""
job="blackbox_kubelet" (7/7 up)
state=up, endpoint=http://blackbox-exporter:9115/probe?module=http_2xx&target=https%3A%2F%2F10.0.10.1%3A10250%2Fhealthz, labels={instance="https://10.0.10.1:10250/healthz", job="blackbox_kubelet", node="c01-kbw01", service="kubelet"}, last_scrape=279.375s ago, scrape_duration=0.006s, error=""
state=up, endpoint=http://blackbox-exporter:9115/probe?module=http_2xx&target=https%3A%2F%2F10.0.10.2%3A10250%2Fhealthz, labels={instance="https://10.0.10.2:10250/healthz", job="blackbox_kubelet", node="c01-kbw02", service="kubelet"}, last_scrape=52.368s ago, scrape_duration=0.010s, error=""
state=up, endpoint=http://blackbox-exporter:9115/probe?module=http_2xx&target=https%3A%2F%2F10.0.10.3%3A10250%2Fhealthz, labels={instance="https://10.0.10.3:10250/healthz", job="blackbox_kubelet", node="c01-kbw03", service="kubelet"}, last_scrape=87.205s ago, scrape_duration=0.009s, error=""
state=up, endpoint=http://blackbox-exporter:9115/probe?module=http_2xx&target=https%3A%2F%2F10.0.10.4%3A10250%2Fhealthz, labels={instance="https://10.0.10.4:10250/healthz", job="blackbox_kubelet", node="c01-kbw04", service="kubelet"}, last_scrape=200.926s ago, scrape_duration=0.007s, error=""
state=up, endpoint=http://blackbox-exporter:9115/probe?module=http_2xx&target=https%3A%2F%2F10.0.9.212%3A10250%2Fhealthz, labels={instance="https://10.0.9.212:10250/healthz", job="blackbox_kubelet", node="i01-kbm01", service="kubelet"}, last_scrape=285.690s ago, scrape_duration=0.027s, error=""
state=up, endpoint=http://blackbox-exporter:9115/probe?module=http_2xx&target=https%3A%2F%2F10.0.9.213%3A10250%2Fhealthz, labels={instance="https://10.0.9.213:10250/healthz", job="blackbox_kubelet", node="i01-kbm02", service="kubelet"}, last_scrape=228.309s ago, scrape_duration=0.007s, error=""
state=up, endpoint=http://blackbox-exporter:9115/probe?module=http_2xx&target=https%3A%2F%2F10.0.9.214%3A10250%2Fhealthz, labels={instance="https://10.0.9.214:10250/healthz", job="blackbox_kubelet", node="i01-kbm03", service="kubelet"}, last_scrape=170.423s ago, scrape_duration=0.009s, error=""
job="cadvisor" (7/7 up)
state=up, endpoint=https://10.0.10.1:10250/metrics/cadvisor, labels={instance="10.0.10.1:10250", job="cadvisor", node="c01-kbw01", service="cadvisor"}, last_scrape=11.142s ago, scrape_duration=0.044s, error=""
state=up, endpoint=https://10.0.10.2:10250/metrics/cadvisor, labels={instance="10.0.10.2:10250", job="cadvisor", node="c01-kbw02", service="cadvisor"}, last_scrape=25.100s ago, scrape_duration=0.037s, error=""
state=up, endpoint=https://10.0.10.3:10250/metrics/cadvisor, labels={instance="10.0.10.3:10250", job="cadvisor", node="c01-kbw03", service="cadvisor"}, last_scrape=26.918s ago, scrape_duration=0.029s, error=""
state=up, endpoint=https://10.0.10.4:10250/metrics/cadvisor, labels={instance="10.0.10.4:10250", job="cadvisor", node="c01-kbw04", service="cadvisor"}, last_scrape=40.241s ago, scrape_duration=0.022s, error=""
state=up, endpoint=https://10.0.9.212:10250/metrics/cadvisor, labels={instance="10.0.9.212:10250", job="cadvisor", node="i01-kbm01", service="cadvisor"}, last_scrape=48.683s ago, scrape_duration=0.034s, error=""
state=up, endpoint=https://10.0.9.213:10250/metrics/cadvisor, labels={instance="10.0.9.213:10250", job="cadvisor", node="i01-kbm02", service="cadvisor"}, last_scrape=5.905s ago, scrape_duration=0.030s, error=""
state=up, endpoint=https://10.0.9.214:10250/metrics/cadvisor, labels={instance="10.0.9.214:10250", job="cadvisor", node="i01-kbm03", service="cadvisor"}, last_scrape=6.939s ago, scrape_duration=0.051s, error=""
job="etcd" (3/3 up)
state=up, endpoint=https://10.0.9.212:2379/metrics, labels={instance="10.0.9.212:2379", job="etcd", node="i01-kbm01", service="etcd"}, last_scrape=18.941s ago, scrape_duration=0.008s, error=""
state=up, endpoint=https://10.0.9.213:2379/metrics, labels={instance="10.0.9.213:2379", job="etcd", node="i01-kbm02", service="etcd"}, last_scrape=2.923s ago, scrape_duration=0.014s, error=""
state=up, endpoint=https://10.0.9.214:2379/metrics, labels={instance="10.0.9.214:2379", job="etcd", node="i01-kbm03", service="etcd"}, last_scrape=46.908s ago, scrape_duration=0.007s, error=""
job="journalbeat" (7/7 up)
state=up, endpoint=http://10.0.224.11:9479/metrics, labels={instance="10.0.224.11:9479", job="journalbeat", namespace="logging", node="i01-kbm01", pod_name="journalbeat-gc7v4", service="journalbeat"}, last_scrape=59.022s ago, scrape_duration=0.004s, error=""
state=up, endpoint=http://10.0.225.12:9479/metrics, labels={instance="10.0.225.12:9479", job="journalbeat", namespace="logging", node="i01-kbm02", pod_name="journalbeat-t92rc", service="journalbeat"}, last_scrape=54.080s ago, scrape_duration=0.003s, error=""
state=up, endpoint=http://10.0.226.10:9479/metrics, labels={instance="10.0.226.10:9479", job="journalbeat", namespace="logging", node="i01-kbm03", pod_name="journalbeat-cx5mk", service="journalbeat"}, last_scrape=56.048s ago, scrape_duration=0.003s, error=""
state=up, endpoint=http://10.0.227.22:9479/metrics, labels={instance="10.0.227.22:9479", job="journalbeat", namespace="logging", node="c01-kbw01", pod_name="journalbeat-k2j9p", service="journalbeat"}, last_scrape=47.488s ago, scrape_duration=0.004s, error=""
state=up, endpoint=http://10.0.228.14:9479/metrics, labels={instance="10.0.228.14:9479", job="journalbeat", namespace="logging", node="c01-kbw02", pod_name="journalbeat-4c85c", service="journalbeat"}, last_scrape=33.571s ago, scrape_duration=0.003s, error=""
state=up, endpoint=http://10.0.229.15:9479/metrics, labels={instance="10.0.229.15:9479", job="journalbeat", namespace="logging", node="c01-kbw04", pod_name="journalbeat-cs4vm", service="journalbeat"}, last_scrape=53.274s ago, scrape_duration=0.003s, error=""
state=up, endpoint=http://10.0.230.19:9479/metrics, labels={instance="10.0.230.19:9479", job="journalbeat", namespace="logging", node="c01-kbw03", pod_name="journalbeat-w5vjd", service="journalbeat"}, last_scrape=33.271s ago, scrape_duration=0.003s, error=""
job="kube-apiserver" (3/3 up)
state=up, endpoint=https://10.0.9.212:6443/metrics, labels={instance="10.0.9.212:6443", job="kube-apiserver", namespace="default", service="kubernetes"}, last_scrape=47.416s ago, scrape_duration=0.072s, error=""
state=up, endpoint=https://10.0.9.213:6443/metrics, labels={instance="10.0.9.213:6443", job="kube-apiserver", namespace="default", service="kubernetes"}, last_scrape=38.587s ago, scrape_duration=0.056s, error=""
state=up, endpoint=https://10.0.9.214:6443/metrics, labels={instance="10.0.9.214:6443", job="kube-apiserver", namespace="default", service="kubernetes"}, last_scrape=22.695s ago, scrape_duration=0.045s, error=""
job="kube-controller-manager" (3/3 up)
state=up, endpoint=http://10.0.9.212:10252/metrics, labels={instance="10.0.9.212:10252", job="kube-controller-manager", namespace="kube-system", node="i01-kbm01", pod_name="kube-controller-manager-i01-kbm01", service="kube-controller-manager"}, last_scrape=0.307s ago, scrape_duration=0.004s, error=""
state=up, endpoint=http://10.0.9.213:10252/metrics, labels={instance="10.0.9.213:10252", job="kube-controller-manager", namespace="kube-system", node="i01-kbm02", pod_name="kube-controller-manager-i01-kbm02", service="kube-controller-manager"}, last_scrape=10.709s ago, scrape_duration=0.003s, error=""
state=up, endpoint=http://10.0.9.214:10252/metrics, labels={instance="10.0.9.214:10252", job="kube-controller-manager", namespace="kube-system", node="i01-kbm03", pod_name="kube-controller-manager-i01-kbm03", service="kube-controller-manager"}, last_scrape=0.699s ago, scrape_duration=0.007s, error=""
job="kube-dns" (3/3 up)
state=up, endpoint=http://10.0.224.5:9153/metrics, labels={instance="10.0.224.5:9153", job="kube-dns", namespace="kube-system", node="i01-kbm01", pod_name="coredns-55c975cd8d-pnzvq", service="kube-dns"}, last_scrape=39.342s ago, scrape_duration=0.005s, error=""
state=up, endpoint=http://10.0.225.3:9153/metrics, labels={instance="10.0.225.3:9153", job="kube-dns", namespace="kube-system", node="i01-kbm02", pod_name="coredns-55c975cd8d-4jg9k", service="kube-dns"}, last_scrape=26.455s ago, scrape_duration=0.004s, error=""
state=up, endpoint=http://10.0.226.3:9153/metrics, labels={instance="10.0.226.3:9153", job="kube-dns", namespace="kube-system", node="i01-kbm03", pod_name="coredns-55c975cd8d-nz8q6", service="kube-dns"}, last_scrape=14.312s ago, scrape_duration=0.002s, error=""
job="kube-router" (7/7 up)
state=up, endpoint=http://10.0.10.1:9101/metrics, labels={instance="10.0.10.1:9101", job="kube-router", namespace="kube-system", node="c01-kbw01", pod_name="kube-router-hkwz9", service="kube-router"}, last_scrape=10.949s ago, scrape_duration=0.003s, error=""
state=up, endpoint=http://10.0.10.2:9101/metrics, labels={instance="10.0.10.2:9101", job="kube-router", namespace="kube-system", node="c01-kbw02", pod_name="kube-router-k2k9t", service="kube-router"}, last_scrape=40.930s ago, scrape_duration=0.003s, error=""
state=up, endpoint=http://10.0.10.3:9101/metrics, labels={instance="10.0.10.3:9101", job="kube-router", namespace="kube-system", node="c01-kbw03", pod_name="kube-router-twmgk", service="kube-router"}, last_scrape=49.872s ago, scrape_duration=0.002s, error=""
state=up, endpoint=http://10.0.10.4:9101/metrics, labels={instance="10.0.10.4:9101", job="kube-router", namespace="kube-system", node="c01-kbw04", pod_name="kube-router-6txz5", service="kube-router"}, last_scrape=32.663s ago, scrape_duration=0.003s, error=""
state=up, endpoint=http://10.0.9.212:9101/metrics, labels={instance="10.0.9.212:9101", job="kube-router", namespace="kube-system", node="i01-kbm01", pod_name="kube-router-2zl9j", service="kube-router"}, last_scrape=0.866s ago, scrape_duration=0.003s, error=""
state=up, endpoint=http://10.0.9.213:9101/metrics, labels={instance="10.0.9.213:9101", job="kube-router", namespace="kube-system", node="i01-kbm02", pod_name="kube-router-qhmxr", service="kube-router"}, last_scrape=49.621s ago, scrape_duration=0.005s, error=""
state=up, endpoint=http://10.0.9.214:9101/metrics, labels={instance="10.0.9.214:9101", job="kube-router", namespace="kube-system", node="i01-kbm03", pod_name="kube-router-z7m79", service="kube-router"}, last_scrape=5.423s ago, scrape_duration=0.002s, error=""
job="kube-scheduler" (3/3 up)
state=up, endpoint=http://10.0.9.212:10251/metrics, labels={instance="10.0.9.212:10251", job="kube-scheduler", namespace="kube-system", node="i01-kbm01", pod_name="kube-scheduler-i01-kbm01", service="kube-scheduler"}, last_scrape=56.854s ago, scrape_duration=0.003s, error=""
state=up, endpoint=http://10.0.9.213:10251/metrics, labels={instance="10.0.9.213:10251", job="kube-scheduler", namespace="kube-system", node="i01-kbm02", pod_name="kube-scheduler-i01-kbm02", service="kube-scheduler"}, last_scrape=48.377s ago, scrape_duration=0.006s, error=""
state=up, endpoint=http://10.0.9.214:10251/metrics, labels={instance="10.0.9.214:10251", job="kube-scheduler", namespace="kube-system", node="i01-kbm03", pod_name="kube-scheduler-i01-kbm03", service="kube-scheduler"}, last_scrape=3.616s ago, scrape_duration=0.002s, error=""
job="kube-state-metrics" (2/2 up)
state=up, endpoint=http://10.0.227.23:8080/metrics, labels={instance="10.0.227.23:8080", job="kube-state-metrics"}, last_scrape=36.194s ago, scrape_duration=0.003s, error=""
state=up, endpoint=http://10.0.228.12:8080/metrics, labels={instance="10.0.228.12:8080", job="kube-state-metrics"}, last_scrape=30.973s ago, scrape_duration=0.003s, error=""
job="kubelet" (7/7 up)
state=up, endpoint=https://10.0.10.1:10250/metrics, labels={instance="10.0.10.1:10250", job="kubelet", node="c01-kbw01", service="kubelet"}, last_scrape=6.876s ago, scrape_duration=0.006s, error=""
state=up, endpoint=https://10.0.10.2:10250/metrics, labels={instance="10.0.10.2:10250", job="kubelet", node="c01-kbw02", service="kubelet"}, last_scrape=53.274s ago, scrape_duration=0.009s, error=""
state=up, endpoint=https://10.0.10.3:10250/metrics, labels={instance="10.0.10.3:10250", job="kubelet", node="c01-kbw03", service="kubelet"}, last_scrape=35.919s ago, scrape_duration=0.005s, error=""
state=up, endpoint=https://10.0.10.4:10250/metrics, labels={instance="10.0.10.4:10250", job="kubelet", node="c01-kbw04", service="kubelet"}, last_scrape=54.922s ago, scrape_duration=0.005s, error=""
state=up, endpoint=https://10.0.9.212:10250/metrics, labels={instance="10.0.9.212:10250", job="kubelet", node="i01-kbm01", service="kubelet"}, last_scrape=5.215s ago, scrape_duration=0.012s, error=""
state=up, endpoint=https://10.0.9.213:10250/metrics, labels={instance="10.0.9.213:10250", job="kubelet", node="i01-kbm02", service="kubelet"}, last_scrape=38.005s ago, scrape_duration=0.006s, error=""
state=up, endpoint=https://10.0.9.214:10250/metrics, labels={instance="10.0.9.214:10250", job="kubelet", node="i01-kbm03", service="kubelet"}, last_scrape=28.134s ago, scrape_duration=0.007s, error=""
job="metricbeat" (1/1 up)
state=up, endpoint=http://10.0.226.9:9479/metrics, labels={instance="10.0.226.9:9479", job="metricbeat", namespace="logging", node="i01-kbm03", pod_name="metricbeat-6f544544d9-9strk", service="metricbeat"}, last_scrape=30.043s ago, scrape_duration=0.005s, error=""
job="node-exporter" (7/7 up)
state=up, endpoint=http://10.0.10.1:30206/metrics, labels={instance="10.0.10.1:30206", job="node-exporter", namespace="monitoring-test", node="c01-kbw01", pod_name="node-exporter-6fnjw", service="node-exporter"}, last_scrape=38.166s ago, scrape_duration=0.662s, error=""
state=up, endpoint=http://10.0.10.2:30206/metrics, labels={instance="10.0.10.2:30206", job="node-exporter", namespace="monitoring-test", node="c01-kbw02", pod_name="node-exporter-xv8nj", service="node-exporter"}, last_scrape=47.952s ago, scrape_duration=0.531s, error=""
state=up, endpoint=http://10.0.10.3:30206/metrics, labels={instance="10.0.10.3:30206", job="node-exporter", namespace="monitoring-test", node="c01-kbw03", pod_name="node-exporter-cqb7z", service="node-exporter"}, last_scrape=5.779s ago, scrape_duration=0.385s, error=""
state=up, endpoint=http://10.0.10.4:30206/metrics, labels={instance="10.0.10.4:30206", job="node-exporter", namespace="monitoring-test", node="c01-kbw04", pod_name="node-exporter-sbv9n", service="node-exporter"}, last_scrape=30.236s ago, scrape_duration=0.467s, error=""
state=up, endpoint=http://10.0.9.212:30206/metrics, labels={instance="10.0.9.212:30206", job="node-exporter", namespace="monitoring-test", node="i01-kbm01", pod_name="node-exporter-b6hr7", service="node-exporter"}, last_scrape=32.319s ago, scrape_duration=0.022s, error=""
state=up, endpoint=http://10.0.9.213:30206/metrics, labels={instance="10.0.9.213:30206", job="node-exporter", namespace="monitoring-test", node="i01-kbm02", pod_name="node-exporter-c9ms4", service="node-exporter"}, last_scrape=56.832s ago, scrape_duration=0.052s, error=""
state=up, endpoint=http://10.0.9.214:30206/metrics, labels={instance="10.0.9.214:30206", job="node-exporter", namespace="monitoring-test", node="i01-kbm03", pod_name="node-exporter-24nw5", service="node-exporter"}, last_scrape=51.161s ago, scrape_duration=0.046s, error=""
job="probes" (7/7 up)
state=up, endpoint=https://10.0.10.1:10250/metrics/probes, labels={instance="10.0.10.1:10250", job="probes", node="c01-kbw01", service="probes"}, last_scrape=10.811s ago, scrape_duration=0.001s, error=""
state=up, endpoint=https://10.0.10.2:10250/metrics/probes, labels={instance="10.0.10.2:10250", job="probes", node="c01-kbw02", service="probes"}, last_scrape=29.287s ago, scrape_duration=0.001s, error=""
state=up, endpoint=https://10.0.10.3:10250/metrics/probes, labels={instance="10.0.10.3:10250", job="probes", node="c01-kbw03", service="probes"}, last_scrape=33.421s ago, scrape_duration=0.005s, error=""
state=up, endpoint=https://10.0.10.4:10250/metrics/probes, labels={instance="10.0.10.4:10250", job="probes", node="c01-kbw04", service="probes"}, last_scrape=39.501s ago, scrape_duration=0.003s, error=""
state=up, endpoint=https://10.0.9.212:10250/metrics/probes, labels={instance="10.0.9.212:10250", job="probes", node="i01-kbm01", service="probes"}, last_scrape=52.152s ago, scrape_duration=0.001s, error=""
state=up, endpoint=https://10.0.9.213:10250/metrics/probes, labels={instance="10.0.9.213:10250", job="probes", node="i01-kbm02", service="probes"}, last_scrape=12.698s ago, scrape_duration=0.001s, error=""
state=up, endpoint=https://10.0.9.214:10250/metrics/probes, labels={instance="10.0.9.214:10250", job="probes", node="i01-kbm03", service="probes"}, last_scrape=22.761s ago, scrape_duration=0.001s, error=""
job="vmagent" (1/1 up)
state=up, endpoint=http://localhost:8429/metrics, labels={instance="localhost:8429", job="vmagent"}, last_scrape=43.400s ago, scrape_duration=0.002s, error=""
Thanks for sharing /targets page! It looks like all the scrape targets are being successfully scraped. Could you also share /metrics page?
valyala
on 4 Sep 2020
Thanks for sharing
/targetspage! It looks like all the scrape targets are being successfully scraped. Could you also share/metricspage?
vm_concurrent_insert_capacity 160
vm_concurrent_insert_current 0
vm_concurrent_insert_limit_reached_total 0
vm_concurrent_insert_limit_timeout_total 0
vm_filestream_buffered_read_bytes_total 91023891
vm_filestream_buffered_read_calls_total 2423
vm_filestream_buffered_write_calls_total 7549
vm_filestream_buffered_written_bytes_total 321197305
vm_filestream_readers 1
vm_filestream_real_read_bytes_total 91223699
vm_filestream_real_read_calls_total 289
vm_filestream_real_write_calls_total 1018
vm_filestream_real_written_bytes_total 320954760
vm_filestream_writers 1
vm_fs_read_bytes_total 0
vm_fs_read_calls_total 0
vm_fs_readers 0
vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="6.0e-4...6.5e-4"} 2
vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="6.5e-4...7.0e-4"} 7
vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="7.0e-4...7.5e-4"} 30
vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="7.5e-4...8.0e-4"} 30
vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="8.0e-4...8.5e-4"} 17
vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="8.5e-4...9.0e-4"} 9
vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="9.0e-4...9.5e-4"} 3
vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="9.5e-4...1.0e-3"} 1
vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="1.0e-3...1.5e-3"} 14
vm_http_request_duration_seconds_bucket{path="/metrics",vmrange="1.5e-3...2.0e-3"} 1
vm_http_request_duration_seconds_sum{path="/metrics"} 0.093898563
vm_http_request_duration_seconds_count{path="/metrics"} 114
vm_http_request_errors_total{path="*", reason="unsupported"} 0
vm_http_requests_all_total 480
vm_http_requests_total{path="/debug/pprof/"} 4
vm_http_requests_total{path="/debug/pprof/cmdline"} 0
vm_http_requests_total{path="/debug/pprof/default"} 2
vm_http_requests_total{path="/debug/pprof/mutex"} 0
vm_http_requests_total{path="/debug/pprof/profile"} 2
vm_http_requests_total{path="/debug/pprof/symbol"} 0
vm_http_requests_total{path="/debug/pprof/trace"} 0
vm_http_requests_total{path="/favicon.ico"} 3
vm_http_requests_total{path="/metrics"} 115
vm_ingestserver_request_errors_total{type="graphite", name="write", net="tcp"} 0
vm_ingestserver_request_errors_total{type="graphite", name="write", net="udp"} 0
vm_ingestserver_request_errors_total{type="influx", name="write", net="tcp"} 0
vm_ingestserver_request_errors_total{type="influx", name="write", net="udp"} 0
vm_ingestserver_request_errors_total{type="opentsdb", name="write", net="tcp"} 0
vm_ingestserver_request_errors_total{type="opentsdb", name="write", net="udp"} 0
vm_ingestserver_request_errors_total{type="opentsdbhttp", name="write", net="tcp"} 0
vm_ingestserver_requests_total{type="graphite", name="write", net="tcp"} 0
vm_ingestserver_requests_total{type="graphite", name="write", net="udp"} 0
vm_ingestserver_requests_total{type="influx", name="write", net="tcp"} 0
vm_ingestserver_requests_total{type="influx", name="write", net="udp"} 0
vm_ingestserver_requests_total{type="opentsdb", name="write", net="tcp"} 0
vm_ingestserver_requests_total{type="opentsdb", name="write", net="udp"} 0
vm_ingestserver_requests_total{type="opentsdbhttp", name="write", net="tcp"} 0
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="error", location="VictoriaMetrics/lib/promscrape/scraper.go:269"} 7
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="error", location="VictoriaMetrics/lib/promscrape/scrapework.go:193"} 3
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/app/vmagent/main.go:102"} 1
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/app/vmagent/main.go:80"} 1
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/app/vmagent/remotewrite/client.go:125"} 1
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/lib/cgroup/cpu.go:26"} 1
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/lib/httpserver/httpserver.go:77"} 1
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/lib/httpserver/httpserver.go:78"} 1
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/lib/logger/flag.go:11"} 1
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/lib/logger/flag.go:12"} 1
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/lib/logger/flag.go:20"} 78
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/lib/memory/memory.go:45"} 1
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/lib/persistentqueue/fastqueue.go:46"} 1
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/lib/promscrape/scraper.go:303"} 4
vm_log_messages_total{app_version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3", level="info", location="VictoriaMetrics/lib/promscrape/scraper.go:78"} 1
vm_nfs_dir_remove_failed_attempts_total 0
vm_nfs_pending_dirs_to_remove 0
vm_persistentqueue_blocks_dropped_total{path="vmagent-remotewrite-data/persistent-queue/1A28D788E0FF4747"} 0
vm_persistentqueue_blocks_read_total{path="vmagent-remotewrite-data/persistent-queue/1A28D788E0FF4747"} 1070
vm_persistentqueue_blocks_written_total{path="vmagent-remotewrite-data/persistent-queue/1A28D788E0FF4747"} 3774
vm_persistentqueue_bytes_dropped_total{path="vmagent-remotewrite-data/persistent-queue/1A28D788E0FF4747"} 0
vm_persistentqueue_bytes_pending{path="vmagent-remotewrite-data/persistent-queue/1A28D788E0FF4747"} 230173414
vm_persistentqueue_bytes_read_total{path="vmagent-remotewrite-data/persistent-queue/1A28D788E0FF4747"} 91015331
vm_persistentqueue_bytes_written_total{path="vmagent-remotewrite-data/persistent-queue/1A28D788E0FF4747"} 321167113
vm_promscrape_config_changes_total{type="consul_sd_configs"} 0
vm_promscrape_config_changes_total{type="dns_sd_configs"} 0
vm_promscrape_config_changes_total{type="ec2_sd_configs"} 0
vm_promscrape_config_changes_total{type="file_sd_configs"} 0
vm_promscrape_config_changes_total{type="gce_sd_configs"} 0
vm_promscrape_config_changes_total{type="kubernetes_sd_configs"} 84
vm_promscrape_config_changes_total{type="static_configs"} 1
vm_promscrape_config_reloads_total 0
vm_promscrape_conn_bytes_read_total 183174458
vm_promscrape_conn_bytes_written_total 3178557
vm_promscrape_conn_read_errors_total 4
vm_promscrape_conn_reads_total 45914
vm_promscrape_conn_write_errors_total 0
vm_promscrape_conn_writes_total 5653
vm_promscrape_conns 71
vm_promscrape_dial_errors_total 3
vm_promscrape_dials_total 85
vm_promscrape_push_data_duration_seconds_bucket{vmrange="5.0e-6...5.5e-6"} 1
vm_promscrape_push_data_duration_seconds_bucket{vmrange="5.5e-6...6.0e-6"} 16
vm_promscrape_push_data_duration_seconds_bucket{vmrange="6.0e-6...6.5e-6"} 42
vm_promscrape_push_data_duration_seconds_bucket{vmrange="6.5e-6...7.0e-6"} 44
vm_promscrape_push_data_duration_seconds_bucket{vmrange="7.0e-6...7.5e-6"} 50
vm_promscrape_push_data_duration_seconds_bucket{vmrange="7.5e-6...8.0e-6"} 49
vm_promscrape_push_data_duration_seconds_bucket{vmrange="8.0e-6...8.5e-6"} 30
vm_promscrape_push_data_duration_seconds_bucket{vmrange="8.5e-6...9.0e-6"} 43
vm_promscrape_push_data_duration_seconds_bucket{vmrange="9.0e-6...9.5e-6"} 49
vm_promscrape_push_data_duration_seconds_bucket{vmrange="9.5e-6...1.0e-5"} 34
vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.0e-5...1.5e-5"} 212
vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.5e-5...2.0e-5"} 424
vm_promscrape_push_data_duration_seconds_bucket{vmrange="2.0e-5...2.5e-5"} 283
vm_promscrape_push_data_duration_seconds_bucket{vmrange="2.5e-5...3.0e-5"} 97
vm_promscrape_push_data_duration_seconds_bucket{vmrange="3.0e-5...3.5e-5"} 44
vm_promscrape_push_data_duration_seconds_bucket{vmrange="3.5e-5...4.0e-5"} 27
vm_promscrape_push_data_duration_seconds_bucket{vmrange="4.0e-5...4.5e-5"} 36
vm_promscrape_push_data_duration_seconds_bucket{vmrange="4.5e-5...5.0e-5"} 7
vm_promscrape_push_data_duration_seconds_bucket{vmrange="5.0e-5...5.5e-5"} 13
vm_promscrape_push_data_duration_seconds_bucket{vmrange="5.5e-5...6.0e-5"} 39
vm_promscrape_push_data_duration_seconds_bucket{vmrange="6.0e-5...6.5e-5"} 255
vm_promscrape_push_data_duration_seconds_bucket{vmrange="6.5e-5...7.0e-5"} 124
vm_promscrape_push_data_duration_seconds_bucket{vmrange="7.0e-5...7.5e-5"} 136
vm_promscrape_push_data_duration_seconds_bucket{vmrange="7.5e-5...8.0e-5"} 122
vm_promscrape_push_data_duration_seconds_bucket{vmrange="8.0e-5...8.5e-5"} 99
vm_promscrape_push_data_duration_seconds_bucket{vmrange="8.5e-5...9.0e-5"} 71
vm_promscrape_push_data_duration_seconds_bucket{vmrange="9.0e-5...9.5e-5"} 59
vm_promscrape_push_data_duration_seconds_bucket{vmrange="9.5e-5...1.0e-4"} 40
vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.0e-4...1.5e-4"} 296
vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.5e-4...2.0e-4"} 88
vm_promscrape_push_data_duration_seconds_bucket{vmrange="2.0e-4...2.5e-4"} 15
vm_promscrape_push_data_duration_seconds_bucket{vmrange="3.0e-4...3.5e-4"} 53
vm_promscrape_push_data_duration_seconds_bucket{vmrange="3.5e-4...4.0e-4"} 483
vm_promscrape_push_data_duration_seconds_bucket{vmrange="4.0e-4...4.5e-4"} 386
vm_promscrape_push_data_duration_seconds_bucket{vmrange="4.5e-4...5.0e-4"} 390
vm_promscrape_push_data_duration_seconds_bucket{vmrange="5.0e-4...5.5e-4"} 283
vm_promscrape_push_data_duration_seconds_bucket{vmrange="5.5e-4...6.0e-4"} 152
vm_promscrape_push_data_duration_seconds_bucket{vmrange="6.0e-4...6.5e-4"} 109
vm_promscrape_push_data_duration_seconds_bucket{vmrange="6.5e-4...7.0e-4"} 123
vm_promscrape_push_data_duration_seconds_bucket{vmrange="7.0e-4...7.5e-4"} 115
vm_promscrape_push_data_duration_seconds_bucket{vmrange="7.5e-4...8.0e-4"} 109
vm_promscrape_push_data_duration_seconds_bucket{vmrange="8.0e-4...8.5e-4"} 47
vm_promscrape_push_data_duration_seconds_bucket{vmrange="8.5e-4...9.0e-4"} 26
vm_promscrape_push_data_duration_seconds_bucket{vmrange="9.0e-4...9.5e-4"} 12
vm_promscrape_push_data_duration_seconds_bucket{vmrange="9.5e-4...1.0e-3"} 9
vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.0e-3...1.5e-3"} 18
vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.5e-3...2.0e-3"} 162
vm_promscrape_push_data_duration_seconds_bucket{vmrange="2.0e-3...2.5e-3"} 3
vm_promscrape_push_data_duration_seconds_bucket{vmrange="4.0e-3...4.5e-3"} 1
vm_promscrape_push_data_duration_seconds_bucket{vmrange="4.5e-3...5.0e-3"} 2
vm_promscrape_push_data_duration_seconds_bucket{vmrange="5.0e-3...5.5e-3"} 1
vm_promscrape_push_data_duration_seconds_bucket{vmrange="9.0e-3...9.5e-3"} 2
vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.0e-2...1.5e-2"} 156
vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.5e-2...2.0e-2"} 103
vm_promscrape_push_data_duration_seconds_bucket{vmrange="2.0e-2...2.5e-2"} 5
vm_promscrape_push_data_duration_seconds_bucket{vmrange="4.0e-2...4.5e-2"} 1
vm_promscrape_push_data_duration_seconds_bucket{vmrange="4.5e-2...5.0e-2"} 1
vm_promscrape_push_data_duration_seconds_bucket{vmrange="1.0e-1...1.5e-1"} 1
vm_promscrape_push_data_duration_seconds_sum 5.98430244199998
vm_promscrape_push_data_duration_seconds_count 5598
vm_promscrape_scrape_duration_seconds_bucket{vmrange="0...0"} 3
vm_promscrape_scrape_duration_seconds_bucket{vmrange="9.5e-4...1.0e-3"} 446
vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.5e-3...2.0e-3"} 561
vm_promscrape_scrape_duration_seconds_bucket{vmrange="2.5e-3...3.0e-3"} 1123
vm_promscrape_scrape_duration_seconds_bucket{vmrange="3.5e-3...4.0e-3"} 493
vm_promscrape_scrape_duration_seconds_bucket{vmrange="4.5e-3...5.0e-3"} 244
vm_promscrape_scrape_duration_seconds_bucket{vmrange="5.5e-3...6.0e-3"} 220
vm_promscrape_scrape_duration_seconds_bucket{vmrange="6.5e-3...7.0e-3"} 228
vm_promscrape_scrape_duration_seconds_bucket{vmrange="7.5e-3...8.0e-3"} 190
vm_promscrape_scrape_duration_seconds_bucket{vmrange="8.5e-3...9.0e-3"} 120
vm_promscrape_scrape_duration_seconds_bucket{vmrange="9.5e-3...1.0e-2"} 114
vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.0e-2...1.5e-2"} 249
vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.5e-2...2.0e-2"} 156
vm_promscrape_scrape_duration_seconds_bucket{vmrange="2.0e-2...2.5e-2"} 234
vm_promscrape_scrape_duration_seconds_bucket{vmrange="2.5e-2...3.0e-2"} 118
vm_promscrape_scrape_duration_seconds_bucket{vmrange="3.0e-2...3.5e-2"} 80
vm_promscrape_scrape_duration_seconds_bucket{vmrange="3.5e-2...4.0e-2"} 77
vm_promscrape_scrape_duration_seconds_bucket{vmrange="4.0e-2...4.5e-2"} 126
vm_promscrape_scrape_duration_seconds_bucket{vmrange="4.5e-2...5.0e-2"} 163
vm_promscrape_scrape_duration_seconds_bucket{vmrange="5.0e-2...5.5e-2"} 109
vm_promscrape_scrape_duration_seconds_bucket{vmrange="5.5e-2...6.0e-2"} 59
vm_promscrape_scrape_duration_seconds_bucket{vmrange="6.0e-2...6.5e-2"} 35
vm_promscrape_scrape_duration_seconds_bucket{vmrange="6.5e-2...7.0e-2"} 42
vm_promscrape_scrape_duration_seconds_bucket{vmrange="7.0e-2...7.5e-2"} 26
vm_promscrape_scrape_duration_seconds_bucket{vmrange="7.5e-2...8.0e-2"} 11
vm_promscrape_scrape_duration_seconds_bucket{vmrange="8.0e-2...8.5e-2"} 4
vm_promscrape_scrape_duration_seconds_bucket{vmrange="8.5e-2...9.0e-2"} 3
vm_promscrape_scrape_duration_seconds_bucket{vmrange="9.0e-2...9.5e-2"} 2
vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.0e-1...1.5e-1"} 4
vm_promscrape_scrape_duration_seconds_bucket{vmrange="2.5e-1...3.0e-1"} 1
vm_promscrape_scrape_duration_seconds_bucket{vmrange="3.0e-1...3.5e-1"} 9
vm_promscrape_scrape_duration_seconds_bucket{vmrange="3.5e-1...4.0e-1"} 46
vm_promscrape_scrape_duration_seconds_bucket{vmrange="4.0e-1...4.5e-1"} 61
vm_promscrape_scrape_duration_seconds_bucket{vmrange="4.5e-1...5.0e-1"} 92
vm_promscrape_scrape_duration_seconds_bucket{vmrange="5.0e-1...5.5e-1"} 16
vm_promscrape_scrape_duration_seconds_bucket{vmrange="5.5e-1...6.0e-1"} 52
vm_promscrape_scrape_duration_seconds_bucket{vmrange="6.0e-1...6.5e-1"} 3
vm_promscrape_scrape_duration_seconds_bucket{vmrange="6.5e-1...7.0e-1"} 35
vm_promscrape_scrape_duration_seconds_bucket{vmrange="7.0e-1...7.5e-1"} 1
vm_promscrape_scrape_duration_seconds_bucket{vmrange="7.5e-1...8.0e-1"} 31
vm_promscrape_scrape_duration_seconds_bucket{vmrange="8.5e-1...9.0e-1"} 7
vm_promscrape_scrape_duration_seconds_bucket{vmrange="9.0e-1...9.5e-1"} 2
vm_promscrape_scrape_duration_seconds_bucket{vmrange="9.5e-1...1.0e0"} 1
vm_promscrape_scrape_duration_seconds_bucket{vmrange="1.0e0...1.5e0"} 1
vm_promscrape_scrape_duration_seconds_sum 254.57499999999328
vm_promscrape_scrape_duration_seconds_count 5598
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="0...0"} 3
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="1.0e3...1.5e3"} 89
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="1.5e3...2.0e3"} 88
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="2.0e3...2.5e3"} 272
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="2.5e3...3.0e3"} 261
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="3.0e3...3.5e3"} 88
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="4.5e3...5.0e3"} 623
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="6.5e3...7.0e3"} 88
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="2.0e4...2.5e4"} 1067
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="2.5e4...3.0e4"} 1
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="3.0e4...3.5e4"} 182
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="3.5e4...4.0e4"} 84
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="9.5e4...1.0e5"} 267
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="1.0e5...1.5e5"} 711
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="1.5e5...2.0e5"} 713
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="5.5e5...6.0e5"} 88
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="6.0e5...6.5e5"} 1
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="6.5e5...7.0e5"} 266
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="7.0e5...7.5e5"} 173
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="7.5e5...8.0e5"} 178
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="8.0e5...8.5e5"} 89
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="2.0e6...2.5e6"} 177
vm_promscrape_scrape_response_size_bytes_bucket{vmrange="2.5e6...3.0e6"} 89
vm_promscrape_scrape_response_size_bytes_sum 1485468288
vm_promscrape_scrape_response_size_bytes_count 5598
vm_promscrape_scraped_samples_bucket{vmrange="0...0"} 3
vm_promscrape_scraped_samples_bucket{vmrange="5.5e0...6.0e0"} 89
vm_promscrape_scraped_samples_bucket{vmrange="7.5e0...8.0e0"} 88
vm_promscrape_scraped_samples_bucket{vmrange="9.5e0...1.0e1"} 94
vm_promscrape_scraped_samples_bucket{vmrange="1.0e1...1.5e1"} 350
vm_promscrape_scraped_samples_bucket{vmrange="1.5e1...2.0e1"} 177
vm_promscrape_scraped_samples_bucket{vmrange="4.0e1...4.5e1"} 623
vm_promscrape_scraped_samples_bucket{vmrange="6.0e1...6.5e1"} 88
vm_promscrape_scraped_samples_bucket{vmrange="1.5e2...2.0e2"} 623
vm_promscrape_scraped_samples_bucket{vmrange="2.0e2...2.5e2"} 444
vm_promscrape_scraped_samples_bucket{vmrange="3.0e2...3.5e2"} 178
vm_promscrape_scraped_samples_bucket{vmrange="4.0e2...4.5e2"} 1
vm_promscrape_scraped_samples_bucket{vmrange="5.0e2...5.5e2"} 5
vm_promscrape_scraped_samples_bucket{vmrange="5.5e2...6.0e2"} 83
vm_promscrape_scraped_samples_bucket{vmrange="1.0e3...1.5e3"} 1334
vm_promscrape_scraped_samples_bucket{vmrange="1.5e3...2.0e3"} 265
vm_promscrape_scraped_samples_bucket{vmrange="2.0e3...2.5e3"} 713
vm_promscrape_scraped_samples_bucket{vmrange="4.5e3...5.0e3"} 174
vm_promscrape_scraped_samples_bucket{vmrange="1.0e4...1.5e4"} 177
vm_promscrape_scraped_samples_bucket{vmrange="1.5e4...2.0e4"} 89
vm_promscrape_scraped_samples_sum 8824561
vm_promscrape_scraped_samples_count 5598
vm_promscrape_scrapes_failed_total 3
vm_promscrape_scrapes_gunzip_failed_total 0
vm_promscrape_scrapes_gunziped_total 5421
vm_promscrape_scrapes_skipped_by_sample_limit_total 0
vm_promscrape_scrapes_timed_out_total 0
vm_promscrape_scrapes_total{status_code="200"} 5595
vm_promscrape_targets{type="consul_sd_configs", status="down"} 0
vm_promscrape_targets{type="consul_sd_configs", status="up"} 0
vm_promscrape_targets{type="dns_sd_configs", status="down"} 0
vm_promscrape_targets{type="dns_sd_configs", status="up"} 0
vm_promscrape_targets{type="ec2_sd_configs", status="down"} 0
vm_promscrape_targets{type="ec2_sd_configs", status="up"} 0
vm_promscrape_targets{type="file_sd_configs", status="down"} 0
vm_promscrape_targets{type="file_sd_configs", status="up"} 0
vm_promscrape_targets{type="gce_sd_configs", status="down"} 0
vm_promscrape_targets{type="gce_sd_configs", status="up"} 0
vm_promscrape_targets{type="kubernetes_sd_configs", status="down"} 0
vm_promscrape_targets{type="kubernetes_sd_configs", status="up"} 70
vm_promscrape_targets{type="static_configs", status="down"} 0
vm_promscrape_targets{type="static_configs", status="up"} 1
vm_protoparser_read_calls_total{type="csvimport"} 0
vm_protoparser_read_calls_total{type="graphite"} 0
vm_protoparser_read_calls_total{type="influx"} 0
vm_protoparser_read_calls_total{type="opentsdb"} 0
vm_protoparser_read_calls_total{type="opentsdbhttp"} 0
vm_protoparser_read_calls_total{type="prometheus"} 0
vm_protoparser_read_calls_total{type="promremotewrite"} 0
vm_protoparser_read_calls_total{type="vmimport"} 0
vm_protoparser_read_errors_total{type="csvimport"} 0
vm_protoparser_read_errors_total{type="graphite"} 0
vm_protoparser_read_errors_total{type="influx"} 0
vm_protoparser_read_errors_total{type="opentsdb"} 0
vm_protoparser_read_errors_total{type="opentsdbhttp"} 0
vm_protoparser_read_errors_total{type="prometheus"} 0
vm_protoparser_read_errors_total{type="promremotewrite"} 0
vm_protoparser_read_errors_total{type="vmimport"} 0
vm_protoparser_rows_read_total{type="csvimport"} 0
vm_protoparser_rows_read_total{type="graphite"} 0
vm_protoparser_rows_read_total{type="influx"} 0
vm_protoparser_rows_read_total{type="opentsdb"} 0
vm_protoparser_rows_read_total{type="opentsdbhttp"} 0
vm_protoparser_rows_read_total{type="prometheus"} 0
vm_protoparser_rows_read_total{type="promremotewrite"} 0
vm_protoparser_rows_read_total{type="promscrape"} 8824561
vm_protoparser_rows_read_total{type="vmimport"} 0
vm_protoparser_unmarshal_errors_total{type="opentsdbhttp"} 0
vm_protoparser_unmarshal_errors_total{type="promremotewrite"} 0
vm_rows_invalid_total{type="csvimport"} 0
vm_rows_invalid_total{type="graphite"} 0
vm_rows_invalid_total{type="influx"} 0
vm_rows_invalid_total{type="opentsdb"} 0
vm_rows_invalid_total{type="opentsdbhttp"} 0
vm_rows_invalid_total{type="prometheus"} 0
vm_rows_invalid_total{type="vmimport"} 0
vm_tcplistener_accepts_total{name="http", addr=":8429"} 736
vm_tcplistener_conns{name="http", addr=":8429"} 2
vm_tcplistener_errors_total{name="http", addr=":8429", type="accept"} 0
vm_tcplistener_errors_total{name="http", addr=":8429", type="close"} 0
vm_tcplistener_errors_total{name="http", addr=":8429", type="read"} 0
vm_tcplistener_errors_total{name="http", addr=":8429", type="write"} 0
vm_tcplistener_read_bytes_total{name="http", addr=":8429"} 67199
vm_tcplistener_read_calls_total{name="http", addr=":8429"} 1338
vm_tcplistener_read_timeouts_total{name="http", addr=":8429"} 490
vm_tcplistener_write_calls_total{name="http", addr=":8429"} 1067
vm_tcplistener_write_timeouts_total{name="http", addr=":8429"} 0
vm_tcplistener_written_bytes_total{name="http", addr=":8429"} 860157
vm_zstd_block_compress_calls_total 0
vm_zstd_block_compressed_bytes_total 0
vm_zstd_block_decompress_calls_total 0
vm_zstd_block_original_bytes_total 0
vmagent_http_request_errors_total{path="/api/v1/import", protocol="vmimport"} 0
vmagent_http_request_errors_total{path="/api/v1/import/csv", protocol="csvimport"} 0
vmagent_http_request_errors_total{path="/api/v1/import/prometheus", protocol="prometheusimport"} 0
vmagent_http_request_errors_total{path="/api/v1/write", protocol="promremotewrite"} 0
vmagent_http_request_errors_total{path="/write", protocol="influx"} 0
vmagent_http_requests_total{path="/-/reload"} 0
vmagent_http_requests_total{path="/api/v1/import", protocol="vmimport"} 0
vmagent_http_requests_total{path="/api/v1/import/csv", protocol="csvimport"} 0
vmagent_http_requests_total{path="/api/v1/import/prometheus", protocol="prometheusimport"} 0
vmagent_http_requests_total{path="/api/v1/write", protocol="promremotewrite"} 0
vmagent_http_requests_total{path="/query", protocol="influx"} 0
vmagent_http_requests_total{path="/targets"} 3
vmagent_http_requests_total{path="/write", protocol="influx"} 0
vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.0e3...1.5e3"} 142
vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.5e3...2.0e3"} 247
vmagent_remotewrite_block_size_bytes_bucket{vmrange="2.0e3...2.5e3"} 267
vmagent_remotewrite_block_size_bytes_bucket{vmrange="2.5e3...3.0e3"} 17
vmagent_remotewrite_block_size_bytes_bucket{vmrange="3.0e3...3.5e3"} 6
vmagent_remotewrite_block_size_bytes_bucket{vmrange="4.5e3...5.0e3"} 89
vmagent_remotewrite_block_size_bytes_bucket{vmrange="7.0e3...7.5e3"} 253
vmagent_remotewrite_block_size_bytes_bucket{vmrange="8.0e3...8.5e3"} 18
vmagent_remotewrite_block_size_bytes_bucket{vmrange="8.5e3...9.0e3"} 177
vmagent_remotewrite_block_size_bytes_bucket{vmrange="9.0e3...9.5e3"} 89
vmagent_remotewrite_block_size_bytes_bucket{vmrange="9.5e3...1.0e4"} 71
vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.0e4...1.5e4"} 18
vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.5e4...2.0e4"} 15
vmagent_remotewrite_block_size_bytes_bucket{vmrange="2.0e4...2.5e4"} 74
vmagent_remotewrite_block_size_bytes_bucket{vmrange="4.0e4...4.5e4"} 352
vmagent_remotewrite_block_size_bytes_bucket{vmrange="4.5e4...5.0e4"} 174
vmagent_remotewrite_block_size_bytes_bucket{vmrange="5.0e4...5.5e4"} 93
vmagent_remotewrite_block_size_bytes_bucket{vmrange="5.5e4...6.0e4"} 85
vmagent_remotewrite_block_size_bytes_bucket{vmrange="6.0e4...6.5e4"} 89
vmagent_remotewrite_block_size_bytes_bucket{vmrange="7.0e4...7.5e4"} 40
vmagent_remotewrite_block_size_bytes_bucket{vmrange="7.5e4...8.0e4"} 145
vmagent_remotewrite_block_size_bytes_bucket{vmrange="8.0e4...8.5e4"} 83
vmagent_remotewrite_block_size_bytes_bucket{vmrange="9.5e4...1.0e5"} 91
vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.0e5...1.5e5"} 524
vmagent_remotewrite_block_size_bytes_bucket{vmrange="1.5e5...2.0e5"} 256
vmagent_remotewrite_block_size_bytes_bucket{vmrange="2.0e5...2.5e5"} 98
vmagent_remotewrite_block_size_bytes_bucket{vmrange="2.5e5...3.0e5"} 92
vmagent_remotewrite_block_size_bytes_bucket{vmrange="3.0e5...3.5e5"} 181
vmagent_remotewrite_block_size_bytes_bucket{vmrange="4.0e5...4.5e5"} 85
vmagent_remotewrite_block_size_bytes_sum 328891477
vmagent_remotewrite_block_size_bytes_count 3871
vmagent_remotewrite_block_size_rows_bucket{vmrange="1.0e1...1.5e1"} 86
vmagent_remotewrite_block_size_rows_bucket{vmrange="1.5e1...2.0e1"} 249
vmagent_remotewrite_block_size_rows_bucket{vmrange="2.0e1...2.5e1"} 54
vmagent_remotewrite_block_size_rows_bucket{vmrange="4.0e1...4.5e1"} 17
vmagent_remotewrite_block_size_rows_bucket{vmrange="4.5e1...5.0e1"} 267
vmagent_remotewrite_block_size_rows_bucket{vmrange="6.5e1...7.0e1"} 6
vmagent_remotewrite_block_size_rows_bucket{vmrange="1.0e2...1.5e2"} 89
vmagent_remotewrite_block_size_rows_bucket{vmrange="1.5e2...2.0e2"} 271
vmagent_remotewrite_block_size_rows_bucket{vmrange="2.0e2...2.5e2"} 355
vmagent_remotewrite_block_size_rows_bucket{vmrange="4.0e2...4.5e2"} 1
vmagent_remotewrite_block_size_rows_bucket{vmrange="5.0e2...5.5e2"} 4
vmagent_remotewrite_block_size_rows_bucket{vmrange="5.5e2...6.0e2"} 84
vmagent_remotewrite_block_size_rows_bucket{vmrange="8.0e2...8.5e2"} 71
vmagent_remotewrite_block_size_rows_bucket{vmrange="8.5e2...9.0e2"} 18
vmagent_remotewrite_block_size_rows_bucket{vmrange="1.0e3...1.5e3"} 884
vmagent_remotewrite_block_size_rows_bucket{vmrange="1.5e3...2.0e3"} 258
vmagent_remotewrite_block_size_rows_bucket{vmrange="2.0e3...2.5e3"} 268
vmagent_remotewrite_block_size_rows_bucket{vmrange="2.5e3...3.0e3"} 88
vmagent_remotewrite_block_size_rows_bucket{vmrange="3.0e3...3.5e3"} 3
vmagent_remotewrite_block_size_rows_bucket{vmrange="3.5e3...4.0e3"} 178
vmagent_remotewrite_block_size_rows_bucket{vmrange="4.5e3...5.0e3"} 88
vmagent_remotewrite_block_size_rows_bucket{vmrange="5.0e3...5.5e3"} 88
vmagent_remotewrite_block_size_rows_bucket{vmrange="5.5e3...6.0e3"} 85
vmagent_remotewrite_block_size_rows_bucket{vmrange="6.0e3...6.5e3"} 3
vmagent_remotewrite_block_size_rows_bucket{vmrange="7.5e3...8.0e3"} 90
vmagent_remotewrite_block_size_rows_bucket{vmrange="9.5e3...1.0e4"} 177
vmagent_remotewrite_block_size_rows_bucket{vmrange="1.0e4...1.5e4"} 89
vmagent_remotewrite_block_size_rows_sum 8241264
vmagent_remotewrite_block_size_rows_count 3871
vmagent_remotewrite_conn_bytes_read_total 105971
vmagent_remotewrite_conn_bytes_written_total 99224763
vmagent_remotewrite_conn_read_errors_total 0
vmagent_remotewrite_conn_reads_total 1168
vmagent_remotewrite_conn_write_errors_total 0
vmagent_remotewrite_conn_writes_total 6707
vmagent_remotewrite_conns 1
vmagent_remotewrite_dial_errors_total 0
vmagent_remotewrite_dials_total 1
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="2.0e-1...2.5e-1"} 204
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="2.5e-1...3.0e-1"} 17
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="3.0e-1...3.5e-1"} 7
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="4.5e-1...5.0e-1"} 61
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="5.0e-1...5.5e-1"} 10
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="5.5e-1...6.0e-1"} 4
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="6.0e-1...6.5e-1"} 1
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="6.5e-1...7.0e-1"} 1
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="7.0e-1...7.5e-1"} 22
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="7.5e-1...8.0e-1"} 6
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="8.0e-1...8.5e-1"} 1
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="9.0e-1...9.5e-1"} 13
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="9.5e-1...1.0e0"} 20
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="1.0e0...1.5e0"} 75
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="1.5e0...2.0e0"} 68
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="2.0e0...2.5e0"} 48
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="2.5e0...3.0e0"} 56
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="3.0e0...3.5e0"} 51
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="3.5e0...4.0e0"} 51
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="4.0e0...4.5e0"} 40
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="4.5e0...5.0e0"} 31
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="5.0e0...5.5e0"} 40
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="5.5e0...6.0e0"} 30
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="6.0e0...6.5e0"} 27
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="6.5e0...7.0e0"} 20
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="7.0e0...7.5e0"} 19
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="7.5e0...8.0e0"} 23
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="8.0e0...8.5e0"} 17
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="8.5e0...9.0e0"} 5
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="9.0e0...9.5e0"} 15
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="9.5e0...1.0e1"} 17
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="1.0e1...1.5e1"} 96
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="1.5e1...2.0e1"} 46
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="2.0e1...2.5e1"} 18
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="2.5e1...3.0e1"} 5
vmagent_remotewrite_duration_seconds_bucket{url="secret-url-1",vmrange="3.0e1...3.5e1"} 1
vmagent_remotewrite_duration_seconds_sum{url="secret-url-1"} 5314.481205986998
vmagent_remotewrite_duration_seconds_count{url="secret-url-1"} 1166
vmagent_remotewrite_errors_total{url="secret-url-1"} 0
vmagent_remotewrite_global_relabel_metrics_dropped_total 0
vmagent_remotewrite_pending_data_bytes{path="vmagent-remotewrite-data/persistent-queue/1A28D788E0FF4747", url="secret-url-1"} 230173414
vmagent_remotewrite_pending_inmemory_blocks{path="vmagent-remotewrite-data/persistent-queue/1A28D788E0FF4747", url="secret-url-1"} 0
vmagent_remotewrite_relabel_metrics_dropped_total{path="vmagent-remotewrite-data/persistent-queue/1A28D788E0FF4747", url="secret-url-1"} 0
vmagent_remotewrite_requests_total{url="secret-url-1", status_code="2XX"} 1166
vmagent_remotewrite_retries_count_total{url="secret-url-1"} 0
vmagent_rows_inserted_total{type="csvimport"} 0
vmagent_rows_inserted_total{type="graphite"} 0
vmagent_rows_inserted_total{type="influx"} 0
vmagent_rows_inserted_total{type="opentsdb"} 0
vmagent_rows_inserted_total{type="opentsdbhttp"} 0
vmagent_rows_inserted_total{type="prometheus"} 0
vmagent_rows_inserted_total{type="promremotewrite"} 0
vmagent_rows_inserted_total{type="vmimport"} 0
go_memstats_alloc_bytes 119162536
go_memstats_alloc_bytes_total 3365267976
go_memstats_buck_hash_sys_bytes 1739573
go_memstats_frees_total 20930331
go_memstats_gc_cpu_fraction 0.00015187926126616682
go_memstats_gc_sys_bytes 13132048
go_memstats_heap_alloc_bytes 119162536
go_memstats_heap_idle_bytes 145989632
go_memstats_heap_inuse_bytes 120479744
go_memstats_heap_objects 410136
go_memstats_heap_released_bytes 136478720
go_memstats_heap_sys_bytes 266469376
go_memstats_last_gc_time_seconds 1.59921479125121e+09
go_memstats_lookups_total 0
go_memstats_mallocs_total 21340467
go_memstats_mcache_inuse_bytes 1736
go_memstats_mcache_sys_bytes 81920
go_memstats_mspan_inuse_bytes 637160
go_memstats_mspan_sys_bytes 737280
go_memstats_next_gc_bytes 123563856
go_memstats_other_sys_bytes 719827
go_memstats_stack_inuse_bytes 1966080
go_memstats_stack_sys_bytes 1966080
go_memstats_sys_bytes 284846104
go_cgo_calls_count 5
go_cpu_count 40
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 1.5387e-05
go_gc_duration_seconds{quantile="1"} 0.001098997
go_gc_duration_seconds_sum 0.013194626
go_gc_duration_seconds_count 65
go_gc_forced_count 0
go_gomaxprocs 1
go_goroutines 169
go_threads 9
go_info{version="go1.15"} 1
go_info_ext{compiler="gc", GOARCH="amd64", GOOS="linux", GOROOT="go"} 1
process_cpu_seconds_system_total 8.98
process_cpu_seconds_total 117.44
process_cpu_seconds_user_total 108.46
process_major_pagefaults_total 0
process_minor_pagefaults_total 716322
process_num_threads 8
process_resident_memory_bytes 233848832
process_start_time_seconds 1599209571
process_virtual_memory_bytes 1471975424
vm_app_version{version="vmagent-20200815-125807-tags-v1.40.0-0-ged00eb3f3"} 1
vm_allowed_memory_bytes 2576980377
vm_app_start_timestamp 1599209571
vm_app_uptime_seconds 5327
flag{name="csvTrimTimestamp", value="1ms"} 1
flag{name="dryRun", value="false"} 1
flag{name="enableTCP6", value="false"} 1
flag{name="envflag.enable", value="false"} 1
flag{name="envflag.prefix", value=""} 1
flag{name="fs.disableMmap", value="false"} 1
flag{name="graphiteListenAddr", value=""} 1
flag{name="graphiteTrimTimestamp", value="1s"} 1
flag{name="http.disableResponseCompression", value="false"} 1
flag{name="http.maxGracefulShutdownDuration", value="7s"} 1
flag{name="http.pathPrefix", value=""} 1
flag{name="http.shutdownDelay", value="0s"} 1
flag{name="httpAuth.password", value="secret"} 1
flag{name="httpAuth.username", value=""} 1
flag{name="httpListenAddr", value=":8429"} 1
flag{name="import.maxLineLen", value="104857600"} 1
flag{name="influxListenAddr", value=""} 1
flag{name="influxMeasurementFieldSeparator", value="_"} 1
flag{name="influxSkipMeasurement", value="false"} 1
flag{name="influxSkipSingleField", value="false"} 1
flag{name="influxTrimTimestamp", value="1ms"} 1
flag{name="insert.maxQueueDuration", value="1m0s"} 1
flag{name="loggerErrorsPerSecondLimit", value="10"} 1
flag{name="loggerFormat", value="default"} 1
flag{name="loggerLevel", value="INFO"} 1
flag{name="loggerOutput", value="stderr"} 1
flag{name="maxConcurrentInserts", value="160"} 1
flag{name="maxInsertRequestSize", value="33554432"} 1
flag{name="memory.allowedBytes", value="0"} 1
flag{name="memory.allowedPercent", value="60"} 1
flag{name="metricsAuthKey", value="secret"} 1
flag{name="opentsdbHTTPListenAddr", value=""} 1
flag{name="opentsdbListenAddr", value=""} 1
flag{name="opentsdbTrimTimestamp", value="1s"} 1
flag{name="opentsdbhttp.maxInsertRequestSize", value="33554432"} 1
flag{name="opentsdbhttpTrimTimestamp", value="1ms"} 1
flag{name="pprofAuthKey", value="secret"} 1
flag{name="promscrape.config", value="/config/scrape.yaml"} 1
flag{name="promscrape.config.dryRun", value="false"} 1
flag{name="promscrape.config.strictParse", value="false"} 1
flag{name="promscrape.configCheckInterval", value="30s"} 1
flag{name="promscrape.consulSDCheckInterval", value="30s"} 1
flag{name="promscrape.disableCompression", value="false"} 1
flag{name="promscrape.disableKeepAlive", value="false"} 1
flag{name="promscrape.discovery.concurrency", value="100"} 1
flag{name="promscrape.discovery.concurrentWaitTime", value="1m0s"} 1
flag{name="promscrape.dnsSDCheckInterval", value="30s"} 1
flag{name="promscrape.ec2SDCheckInterval", value="1m0s"} 1
flag{name="promscrape.fileSDCheckInterval", value="30s"} 1
flag{name="promscrape.gceSDCheckInterval", value="1m0s"} 1
flag{name="promscrape.kubernetesSDCheckInterval", value="30s"} 1
flag{name="promscrape.maxScrapeSize", value="16777216"} 1
flag{name="promscrape.suppressScrapeErrors", value="false"} 1
flag{name="remoteWrite.basicAuth.password", value="secret"} 1
flag{name="remoteWrite.basicAuth.username", value="agent"} 1
flag{name="remoteWrite.bearerToken", value="secret"} 1
flag{name="remoteWrite.decimalPlaces", value="0"} 1
flag{name="remoteWrite.flushInterval", value="1s"} 1
flag{name="remoteWrite.label", value=""} 1
flag{name="remoteWrite.maxBlockSize", value="33554432"} 1
flag{name="remoteWrite.maxDiskUsagePerURL", value="0"} 1
flag{name="remoteWrite.proxyURL", value=""} 1
flag{name="remoteWrite.queues", value="1"} 1
flag{name="remoteWrite.relabelConfig", value=""} 1
flag{name="remoteWrite.sendTimeout", value="1m0s"} 1
flag{name="remoteWrite.showURL", value="false"} 1
flag{name="remoteWrite.tlsCAFile", value=""} 1
flag{name="remoteWrite.tlsCertFile", value=""} 1
flag{name="remoteWrite.tlsInsecureSkipVerify", value="false"} 1
flag{name="remoteWrite.tlsKeyFile", value="secret"} 1
flag{name="remoteWrite.tlsServerName", value=""} 1
flag{name="remoteWrite.tmpDataPath", value="vmagent-remotewrite-data"} 1
flag{name="remoteWrite.url", value="secret"} 1
flag{name="remoteWrite.urlRelabelConfig", value=""} 1
flag{name="tls", value="false"} 1
flag{name="tlsCertFile", value=""} 1
flag{name="tlsKeyFile", value="secret"} 1
flag{name="version", value="false"} 1
Btw, can you measure bandwidth with iperf or similar tool to victoria-metrics from vmanget node?
Also, check metric: vmagent_remotewrite_conn_bytes_written_total, is it increase after vmagent stopped to send metrics?
I was able to reproduce it with bandwidth limit. Neither high ping (500ms), nor packet loss (10%) wasn't seriously affect data ingestion. So probably, its your case. I assume, that you run vmagent as kubernetes pod and restart recreates it with losing local data, so after restart you are able to see some metrics.
f41gh7
on 4 Sep 2020
Btw, can you measure bandwidth with iperf or similar tool to victoria-metrics from vmanget node?
Also, check metric:
vmagent_remotewrite_conn_bytes_written_total, is it increase after vmagent stopped to send metrics?I was able to reproduce it with bandwidth limit. Neither high ping (500ms), nor packet loss (10%) wasn't seriously affect data ingestion. So probably, its your case. I assume, that you run vmagent as kubernetes pod and restart recreates it with losing local data, so after restart you are able to see some metrics.
Connecting to host 10.12.110.72, port 5201
[ 4] local 10.0.10.1 port 57512 connected to 10.12.110.72 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 105 KBytes 856 Kbits/sec 4 8.48 KBytes
[ 4] 1.00-2.00 sec 65.0 KBytes 533 Kbits/sec 0 14.1 KBytes
[ 4] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 4 9.90 KBytes
[ 4] 3.00-4.00 sec 69.3 KBytes 568 Kbits/sec 0 14.1 KBytes
[ 4] 4.00-5.00 sec 67.9 KBytes 556 Kbits/sec 1 9.90 KBytes
[ 4] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 2 8.48 KBytes
[ 4] 6.00-7.00 sec 62.2 KBytes 510 Kbits/sec 4 5.66 KBytes
[ 4] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 9.90 KBytes
[ 4] 8.00-9.00 sec 63.6 KBytes 521 Kbits/sec 0 9.90 KBytes
[ 4] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 1 7.07 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 433 KBytes 354 Kbits/sec 16 sender
[ 4] 0.00-10.00 sec 389 KBytes 319 Kbits/sec receiver
iperf Done.
Btw, can you measure bandwidth with iperf or similar tool to victoria-metrics from vmanget node?
Also, check metric:
vmagent_remotewrite_conn_bytes_written_total, is it increase after vmagent stopped to send metrics?I was able to reproduce it with bandwidth limit. Neither high ping (500ms), nor packet loss (10%) wasn't seriously affect data ingestion. So probably, its your case. I assume, that you run vmagent as kubernetes pod and restart recreates it with losing local data, so after restart you are able to see some metrics.
According to Grafana last vmagent_remotewrite_conn_bytes_written_total measurement 39437199 then it's stops sending any metrics. According to /metrics dump which was provided here this metric still rising 99224763.
ofen
on 4 Sep 2020
Btw, can you measure bandwidth with iperf or similar tool to victoria-metrics from vmanget node?
Also, check metric:vmagent_remotewrite_conn_bytes_written_total, is it increase after vmagent stopped to send metrics?
I was able to reproduce it with bandwidth limit. Neither high ping (500ms), nor packet loss (10%) wasn't seriously affect data ingestion. So probably, its your case. I assume, that you run vmagent as kubernetes pod and restart recreates it with losing local data, so after restart you are able to see some metrics.According to Grafana last vmagent_remotewrite_conn_bytes_written_total measurement 39437199 then it's stops sending any metrics. According to /metrics dump which was provided here this metric still rising 99224763.
Seems like 319 Kbits/sec isn't enough for vmagent to send your scraped data to VictoriaMetrics. You can check it, by removing all jobs, except job="vmagent", clean up local data and restart vmagent.
f41gh7
on 4 Sep 2020
You are right if there are only vmagent as target then there is no such issue. As temporary workaround for such cases Grafana cloud agent could be used (as tradeoff it will consome more RAM but works good on low bandwidth).
ofen
on 4 Sep 2020
From my point of view, following solutions could help:
- increase bandwidth, if its possible.
- drop unneeded metrics (Kubernetes exports a lot of them, especially api server) with relabel config.
- increase scrape interval.
- split scrape jobs between two vmagents and place them to different nodes ( if per node limit is applied).
f41gh7
on 5 Sep 2020
👍1
As temporary workaround for such cases Grafana cloud agent could be used
Grafana cloud agent uses the same Prometheus remote_write protocol for sending data to remote storage as vmagent does. The only difference between them is that Grafana cloud agent sends the data to remote storage over multiple concurrent connections, while vmagent uses a single connection by default. Probably a single connection isn't enough due to high network latency. Could you try running vmagent with -remoteWrite.queues=10 or something like that? This will increase the default number of connections to remote storage from 1 to 10. See more details at https://victoriametrics.github.io/vmagent.html#troubleshooting .
valyala
on 7 Sep 2020
As temporary workaround for such cases Grafana cloud agent could be used
Grafana cloud agent uses the same Prometheus remote_write protocol for sending data to remote storage as vmagent does. The only difference between them is that Grafana cloud agent sends the data to remote storage over multiple concurrent connections, while vmagent uses a single connection by default. Probably a single connection isn't enough due to high network latency. Could you try running
vmagentwith-remoteWrite.queues=10or something like that? This will increase the default number of connections to remote storage from 1 to 10. See more details at https://victoriametrics.github.io/vmagent.html#troubleshooting .
Still testing but after setting --remoteWrite.queues=10 having no issue with same set-up (agent working more that 20 minutes and according to /metrics there are no pending metrics vm_persistentqueue_bytes_pending{path="vmagent-remotewrite-data/persistent-queue/1A28D788E0FF4747"} 0).
ofen
on 8 Sep 2020
🎉1
after setting --remoteWrite.queues=10 having no issue with same set-up (agent working more that 20 minutes and according to /metrics there are no pending metrics vm_persistentqueue_bytes_pending{path="vmagent-remotewrite-data/persistent-queue/1A28D788E0FF4747"} 0).
Cool! Then closing this issue as resolved.
valyala
on 8 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
Serrvosky
路
3Comments
EricAntoni
路
3Comments
jelmd
路
3Comments
localpref
路
3Comments
abualy
路
3Comments