Cadvisor: Feature Request: Support ELK

[catrixs]

ELK stack is heavily used, but cAdvisor only support influxdb. So we want to implement a log stash forward drive base on Lumberjack Protocol.

Is it a good idea?

[vishh]

Is ELK built to handle metrics data?

On Tue, Apr 7, 2015 at 1:20 AM, 陈飞 [email protected] wrote:

ELK stack is heavily used, but cAdvisor only support influxdb. So we want
to implement a log stash forward drive base on Lumberjack Protocol.

Is it a good idea?

—
Reply to this email directly or view it on GitHub
https://github.com/google/cadvisor/issues/634.

[catrixs]

@vishh Yah. We use ELK to collect container metrics data in production environment.

ELK represents ElasticSearch + LogStash + Kibana. We think cAdvisor could be able to forward metrics data to LogStash, and then we could use ElasticSearch and Kibana to create dashboard, and like this one:

[wangjingbomail]

It is a good idea

[vishh]

Ok! Interesting! Are you interested in posting a PR to add elasticsearch as
a backend?

On Tue, Apr 7, 2015 at 8:38 PM, wangjingbomail [email protected]
wrote:

It is a good idea

—
Reply to this email directly or view it on GitHub
https://github.com/google/cadvisor/issues/634#issuecomment-90794653.

[thaJeztah]

Nice looking screenshot! Definitely interesting

[catrixs]

@vishh I'm working on it.

[vishh]

Great!

On Wed, Apr 8, 2015 at 7:26 PM, 陈飞 [email protected] wrote:

@vishh https://github.com/vishh I'm working on it.

—
Reply to this email directly or view it on GitHub
https://github.com/google/cadvisor/issues/634#issuecomment-91095253.

[ternel]

Would be great :)

[bastnic]

:+1:

[catrixs]

https://github.com/google/cadvisor/pull/693

We just submit a pull request which is very simple. Just implements a redis driver which use lpush to push metrics json to redis, then logstash could forward them to ElasticSearch.

Next step, we want implement a native ElasticSearch driver.

[rjnagal]

awesome! thanks :+1:

[kangman]

:+1:

[jason2055141]

Hi @feiyang21687 How did you get that? What plugin are you using to collect data's or logs in containers? Can you please advice? Thanks. By the way this is a nice and interesting post. Thanks men.

[catrixs]

Hi @jason2055141 We use cAdvisor to collect stats data of container, use scribe to collect logs of container, use --volume to store the logs of container on local host, use ELK and Graphite to visualize the data.

[jason2055141]

@feiyang21687
Thanks men. I will try that scribe to collects logs from container. Thanks again.

[kerwin]

+1

[trompx]

An elasticsearch storage driver would definitly be a great add ! instead of having to store data to redis first then forward to logstash to save it on elasticsearch.

[cteyton]

+1 for a native elastic search driver

[philipz]

+1

[rjnagal]

@feiyang21687 Are you still planning to work on this?

[catrixs]

@rjnagal yeah, this is the PR, thank you @hacpai
https://github.com/google/cadvisor/pull/875

[cteyton]

thanks @hacpai

[hacpai]

You are welcome. @feiyang21687 @cteyton

I hope this could help you. :)

[gregbkr]

Hello, thank you for adding the ELK support to cadvisor, that is a great news! ^^
This command work, and I can see the data in kibana (in index: cadvisor):
docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
google/cadvisor:latest \
-storage_driver="elasticsearch" -alsologtostderr=true -storage_driver_es_host="http://elasticsearch_ip:9200"

By any chance, anyone tried to make a dashboard of these data? I am stuck with the field container_Name, which container some "/" and is analysed (and cut). Usually I use the field container_Name.raw to escape but it is not available. Anyone could help, or share some dashboards?

Thanks a lot! and good weekend!
Greg.

[cteyton]

To overcome the missing ".raw" issue, i think you should get into the es driver of cadvisor and figure out what's missing. If the container name is a String, thus it's strange that no "raw" value is available.

[jimmidyson]

.raw suffixes are added via Elasticsearch configuration AFAIK. When you're using Logstash it uses an Elasticsearch template to create these multifields (I think they're called) in Elasticsearch. There isn't really anything for cadvisor to do.

[pmoust]

@gregbkr In Kibana 4, go to Settings -> Indicies -> Select your index pattern -> hit Refresh to repopulate, find the appropriate fields and index them as needed.

[cteyton]

sorry, thanks for correcting me !

[gregbkr]

Good evening everyone!

Thank you for your posts. @pmoust : I followed your path and the issue seems coming from the fact that the field container_Name is analyzed (so split in sub-field) , so I try to edit via setting/indices as showed, but there isn't an option to do that. So I tried via command line.

• get field mapping :

curl -XGET 'http://localhost:9200/cadvisor/_mapping/stats/field/container_Name'
{"cadvisor":{"mappings":{"stats":{"container_Name":{"full_name":"container_Name","mapping":{"container_Name":{"type":"string"}}}}}}}

• force container_Name to not_analysed

curl -XPUT 'http://localhost:9200/cadvisor/_mapping/stats' -d '
{
"properties" : {
"container_Name" : {"type" : "string", "index" : "not_analyzed" }
}
}'

• Error, cannot change index of a analysed field already in base

{"error":"MergeMappingException[Merge failed with failures {[mapper [container_Name] has different index values, mapper [container_Name] has different tokenize values, mapper [container_Name] has different index_analyzer]}]","status":400

• So I delete cadvisor index, and will try to create a template for mapping when cadvisor fresh start and create the index. The template say, for index=cadvisor, set container_Name = not_analysed

curl -XPUT http://localhost:9200/_template/cadvisor -d '
{
"template" : "cadvisor*",
"mappings" : {
"stats" : {
"container_Name":{"type":"string", "index":"not_analyzed"}
}
}
}'

• Run cadvisor but error. Seems like cadvisor don't like when the field is preconfigured via template.

panic: failed to write stats to ElasticSearch- elastic: Error 400 (Bad Request): MapperParsingException[mapping [stats]]; nested: MapperParsingException[Root type mapping not empty after parsing! Remaining fields: [container_Name : {index=not_analyzed, type=string}]];

I had another idea before to post these comments, I copied the template from Logstash-* and create a same one for cadvisor. In string_fields" section, there is a automatic operation which get the raw value of analysed field.

• Get your actual setting through:

curl -XGET 'http://localhost:9200/_template/'?pretty

• Copy and change "logstash.." for cadvisor, and send that template to elastic:

curl -XPUT http://localhost:9200/_template/cadvisor -d '
{
"template" : "cadvisor",
"mappings" : {
"_default_" : {
"dynamic_templates" : [ {
"message_field" : {
"mapping" : {
"index" : "analyzed",
"omit_norms" : true,
"type" : "string"
},
"match_mapping_type" : "string",
"match" : "message"
}
}, {
"string_fields" : {
"mapping" : {
"index" : "analyzed",
"omit_norms" : true,
"type" : "string",
"fields" : {
"raw" : {
"ignore_above" : 256,
"index" : "not_analyzed",
"type" : "string"
}
}
},
"match_mapping_type" : "string",
"match" : ""
}
} ],
"_all" : {
"omit_norms" : true,
"enabled" : true
},
"properties" : {
"geoip" : {
"dynamic" : true,
"type" : "object",
"properties" : {
"location" : {
"type" : "geo_point"
}
}
},
"@version" : {
"index" : "not_analyzed",
"type" : "string"
}
}
}
},
"aliases" : { }
}
}'

• delete index cadvisor, delete cadvisor container and recreate it. I found out that everything works, the field container_Name got now a raw value, which can be used in visualization.

Thank you all, this will make great dashboard!
And good week!
Greg.

[patrick-bark]

I like where this is headed, but is there no way to send this to logstash? Otherwise, creating daily indices is difficult, making retiring old data even more difficult. Furthermore that would take care of the .raw value issue. I rarely see ES ran without logstash and they really always should be paired. I think this is a real miss by going straight to ES.

[fiunchinho]

@gregbkr I have your same problem, but your solution doesn't seem to work for me. I still get the '/' container name after doing your proposed solution. Any help?

[gregbkr]

@fiunchinho : sorry, it was working for me for sometimes, but had few performance problems on ELK. I move on to prometheus container for monitoring stats.

[fiunchinho]

I'd love to hear more details on your problems. We are using ELK for the logs, so having system metrics on ElasticSearch looks the way to go for me, instead of adding yet another tool (presumably prometheus)

[gregbkr]

I am still not fixed to one solution ofr monitoring docker infra, but I will get back to elk when I got more time. I did that doc if that could help anyone. https://github.com/gregbkr/docker-elk-cadvisor-dashboards

[JamieCressey]

Sorry to drag this back up, but I can see a LogStash driver being hugely beneficial here.

The use case I'm thinking of being LogStash has native support to insert data into dynamic indexes, so metrics could be stored in a multitude of ways based on a user's requirements.

E.g. Two departments are running docker services, each get a department=foo or department=barlabel assigned to their containers. With labels being passed through with each metric to LogStash the output index by can be dynamically chosen, either foo_metrics_2017-02-08 or bar_metrics_2017-02-08 dependant on the metric's origin. Using something along the lines of;

input {
...
}
filter {
  json {
    add_field => { "department" => ... }
  }
}
output {  
    elasticsearch {  
        index => "%{department}_metrics_%{index_day}"   
    }
} 

This also means the processing of metrics then becomes outside the scope of CAdvisor.

[dashpole]

Storage discussions: #1458