Kops: Simple method for creating cluster centralized logging?

[ericsage]

With the kube-up script one could automatically setup elastisearch, kibana, and fluentd on each node to inject container logs for analysis. I need this functionality for a production deployment, but I don't see any documentation in Kops to set this up. Can Kops handle this during creation or as an add-on?

[chrislovecnm]

@ericsage we only handle the base install. We provision a production ready cluster, and the bare components to bootstrap the cluster. No bells and whistles :) You get to have fun and do that yourself!!

Questions? Othewise, please close the issue give it a close.

[kris-nova]

Is container logging bells and whistles? Or is this just one of the (more) rich features we could see from kops in the future?

Not that I think we should include this functionality in kops, but offering an avenue to extend kops might be worthwhile in cases like this. Thoughts?

[chrislovecnm]

Plugin model that we have talked about, could address this, but right now let's get deployments awesome. The problem is there are three different dashboards and 30 different logging options that I can pull off the top of my head. There are sister projects like helm that address installs very well. Deis has a couple of other projects as well.

Thoughts?

[chrislovecnm]

We could look at a helm plugin ... Thoughts?

[yissachar]

See https://github.com/kubernetes/kops/issues/278

[ericsage]

I would much prefer an opinionated solution exist today like the one we received with the kube up script, it was a big selling point for my team that we would not need to setup a centralized logging stack on our own. We are a small team in a cancer research lab with limited capacity for Ops. In the meantime, if there are instructions on how to setup such a stack manually, I have had a hard time finding them and would love a push in the right direction. The addons documentation also details how to run heapster, but doesn't provide any information about how to connect it to graphana and influxdb which would also be a 👍 for people looking for turn key solutions.

[chrislovecnm]

@ericsage I get you and I think we can beef up documentation more on how to install these components. The components included are not difficult to install, just the knowledge I think is buried.

K8s and turn key :) That is a lofty goal, not saying it is impossible, but lofty.

[overip]

I am running in the same problem. I am struggling finding a good tutorial on how to install ELK (Elasticsearch, Logstash, Kibana) or EFK (Elasticsearch, Fluentd, Kibana) in containers throughout the pods in my cluster to centralize container logs. Is documentation on this topic still lacking? If that's the case, I promise that as soon as I figure it out, I'll offer my documentation to the community

[justinsb]

Also see #1305

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close