Kubevirt: Add log aggregator and tooling

[fabiand]

To simplify debugging in the cluster we should add a log aggregation stack like EFK (elasticsearch, fluentd, kibana), as it is developed i.e. here

• https://github.com/openshift/origin-aggregated-logging
• https://github.com/Skillshare/kubernetes-efk

[cynepco3hahue]

It looks interesting, but I do not sure if it must be part of Kubevirt, like we can see it can be added to k8s and openshift in different ways.
Maybe we can add some anisble playbook to https://github.com/kubevirt-incubator/kubevirt-ansible that will deploy EFK stack on k8s, for dev purpose only

• k8s EFK stack deployment via scripts - https://github.com/kayrus/elk-kubernetes

[fabiand]

I agree. It must not be part of the core KubeVirt manifests.

But it does make sense to provide it in the ansible playbooks, just to be able to define the best suited environment.

@cynepco3hahue can you reopen this bug on the playbook side?

[cynepco3hahue]

@fabiand I think I missed the fact that we will run each component as container so we do not really need ansible at all, so we can start addition project under incubator, that will include Dockerfiles and manifests for deployment

[kfox1111]

I added a helm chart for log fowarding here:
https://github.com/kubernetes/charts/tree/master/stable/fluent-bit

Should pull the container logs off of all the hosts and ship them.

I'm interested in an e/k set of charts for standing up the rest.

[fabiand]

@cynepco3hahue We do not need to do the packaging.

It's done already in the opneshift package, see:

OpenShift Ansible Logging Role (openshift_logging)
The openshift-ansible openshift_logging role orchestrates the deployment of the logging stack including: resource definitions, key/cert generation, component start and stop order.

From https://github.com/openshift/origin-aggregated-logging#openshift-ansible-logging-role-openshift_logging

@kfox1111 helm charts for kubevirt would also be intresting.

[kfox1111]

+1 for helm charts. :)

[cynepco3hahue]

@fabiand helm charts look good, it can be also a good way to distribute KubeVirt manifests for k8s environment

[fabiand]

Yep - But also here we need to see that we get #265 fixed to simplify the charts creation.

[cynepco3hahue]

@fabiand I checked EFK stuck for dev environment, and I do not sure if it reasonable to run such heavy thing on dev environment with limited resources. I do not talk about fluent thanks to @kfox1111 chart it has easy installation, but the installation of es from https://github.com/kubernetes/charts/tree/master/incubator/elasticsearch more complex and needs more resources(es data pod requires a lot of memory for java heap). In additional it uses a stateful set for deployment that requires dynamic provisioning of persistent disks.

[kfox1111]

There is a file backend to the fluent-bit chart that can dump everything to a single file. that might work for devel purposes?

[fabiand]

I'm closing this bug because it must not be part of core kubevirt - after all it's the decision of the operator or k8s distribution to provide the right aggregator.

[rmohr]

Just for reference, for our development and CI clusters, such things are now mostly done in https://github.com/kubevirt/kubevirtci.