Vector: Kubernetes documentation

[binarylogic]

I'm opening this issue to represent a single, final, place for Kubernetes documentation. A lot of this is scattered throughout pull requests and issues, and I'd like a single final document that we can use to build out the website, docs, and marketing pages.

• [x] Include a step-by-step guide for deploying Kubernetes. If there are different deployment models (agent vs sidecar, etc) please think about the from the user's perspective. Ideally we would recommend a single deployment model that we think is the easiest/best. I don't want to overwhelm or confuse the user. Simplicity is the best.
• [x] Add sections any other Kuberenetets specific details that might be relevant for marketing or documentation.
  
  
  
  • [x] Minimal supported Kubernetes version.
  
  

[ktff]

Out of three deployments that we do/probably-will support, agent per node is simplest to deploy as it doesn't require networking(as centralized) or modifying user files(as sidecar). It is also a deployment that we currently support.

---

Deployment guide

This is a guide on how to deploy Vector to every node in a Kubernetes cluster and collect logs from node's Pods. Also known as distributed topology.

This guide requires kubectl.

1. Download configuration

Download:
https://raw.githubusercontent.com/timberio/vector/master/config/kubernetes/vector-daemonset.yaml

It contains multiple yaml documents that define how Vector is to be deployed to a Kubernetes cluster.

2. Configure Vector

In the downloaded file, the section marked with:

# NOTE: Here, vector can be configured

contains embedded configuration file that will be passed to Vector instances, and can be customized as any other Vector configuration.

The configuration already contains configured kubernetes source.

Also do mind the indentation as it defines what is and isn't Vector configuration.

3. Deploy it

kubectl apply -f vector-daemonset.yaml

This will deploy Vector with customized configuration to all nodes in the cluster, and will start collecting logs.

---

Other deployment related sections

Besides deployment, there are other relevant actions.

Reconfigure Vector

Repeat deployment steps 2. and then 3. That's it.

Or alternatively, Vector configuration could be copied to a separate vector.toml file from the downloaded file. Then reconfiguration would look like:

1. Change Vector configuration in vector.toml.

2. Update configuration of deployed Vector instances with:

   kubectl --namespace telemetry create configmap vector-config --from-file=vector-agent-config=./vector.toml -o yaml --dry-run | kubectl replace -f -
   

   this will create configmap with new Vector configuration and replace existing one with it.

Both ways will reconfigure all Vector instances with the new configuration on the fly.

It will take some time for the changes to be propagated.

Do note that if Vector failed to start because of invalid configuration, it will take longer for Kubernetes to restart it with new configuration, because it will enter CrashLoopBackOff state. In that case, it would probably be faster to delete Vector before doing deployment step 3.

Global configurations can only be changed via first way by deleting Vector before doing deployment step 3.

Delete Vector

kubectl delete -f vector-daemonset.yaml

This will delete deployed Vector, but will also delete namespace telemetry that may or may not have existed before Vector deployment, so make sure you aren't using it for other things. And if you are, the easiest thing to do is to change vector-daemonset.yaml to use different namespace before deploying it.

Debug Vector

If there is something wrong, usually an error in Vector configuration, it will be hardly visible unless it's checked.

1. List pods in Vector namespace with:
   kubectl --namespace telemetry get pod
2. Select one of Vector pods, generally any is fine, but if it has Error state, it's even better. And, with it's Name, ex. vector-agent-jn5fn, replacing POD_NAME, run:
   kubectl --namespace telemetry logs POD_NAME
   This will print out it's logs and hopefully be useful in determining what's wrong, if anything.

---

Other sections

Default exclude

If all include options are empty, kube-system namespace and vector* container names are excluded.

Node operating system

Nodes with Windows are not supported.

Supported Kubernetes

Currently all Kubernetes related features are supported for Kubernetes v1.14 and higher, but some features do work with lower versions. These minimal versions can be found in the documentation of their features.

---

cc @binarylogic

[LucioFranco]

@ktff for configuring vector, do you think it would be good to show how to directly pass a vector.toml to kube. I think this can be done via kubectl. The other thing would be good to talk about credentials, like aws access keys and how users should deploy those. Otherwise, this looks great.

[binarylogic]

Good points. @ktff please feel free to update your original comment to keep things focused and reduce noise. We can treat that as the master document that we'll run off of.

[ktff]

How to directly pass vector.toml has been added.

I think it's fine having it, so long as we emphasize it as an alternative.

[ktff]

Regarding credentials, I think it's out of scope for the guide, as it more has to do with kubectl/AWS configuration then with Vector.

But, we could add it to an AWS EKS guide which would reference this one, and have a section about authentication where we would reference an official AWS guide. Such a guide could also be used to cover alternative vector deployments specific to the cluster provider. In AWS EKS case, a deployment over it's website.

[binarylogic]

@ktff would you mind updating this guide with any relevant changes? Specifically the config watching change (if necessary).

[ktff]

@binarylogic it's updated, so the current guide reflects the current state of Vector.

But, I'll open a few issues that will remove the gotchas in the update story. Although it's a question when they will be added.

Regarding supported Kubernetes version, there are two compatible ways for us to mention it:

1. Have a section where we mention that with Kubernetes v1.14 and higher, every feature interacting with Kubernetes will work.
   
   
   
   • This mention will have to be updated at times when we add new feature which requires higher version.
   
   
   • Simplest for users.
   
   
2. At every feature, mention it's minimal supported version.
   
   
   
   • On it's own, users would have to check every feature that they want to use.
   
   
   • It would be clearer that increasing the minimal version for a feature is a breaking change.
   
   

[binarylogic]

For the installation guide, I'd prefer to pick the lowest common denominator. So "v1.14 and higher".

[ktff]

Supported Kubernetes section has been added, and timberio/vector-website#33 has been opened for documenting minimal version per feature.

[zzswang]

when I try to deploy https://github.com/timberio/vector/blob/master/distribution/kubernetes/vector-namespaced.yaml on minikube.

The pod throws error

╰─ kubectl logs -f vector-lclmx                              
Aug 02 01:21:23.380  INFO vector: Log level "info" is enabled.
Aug 02 01:21:23.385  INFO vector: Loading configs. path=["/etc/vector/managed.toml"]
Aug 02 01:21:23.388 ERROR vector: Configuration error: "/etc/vector/managed.toml": unknown variant `kubernetes_logs`, expected one of `docker`, `file`, `generator`, `http`, `internal_metrics`, `journald`, `kafka`, `logplex`, `prometheus`, `socket`, `splunk_hec`, `statsd`, `stdin`, `syslog`, `vector` for key `sources.kubernetes_logs.type`

It seems the source.kubernetes_logs is illegal.

[MOZGIII]

@zzswang thank you for trying our new implementation!

We haven't released the new Kubernetes implementation yet, and we're planning to ship it at v0.11. Until then, you can use a nightly docker image.

Unfortunately, we've had some issues with nightly images in the last few days, so I'll just prepare an image for you off the master branch.

[zzswang]

Thank you I’ll have a try.

[MOZGIII]

I'll upload as mozgiii/vector-k8s-wip:rev8 to Docker Hub in a bit.

UPD: done!

[zzswang]

Thanks @MOZGIII , It works.