Minikube: How to enable auditing?

[bygui86]

The exact command to reproduce the issue:

minikube start \
--vm-driver hyperkit \
--extra-config=apiserver.authorization-mode=RBAC \
--extra-config=apiserver.Audit.LogOptions.Path=/var/log/apiserver/audit.log \
--extra-config=apiserver.Audit.PolicyFile=/etc/kubernetes/addons/audit-policy.yaml

I tried several extra-config variants:

• apiserver.Audit.LogOptions.Path, apiserver.Audit.PolicyFile
• apiserver.audit-logoptions-path, apiserver.audit-policyfile
• apiserver.audit-log-options-path, apiserver.audit-policy-file
• apiserver.audit-policy-path, apiserver.audit-log-dir, apiserver.audit-log-max-age

together with different values:

• log options path / log path / path
  
  
  
  • /var/log/kube-apiserver-audit.log
  
  
  • /etc/kubernetes/logs/apiserver-audit.log
  
  
• log dir
  
  
  
  • /var/log/kubernetes/
  
  
• policy path
  
  
  
  • /etc/kubernetes/addons/audit-policy.yaml
  
  

I tried also:

• --feature-gates=Auditing=true
• --feature-gates=AdvancedAuditing=true

It does not matter how many CPUs or memory I give, minikube does not start in any case :(

The full output of the command that failed:
Please see all attached files

The output of the minikube logs command:
No output because minikube does not start.

The operating system version:
MacBook Pro - macOS Mojave 10.14.5
Minikube v1.1.1

I already had a look at the issue #1609, but it doesn't helped :(

I attached all attempts I did.
If someone has ideas or can help it would be great!
Thanks guys!!

attempt-1.log
attempt-2.log
attempt-3.log
attempt-4.log
attempt-5.log
attempt-6.log
attempt-7.log
attempt-8.log
attempt-9.log
attempt-10.log

[tstromberg]

Our documentation isn't great here, but the output is fairly specific about what the problem is in your various attempts. I only checked 3, but:

❌  Problems detected in "kube-apiserver":
    error: unknown flag: --Audit.LogOptions.Path

    error: unknown flag: --audit-path

    error: unknown flag: --audit-max-age

https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/ has the list of known good apiserver flags.

If you don't mind telling me which of the supported apiserver flags you are trying to set, I can help you translate it into the appropriate minikube command-line.

[tstromberg]

It's also worth noting that the documentation probably reflects Kubernetes v1.15, so you may want to upgrade to minikube v1.2 first.

[bygui86]

@tstromberg thank you very much for your answer.
I did this test before the release of Kubernetes 1.15 and Minikube 1.2. Now I upgraded already both.

This is the list of flags I want to enable:
--audit-dynamic-configuration
--audit-policy-file
--audit-log-path
--audit-log-mode
--audit-log-maxage
--audit-log-maxbackup
--audit-log-maxsize
--feature-gates
DynamicAuditing=true

The final goal is to enable auditing to run some security checks, for example using JQ filter for audit-events at the beginning and later on Falco for production.

[tstromberg]

Try this on:

minikube start \
  --extra-config=apiserver.audit-dynamic-configuration=true \
  --extra-config=apiserver.audit-policy-file=/some-path \
  --extra-config=apiserver.audit-log-path=- \
  --extra-config=apiserver.audit-log-mode=batch \
  --extra-config=apiserver.audit-log-maxage=5 \
  --extra-config=apiserver.audit-log-maxbackup=5 \
  --feature-gates=DynamicAuditing=true

Two notes:

• You'll need to provide a policy file. This example assumes that you've added one to ~/.minikube/files/some-path (which gets copied into the VM).
• If you have to wait >30 seconds for Kubernetes to start up, run this in another window to show what is almost certainly a flag issue:
  minikube logs --problems

[tstromberg]

If you find a command that works for you - please share it!

[bygui86]

Great thanks! I will try asap and keep you posted!

[bygui86]

After some tests, I have some findings...

These flags seems to be correct:

--extra-config=apiserver.audit-dynamic-configuration=true
--extra-config=apiserver.audit-log-path=-
--extra-config=apiserver.audit-log-mode=batch
--extra-config=apiserver.audit-log-maxage=5
--extra-config=apiserver.audit-log-maxbackup=5
--feature-gates=DynamicAuditing=true

but I can't be sure because I don't know how to verify and where to find logs... But at least Minikube starts properly.

This flag is not correct or the path is not correct:

--extra-config=apiserver.audit-policy-file=/some-path

Output of minikube logs --problems:

❌  Problems detected in "kube-addon-manager":
    error: unable to recognize "STDIN": Get https://localhost:8443/api?timeout=32s: dial tcp 127.0.0.1:8443: connect: connection refused
❌  Problems detected in "kube-apiserver":
    error: loading audit policy file: failed to read file path "/audit-policy/audit-policy.yaml": open /audit-policy/audit-policy.yaml: no such file or directory

I put the attached file (audit-policy.yaml) in the folder ~/.minikube/files/audit-policy/

@tstromberg what do you think?

[bygui86]

Sorry @tstromberg, any news about this flag?
Do you have the same issue?

[brichins]

For what it's worth, I think you'd have to put the audit-policy.yaml file in a minikube directory that gets mounted by kubeadm to apiserver. I'm guessing you'd need to access the kubeadm extraVolumes config (https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1).

[brichins]

For what it's worth, I was able to create a policy file as ~/.minikube/files/etc/ssl/certs/audit-policy.yaml since /etc/ssl/certs is already mounted by the apiserver container.

minikube start \
  --extra-config=apiserver.audit-policy-file=/etc/ssl/certs/audit-policy.yaml \
  --extra-config=apiserver.audit-log-path=-

[bygui86]

@brichins thanks for the tip! I will try once back from holidays :)
Meanwhile I found something like your example, but I was not able to test it yet.
Keep you posted on this!

[bygui86]

@brichins can you share the audit-policy.yaml you used? I want to try to reproduce a sample same as yours

[bygui86]

I tested and I can run same as @brichins

minikube start \
  --extra-config=apiserver.audit-policy-file=/etc/ssl/certs/audit-policy.yaml \
  --extra-config=apiserver.audit-log-path=-

But running minikube logs -f I don't see any specific auditing logs :(

[g-boros]

@bygui86 above steps worked fine with the example config from audit doc:

± % cat ~/.minikube/files/etc/ssl/certs/audit-policy.yaml
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata

I can see audit logs in apiserver's stdout:

± % k logs kube-apiserver-minikube -n kube-system | tail -n5
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"2147b766-6469-4c74-bf20-1fd7b2c52712","stage":"RequestReceived","requestURI":"/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s","verb":"update","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-controller-manager/v1.16.0 (linux/amd64) kubernetes/2bd9643/leader-election","objectRef":{"resource":"endpoints","namespace":"kube-system","name":"kube-controller-manager","apiVersion":"v1"},"requestReceivedTimestamp":"2019-09-20T12:29:15.164614Z","stageTimestamp":"2019-09-20T12:29:15.164614Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"1e20cce1-8aef-4cb9-94f9-25b8a4207db3","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s","verb":"get","user":{"username":"system:kube-scheduler","groups":["system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-scheduler/v1.16.0 (linux/amd64) kubernetes/2bd9643/leader-election","objectRef":{"resource":"endpoints","namespace":"kube-system","name":"kube-scheduler","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2019-09-20T12:29:15.154858Z","stageTimestamp":"2019-09-20T12:29:15.175613Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-scheduler\" of ClusterRole \"system:kube-scheduler\" to User \"system:kube-scheduler\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"f8df99ff-4408-4bb3-a321-ef9ca64706ed","stage":"RequestReceived","requestURI":"/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s","verb":"update","user":{"username":"system:kube-scheduler","groups":["system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-scheduler/v1.16.0 (linux/amd64) kubernetes/2bd9643/leader-election","objectRef":{"resource":"endpoints","namespace":"kube-system","name":"kube-scheduler","apiVersion":"v1"},"requestReceivedTimestamp":"2019-09-20T12:29:15.180177Z","stageTimestamp":"2019-09-20T12:29:15.180177Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"2147b766-6469-4c74-bf20-1fd7b2c52712","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s","verb":"update","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-controller-manager/v1.16.0 (linux/amd64) kubernetes/2bd9643/leader-election","objectRef":{"resource":"endpoints","namespace":"kube-system","name":"kube-controller-manager","uid":"7b6ac3ea-bbde-4c42-8438-d44a4c02558b","apiVersion":"v1","resourceVersion":"4055"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2019-09-20T12:29:15.164614Z","stageTimestamp":"2019-09-20T12:29:15.182485Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"f8df99ff-4408-4bb3-a321-ef9ca64706ed","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s","verb":"update","user":{"username":"system:kube-scheduler","groups":["system:authenticated"]},"sourceIPs":["127.0.0.1"],"userAgent":"kube-scheduler/v1.16.0 (linux/amd64) kubernetes/2bd9643/leader-election","objectRef":{"resource":"endpoints","namespace":"kube-system","name":"kube-scheduler","uid":"44fc8be7-af10-44e9-9350-54e63fa3871e","apiVersion":"v1","resourceVersion":"4056"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2019-09-20T12:29:15.180177Z","stageTimestamp":"2019-09-20T12:29:15.182903Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-scheduler\" of ClusterRole \"system:kube-scheduler\" to User \"system:kube-scheduler\""}}

[bygui86]

Thanks @g-boros!! I will git it a try asap.

[tstromberg]

Does someone mind writing this up a tutorial? Depending on how it's written, I'd love to see it added to either

https://minikube.sigs.k8s.io/docs/tutorials/ or https://minikube.sigs.k8s.io/docs/reference/

[bygui86]

@g-boros I tested it as well and it works also on my side!

@tstromberg good idea!

[tstromberg]

Closing as a workaround was found.

@bygui86 - if you write a tutorial, I would be happy to review it for addition to our docs =)

[tstromberg]

Re-opening as the issue of this not being documented has not been resolved yet.

[medyagh]

This is a good first issue, to add a tutorial on minikube website, how to enable minikube auditing and maybe add a simple example of audting usage

I will be happy to review the PR for it

[bygui86]

Sorry @tstromberg :( too busy to write down the documentation...
Many thanks to @JanAhrens to have made it!

[bygui86]

The documentation of @JanAhrens is really good! Precise, concise and clear.
Great!

[JanAhrens]

Thanks @bygui86 and everyone who contributed. I found the steps in this issue very helpful.

[joelhoisko]

I've tried following the tutorial with the audit-policy.yaml workaraound file with this command :

minikube start \                                   
  --extra-config=apiserver.audit-policy-file=/etc/ssl/certs/audit-policy.yaml \
  --extra-config=apiserver.audit-log-path=-                               
😄  minikube v1.14.1 on Ubuntu 20.04
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.19.2 on Docker 19.03.8 ...
    ▪ apiserver.audit-policy-file=/etc/ssl/certs/audit-policy.yaml
    ▪ apiserver.audit-log-path=/var/log/audit.log
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" by default

but I just can't seem to get it to work on my machine. Running

kubectl logs kube-apiserver-minikube -n kube-system | grep audit.k8s.io/v1

Returns nothing, while the normal logs still keep looking that look like this with no extra loging:

...
I1211 10:25:26.393340       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I1211 10:26:03.066193       1 client.go:360] parsed scheme: "passthrough"
I1211 10:26:03.066233       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379  <nil> 0 <nil>}] <nil> <nil>}
I1211 10:26:03.066241       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
...

I've also tried to add the --feature-gates=AdvancedAuditing=true and that gives me some extra Trace logs but still nothing regarding audit.k8s. I've also tried to set the --extra-config=apiserver.audit-log-path=/var/log/audit.log but that file never appeared. I can find the audit-policy.yaml file on the minikube but it doesn't seem to be doing anything.

Any leads? Do I need to enable some addons or something for this to work?