Kops: cant see audit logs

[shavo007]

1. What kops version are you running? The command kops version, will display
   this information.
   kops 1.8.0

2. What Kubernetes version are you running? kubectl version will print the
   version if a cluster is running or provide the Kubernetes version specified as
   a kops flag.

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:23:29Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

1. What cloud provider are you using?
   AWS
2. What commands did you run? What is the simplest way to reproduce this issue?
   updated kops with audit config

spec:
  kubeAPIServer:
    auditLogPath: /var/log/kube-apiserver-audit.log
    auditLogMaxAge: 10
    auditLogMaxBackups: 1
    auditLogMaxSize: 100

Then restarted my master

kops get instancegroups
 kops rolling-update cluster  --instance-group <masterName> --force --yes

1. What happened after the commands executed?
   rolling update of cluster but dont see the audit logs

2. What did you expect to happen?
   ssh into master and see log file for audit

3. Please provide your cluster manifest. Execute
   kops get --name my.example.com -oyaml to display your cluster manifest.
   You may want to remove your cluster name and other sensitive information.

4. Please run the commands with most verbose logging by adding the -v 10 flag.
   Paste the logs into this report, or in a gist and provide the gist link here.

5. Anything else do we need to know?
   Kube api server docker command

          "/usr/local/bin/kube-apiserver --address=127.0.0.1 --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,Priority,ResourceQuota --allow-privileged=true --anonymous-auth=false --apiserver-count=1 --audit-log-maxage=10 --audit-log-maxbackup=1 --audit-log-maxsize=100 --audit-log-path=/var/log/kube-apiserver-audit.log --authorization-mode=AlwaysAllow --basic-auth-file=/srv/kubernetes/basic_auth.csv --client-ca-file=/srv/kubernetes/ca.crt --cloud-provider=aws --etcd-servers-overrides=/events#http://127.0.0.1:4002 --etcd-servers=http://127.0.0.1:4001 --insecure-port=8080 --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP --proxy-client-cert-file=/srv/kubernetes/apiserver-aggregator.cert --proxy-client-key-file=/srv/kubernetes/apiserver-aggregator.key --requestheader-allowed-names=aggregator --requestheader-client-ca-file=/srv/kubernetes/apiserver-aggregator-ca.cert --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --runtime-config=batch/v2alpha1=true --secure-port=443 --service-cluster-ip-range=xxx/13 --storage-backend=etcd2 --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key --token-auth-file=/srv/kubernetes/known_tokens.csv --v=2 2>&1 | /bin/tee -a /var/log/kube-apiserver.log"

Thanks,
Shane.

[KashifSaadat]

Hey @shavo007, looking at #4202 did you solve this in the end?

[shavo007]

Nah @kashifsaadat, I initially was using the wrong version of kops on my cluster.

I added in the config above and i don't see the log file on the machine.

[shavo007]

Actually checking doc it states You can pass a file with the policy to kube-apiserver using the --audit-policy-file flag. If the flag is omitted, no events are logged."

Is that true?

I thought that was for advanced auditing and without policy file it would use default?

[shavo007]

"Note: you could use the fileAssets feature to push an advanced audit policy file on the master nodes."

Cant find anywhere how to use this fileassets feature.

Any examples?

Also the doc is wrong if it says could. Sounds like you need a policy file or auditing doesn't work.

[KashifSaadat]

Hi @shavo007, there's some docs here on defining FileAssets within your ClusterSpec: https://github.com/kubernetes/kops/blob/master/docs/cluster_spec.md#fileassets

For example:

spec:
  fileAssets:
  - name: apiserver-audit-policy
    path: /srv/kubernetes/audit.yaml
    roles: [Master]
    content: |
      apiVersion: audit.k8s.io/v1beta1 # This is required.
      kind: Policy
      # Don't generate audit events for all requests in RequestReceived stage.
      omitStages:
        - "RequestReceived"
      rules:
        # Log pod changes at RequestResponse level
        - level: RequestResponse
          resources:
          - group: ""
            # Resource "pods" doesn't match requests to any subresource of pods,
            # which is consistent with the RBAC policy.
            resources: ["pods"]
        # Log "pods/log", "pods/status" at Metadata level
        - level: Metadata
          resources:
          - group: ""
            resources: ["pods/log", "pods/status"]

A contribution is welcome to update the documentation! :)

[shavo007]

Can you answer the questions above please? Is the documentation up to date?

[KashifSaadat]

The documentation does appear to need updating to say that specifying an audit policy file is required. A PR would be welcome if you're able to, just need to add a note under the kubeAPIServer - audit logging section and could reference this example file: https://raw.githubusercontent.com/kubernetes/website/master/docs/tasks/debug-application-cluster/audit-policy.yaml

[shavo007]

no worries @KashifSaadat ill look to do that next week and test it out

[shavo007]

all good, got it working.

ill create a PR this week to update the doc.

config is

  kubeAPIServer:
    auditLogMaxAge: 10
    auditLogMaxBackups: 1
    auditLogMaxSize: 100
    auditLogPath: /var/log/kube-apiserver-audit.log
    auditPolicyFile: /srv/kubernetes/audit.yaml

file asset:

  fileAssets:
  - content: |
      apiVersion: audit.k8s.io/v1beta1 # This is required.
      kind: Policy
      # Don't generate audit events for all requests in RequestReceived stage.
      omitStages:
        - "RequestReceived"
      rules:
        # Log pod changes at RequestResponse level
        - level: RequestResponse
          resources:
          - group: ""
            # Resource "pods" doesn't match requests to any subresource of pods,
            # which is consistent with the RBAC policy.
            resources: ["pods"]
        # Log "pods/log", "pods/status" at Metadata level
        - level: Metadata
          resources:
          - group: ""
            resources: ["pods/log", "pods/status"]
    name: apiserver-audit-policy
    path: /srv/kubernetes/audit.yaml

[KashifSaadat]

Great to hear, also thanks for the PR much appreciated! 👍

I'll close this issue, feel free to re-open if you are still having any issues with this.

/assign
/close