Okd: OKD creates `/etc/localtime` directory, breaking fluentd

[eitane]

Describe the bug
After installing a new cluster on AWS, I have deployed cluster logging as documented here:
https://docs.okd.io/latest/logging/cluster-logging-deploying.html
but Fluentd pods won't start on the nodes with the following error:

(combined from similar events): Error: container create failed: time="2020-08-23T07:12:57Z" level=warning msg="exit status 1" time="2020-08-23T07:12:57Z" level=error msg="container_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"rootfs_linux.go:58: mounting \\\\\\\"/etc/localtime\\\\\\\" to rootfs \\\\\\\"/var/lib/containers/storage/overlay/14d76c25e910358b9ed7acffb22140718e5827fddeb3f14b9b7c81808c6b250f/merged\\\\\\\" at \\\\\\\"/var/lib/containers/storage/overlay/14d76c25e910358b9ed7acffb22140718e5827fddeb3f14b9b7c81808c6b250f/merged/usr/share/zoneinfo/UTC\\\\\\\" caused \\\\\\\"not a directory\\\\\\\"\\\"\"" container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/etc/localtime\\\" to rootfs \\\"/var/lib/containers/storage/overlay/14d76c25e910358b9ed7acffb22140718e5827fddeb3f14b9b7c81808c6b250f/merged\\\" at \\\"/var/lib/containers/storage/overlay/14d76c25e910358b9ed7acffb22140718e5827fddeb3f14b9b7c81808c6b250f/merged/usr/share/zoneinfo/UTC\\\" caused \\\"not a directory\\\"\""

I've checked the worker and master nodes and made sure that /etc/localtime exists on the hosts path.

This is similar to #220 but the workaround found in #220 is not applicable.
Version
OKD 4.5.0-0.okd-2020-08-12-020541

How reproducible
100% reproducible

Log bundle
Please let me know if you need any further issue.

[pflaeging]

I've got the same problem. Dirty hack for now:

ssh core@node1 "sudo rm -rf /etc/localtime; sudo ln -s /usr/share/zoneinfo/UTC /etc/localtime"

And this for every node
Please don't kill me, just testing ;-)

[vrutkovs]

That's weird, seems every node has empty /etc/localtime (but its not present in plain FCOS).

Most likely its an artifact from machine-os-content build

[eitane]

rm -rf /etc/localtime; sudo ln -s /usr/share/zoneinfo/UTC /etc/localtime

Doesn't work for me since /etc/localtime is already a link to /usr/share/zoneinfo/UTC and not an empty directory.
ls -l /etc/localtime lrwxrwxrwx. 1 root root 25 Jun 16 04:17 /etc/localtime -> ../usr/share/zoneinfo/UTC

Maybe there is a difference in the image that is used when running on AWS or most likely a permission issue?

[pflaeging]

I've got the same problem. Dirty hack for now:

ssh core@node1 "sudo rm -rf /etc/localtime; sudo ln -s /usr/share/zoneinfo/UTC /etc/localtime"

Sorry, I forget to mention: I'm on bare metal and my machines are reporting to run:
Fedora CoreOS 32.20200629.3.0

[foosinn]

To fix this without logging into each node:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: localtime-fixer
  namespace: openshift-logging
  annotations:
    openshift.io/scc: node-exporter
  labels:
    app: localtime-fixer
spec:
  selector:
    matchLabels:
      app: localtime-fixer
  template:
    metadata:
      annotations:
        sheduler.alpha.kubernetes.io/critical-pod: ""
      labels:
        app: localtime-fixer
    spec:
      containers:
        - name: fixer
          image: alpine:3.12
          command:
            - /bin/ash
            - -c
            - |
              if [ -d /host/etc/localtime ]; then
                rmdir /host/etc/localtime
              fi
              if [ ! -h /host/etc/localtime ]; then
                ln -fvs /usr/share/zoneinfo/UTC /host/etc/localtime
              fi
              ls -lah /host/etc/localtime
              exec /bin/sleep infinity
          securityContext:
            privileged: true
            runAsUser: 0
          volumeMounts:
            - name: etc
              mountPath: /host/etc
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
        - key: node.kubernetes.io/disk-pressure
          operator: Exists
          effect: NoSchedule
      nodeSelector:
        kubernetes.io/os: linux
      volumes:
        - name: etc
          hostPath:
            path: /etc
            type: Directory

[JoaoBraveCoding]

If I'm not mistaken this was supposed to have been fixed with the release 4.5.0-0.okd-2020-10-03 but we have tried it and there is still no file on /etc/localtime. For our FCOS version we are using 32.20201004.3.0 and as a current workaround we are using MachineConfigs to create the symlink similar to the other solutions shown in this issue.

[vrutkovs]

Should be fixed in OKD 4.6, could you give that a try?

[timbrd]

Should be fixed in OKD 4.6, could you give that a try?

Still not working on a fresh installation of okd 4.6:

$ oc get events
[...]
16m         Warning   Failed                         pod/fluentd-m2sz7                                                  Error: container create failed: time="2020-12-11T17:14:20Z" level=warning msg="exit status 1"
time="2020-12-11T17:14:20Z" level=error msg="container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting \"/etc/localtime\" to rootfs at \"/var/lib/containers/storage/overlay/1239a41eae7642c5daf71374a6ed861dba5c70411f52a21952b58ede80d84765/merged/usr/share/zoneinfo/Etc/UTC\" caused: not a directory"
16m         Warning   Failed                         pod/fluentd-m2sz7                                                  Error: container create failed: time="2020-12-11T17:14:32Z" level=warning msg="exit status 1"
time="2020-12-11T17:14:32Z" level=error msg="container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting \"/etc/localtime\" to rootfs at \"/var/lib/containers/storage/overlay/97342b184cde634f0ac71525d650d236281d24bfa08de3a084456c3a148779bd/merged/usr/share/zoneinfo/Etc/UTC\" caused: not a directory"
[...]

$ oc version
Client Version: 4.6.0-0.okd-2020-11-27-200126
Kubernetes Version: v1.19.0-rc.2.1077+43983cda8af930-dirty

$ cat /etc/os-release
[...]
33.20201124.10.1 (CoreOS)
[...]

[vrutkovs]

Oh snap, it was reverted during okd-machine-os rework. I'll bring it back soon

[lmbss]

@JoaoBraveCoding you can share the machineconfig workaround?

[JoaoBraveCoding]

@JoaoBraveCoding you can share the machineconfig workaround?

@lmbss Sure!

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: master-localtime-patch
spec:
  config:
    ignition:
      version: 3.0.0
    systemd:
      units:
        - contents: |
            [Unit]
            Description=Creation of localtime
            Before=kubelet.service

            [Service]
            # Fix nodes that already tried to run fluentd pods
            ExecStart=bash -c 'test -d /etc/localtime && rmdir /etc/localtime || true'
            # Create symlink if not present
            ExecStart=bash -c 'test -e /etc/localtime && true || ln -s ../usr/share/zoneinfo/UTC /etc/localtime'
            Type=oneshot

            [Install]
            WantedBy=multi-user.target
          enabled: true
          name: localtime.service

[lmbss]

@JoaoBraveCoding thank you

[vrutkovs]

Fixed in https://github.com/openshift/okd-machine-os/pull/60

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale