Okd: the cluster logging cannot be deployed

[kai-uwe-rommel]

Describe the bug
After installing a new cluster, I try to deploy cluster logging as documented here:
https://docs.okd.io/latest/logging/cluster-logging-deploying.html
First, the necessary catalog sources are missing.
I could manually install these into the cluster with sources from OCP.
BTW, is this legal? But the Red Hat operator for cluster logging is open source ...
I can then deploy cluster logging, but the fluentd containers do not start.
I get strange messages there:

  Warning  Failed     68s   kubelet, master-03.okd-demo.ars.de  Error: container create failed: time="2020-06-17T13:49:18Z" level=warning msg="exit status 1"
time="2020-06-17T13:49:18Z" level=error msg="container_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"rootfs_linux.go:58: mounting \\\\\\\"/etc/localtime\\\\\\\" to rootfs \\\\\\\"/var/lib/containers/storage/overlay/1f9ba97c268bbd000e5844b46b52a29f8a199a931cb1d6ab3457c385a31a31b4/merged\\\\\\\" at \\\\\\\"/var/lib/containers/storage/overlay/1f9ba97c268bbd000e5844b46b52a29f8a199a931cb1d6ab3457c385a31a31b4/merged/usr/share/zoneinfo/UTC\\\\\\\" caused \\\\\\\"not a directory\\\\\\\"\\\"\""
container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/etc/localtime\\\" to rootfs \\\"/var/lib/containers/storage/overlay/1f9ba97c268bbd000e5844b46b52a29f8a199a931cb1d6ab3457c385a31a31b4/merged\\\" at \\\"/var/lib/containers/storage/overlay/1f9ba97c268bbd000e5844b46b52a29f8a199a931cb1d6ab3457c385a31a31b4/merged/usr/share/zoneinfo/UTC\\\" caused \\\"not a directory\\\"\""

Version
OKD 4.4-beta5

How reproducible
100%

Log bundle
Let me know if you need something.

[vrutkovs]

If the non-core operator is failing the bug should be filed against the respective operator

[kai-uwe-rommel]

@vrutkovs : the failing operator may be a subsequent fault.
The original issue is that the catalog sources are missing at all that enable "cluster logging" to be deployed.
The deployment of cluster logging is documented in the official OKD documentation.
See URL above.
If that is not working, it is an issue of OKD and not of the operator.

I had just reported that I tried to workaround the problem until OKD fixes it!
But with my workaround I did not get completely through.
And I described this subsequent fault here for completeness and in hope that someone has an idea.
Also, since the SAME operator works fine on Openshift 4.4, I would expect it should work on OKD 4.4 as well.
If not, it might be a sign of some incompatibility which again would be an issue of OKD.

So please don't toss it away too quickly. :-)

[vrutkovs]

The deployment of cluster logging is documented in the official OKD documentation.

Correct, but its an optional operator failing, not OKD-deployed ones, so it needs to be fixed on cluster-logging side first.

Also, since the SAME operator works fine on Openshift 4.4, I would expect it should work on OKD 4.4 as well.

OKD uses a different base - FCOS, while cluster-logging assumes its RHCOS. Cluster logging operator needs to take both into account

[kai-uwe-rommel]

Again: you may be right about the fluentd sub-issue.

However, please also check the catalogsource main issue!
This is something that can only be fixed in OKD.

[vrutkovs]

What "necessary catalog sources" are missing? Those which are used in OCP require pull-secret from cloud.redhat.com, so it can't be used out of the box in OKD

[kai-uwe-rommel]

The cluster logging comes from one of the three more catalog sources that OCP includes and OKD doesn't. I can't tell you exactly from which one it comes.
At least the OKD docs need to document if and how you can add those if you actually want (or need) cluster logging.
Otherwise it makes no sense to document cluster logging in the OKD docs.
And it would be a pity, if cluster logging would not be available for/with OKD.
Sure you can deply ELK/EFK in other ways. But it would be different ...

On the other hand, as far as I can see, the Red Hat cluster logging operator IS open source (it is available as such on Github). So the question is, why it is not rather available via the community catalog source? This is something that Red Hat would need to change. Who has the right power to get Red Hat moving that way?

OKD has:

C:\>oc get catalogsource -n openshift-marketplace
NAME                  DISPLAY               TYPE   PUBLISHER   AGE  
community-operators   Community Operators   grpc   Red Hat     15d  

while OCP has:

NAME                  DISPLAY               TYPE   PUBLISHER   AGE 
certified-operators   Certified Operators   grpc   Red Hat     27h 
community-operators   Community Operators   grpc   Red Hat     27h 
redhat-marketplace    Red Hat Marketplace   grpc   Red Hat     27h 
redhat-operators      Red Hat Operators     grpc   Red Hat     27h 

[kai-uwe-rommel]

I looked it up: cluster-logging comes from the redhat-operators catalog.

[kai-uwe-rommel]

... and I also found the fix for the fluentd failure. Thanks for mentioning FCOS vhs. RHCOS.
The fix is simple:
rmdir /etc/localtime ; ln -s /usr/share/zoneinfo/UTC /etc/localtime
To be done in the node's CoreOS.

(For some reason, after installing OKD on FCOS, this empty /etc/localtime directory exists. On plain installed FCOS it does not exist.)

[evertmulder]

Did you manage to get the cluster-logging working?

[kai-uwe-rommel]

Yes. With these two mentioned fixes it works

• add the redhat-operators catalog
• fix /etc/locatime in the underlying FCOS

[kai-uwe-rommel]

Actually, that is the same issue:
https://github.com/openshift/okd/issues/247

[foosinn]

fix this without logging into each node:

```
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: localtime-fixer
namespace: openshift-logging
annotations:
openshift.io/scc: node-exporter
labels:
app: localtime-fixer
spec:
selector:
matchLabels:
app: localtime-fixer
template:
metadata:
annotations:
sheduler.alpha.kubernetes.io/critical-pod: ""
labels:
app: localtime-fixer
spec:
containers:
- name: fixer
image: alpine:3.12
command:
- /bin/ash
- -c
- |
if [ -d /host/etc/localtime ]; then
rmdir /host/etc/localtime
fi
if [ ! -h /host/etc/localtime ]; then
ln -fvs /usr/share/zoneinfo/UTC /host/etc/localtime
fi
ls -lah /host/etc/localtime
exec /bin/sleep infinity
securityContext:
privileged: true
runAsUser: 0
volumeMounts:
- name: etc
mountPath: /host/etc
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- key: node.kubernetes.io/disk-pressure
operator: Exists
effect: NoSchedule
nodeSelector:
kubernetes.io/os: linux
volumes:
- name: etc
hostPath:
path: /etc
type: Directory