Crc: [BUG] In CRC 1.17.0, the node tuning operator is not running, vm.max_map_count tunable is not applied

[syedriko]

Please see https://github.com/openshift/elasticsearch-operator/issues/527 for background.

General information

• OS: Linux / macOS / Windows
  Fedora 32
• Hypervisor: KVM / Hyper-V / hyperkit
  KVM
• Did you run crc setup before starting it (Yes/No)?
  Yes
• Running CRC on: Laptop / Baremetal-Server / VM
  Laptro

CRC version

[localhost crc-linux-1.17.0-amd64]$ ./crc version
CodeReady Containers version: 1.17.0+99f5c87
OpenShift version: 4.5.14 (embedded in binary)
[localhost crc-linux-1.17.0-amd64]$ 

CRC status

[localhost crc-linux-1.17.0-amd64]$ ./crc status --log-level debug
DEBU CodeReady Containers version: 1.17.0+99f5c87 
DEBU OpenShift version: 4.5.14 (embedded in binary) 
DEBU About to run SSH command with hidden output  
DEBU SSH command succeeded                        
DEBU About to run SSH command:
df -B1 --output=size,used,target /sysroot | tail -1 
DEBU SSH command results: err: <nil>, output: 32720793600 12807503872 /sysroot 
CRC VM:          Running
OpenShift:       Running (v4.5.14)
Disk Usage:      12.81GB of 32.72GB (Inside the CRC VM)
Cache Usage:     13.04GB
Cache Directory: /home/syedriko/.crc/cache
[localhost crc-linux-1.17.0-amd64]$ 

CRC config

[localhost crc-linux-1.17.0-amd64]$ ./crc config view
- cpus                                  : 6
- memory                                : 12288
- pull-secret-file                      : /home/syedriko/crc/pull-secret
- skip-check-libvirt-running            : true
[localhost crc-linux-1.17.0-amd64]$ 

Host Operating System

[localhost crc-linux-1.17.0-amd64]$ cat /etc/os-release
NAME=Fedora
VERSION="32 (Workstation Edition)"
ID=fedora
VERSION_ID=32
VERSION_CODENAME=""
PLATFORM_ID="platform:f32"
PRETTY_NAME="Fedora 32 (Workstation Edition)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:32"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f32/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=32
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=32
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation
[localhost crc-linux-1.17.0-amd64]$ 

Steps to reproduce

1. crc setup; crc start
2. Check vm.max_map_count on the CRC VM:

[localhost crc-linux-1.16.0-amd64]$ oc debug node/crc-zqfk6-master-0
Starting pod/crc-zqfk6-master-0-debug ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.126.11
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# cat /proc/sys/vm/max_map_count
65530

1. Check if the node tuning operator is running (it's not):

[localhost crc-linux-1.17.0-amd64]$ oc get pods -n openshift-cluster-node-tuning-operator
No resources found in openshift-cluster-node-tuning-operator namespace.
[localhost crc-linux-1.17.0-amd64]$ 

1. Expected

The node tunining operator is running and vm.max_map_count == 262144 on the CRC VM

# this is from CRC 1.16.0:
Using project "openshift-cluster-node-tuning-operator".
[localhost crc-linux-1.16.0-amd64]$ oc get all
NAME                                               READY   STATUS    RESTARTS   AGE
pod/cluster-node-tuning-operator-576df548b-jmz8z   1/1     Running   0          26d
pod/tuned-d5frc                                    1/1     Running   0          26d

Actual

The node tuning operator is not running and the vm.max_map_count == 65530 on the CRC VM.

Logs

crc setup; crc start --log-level debug;
https://gist.github.com/syedriko/83a155195dad64689235f0c138c51017

~/.crc/crc.log
https://gist.github.com/syedriko/8b2a974b307472dececeecf4b480e3e5

[guillaumerose]

The node-tuning-operator was removed in the last release (1.17.0). We thought it was not necessary to have it running all the time in the VM. See https://github.com/code-ready/snc/pull/240

I guess we need to revert this or at least ensure it runs once. WDYT @praveenkumar ?

[gbraad]

Is it not something that can be enabled 'on-demand' ?
I do not believe it is an esential part of OpenShift to function

[huikang]

@syedriko , a quick fix would be to add some text about the solution at
https://github.com/openshift/cluster-logging-operator/blob/master/docs/HACKING.md#setting-up-a-test-cluster
to gives users of crc a heads-up.

[guillaumerose]

Elastic Operator should be able to run without any tweaks in CRC. It's a pretty common to use it.
As a developer, I always end up adding it to my project.

[cfergeau]

Got a similar question on IRC by someone who had to use an older crc release because of this (he also needs Elastic Search in order to use the OpenShift Telemetry Framework).

[guillaumerose]

We reverted the change. It will be fixed in the next release.

[leifmadsen]

This bit me today too :)

Are there instructions to turn it on in the meantime?

[syedriko]

This bit me today too :)

Are there instructions to turn it on in the meantime?

Yep, https://github.com/openshift/elasticsearch-operator/issues/527#issuecomment-706484497

[leifmadsen]

Question... this was reverted in the next release (1.18.0?) but that release is OCP 4.6 based correct?

[guillaumerose]

1.18.0 and 1.19.0 are based on OCP 4.6, yes. You need it for 4.5?

[leifmadsen]

1.18.0 and 1.19.0 are based on OCP 4.6, yes. You need it for 4.5?

Unfortunately yes. Some dependencies that we rely on for testing are not available in OCP 4.6 (as of now) so all our testing is with 4.5, and this bug bites us.

[cfergeau]

Can you use the workaround from https://github.com/code-ready/crc/issues/1590#issuecomment-708561285 in the mean time? You probably can also use oc edit clusterversion version and remove

 - kind: Deployment
    group: apps/v1
    name: cluster-node-tuning-operator
    namespace: openshift-cluster-node-tuning-operator
    unmanaged: true

[leifmadsen]

Can you use the workaround from #1590 (comment) in the mean time? You probably can also use oc edit clusterversion version and remove

 - kind: Deployment
    group: apps/v1
    name: cluster-node-tuning-operator
    namespace: openshift-cluster-node-tuning-operator
    unmanaged: true

Ah right I forgot about the workaround when I responded. Also thanks for the other idea.

Yea I think these workarounds are satisfactory. I'd close this issue out as Resolved.