Rh-che: Apache Camel example not working on production

[Katka92]

Issue problem:
When running Apache Camel based on Spring Boot on production, the workspace can not start. This issue is not reproducible on prod-preview. Devfiles are same on production and prod-preview.

Red Hat Che version:

version: 7.1.0

Reproduction Steps:
Create a default Apache Camel workspace with default settings and with example project.

Runtime:

runtime used:

• [ ] minishift (include output of minishift version)
• [x] OpenShift.io
• [ ] Openshift Container Platform (include output of oc version)

[ibuziuk]

@Katka92 could you please add namespaces example where the same devfile works and where it is not, since this looks like admission controller setup issue:

Pod "workspacezech2cscku1nu8nb.maven-bd4d75d56-v2l5s" is invalid: spec.containers[3].resources.requests:
Invalid value: "150Mi": must be less than or equal to memory limit

[Katka92]

@ibuziuk Sure :)
Fails for kkanova cluster 2 and osio-ci-e2e-004 cluster 1b
Succeeds for osiotest2 on cluster 2 and osiotest-workspace-1b on cluster 1b

[RickJWagner]

I can replicate this on che.openshift.io, asking for a Camel/SpringBoot application.

There are other users (i.e. external) that are hitting this problem also.

[l0rd]

On osio containers cannot have memory limit < 150MB. For instance that doesn't work.

[ibuziuk]

@l0rd the funny thingy that this works in some namespaces
@pbergene @jfchevrette looks like that some of the admission controllers are not applied across all the *-namespaces

[amisevsk]

FWIW from some discussion earlier, it seems that not all namespaces have the same minimum memory limit. Some have a minimum of 32Mi, others have 150Mi.

The error message is deceptive, since I think there's an admission controller that rewrites requests to be the minimum, which causes the issue shown. See https://coreos.slack.com/archives/CCRND57FW/p1568383360487000 for details.

[l0rd]

Anyway the "easy fix" here is to change the limit to 150MB in the camel devfile. And since we should rollout the 7.2 release soon we may have that in prod by the end of the week.

[ibuziuk]

hmm... today the Apache Camel worked for me @Katka92 can you reproduce with the kkanova against cluster 2 ?

[Katka92]

For user kkanova still failing. You can see oc events here: https://gist.github.com/Katka92/8f4268790766e3680bc8c4369741812f

[ibuziuk]

The root cause is that https://github.com/fabric8-services/fabric8-tenant/pull/727 was not applied for all namespaces and we do need to proceed with global tenant-update in order to apply the same quotas. For some reason the quotas were applied not to all namespaces for example:

• ibuziuk-che (us-east-2) - Limit is 30Mi

• kkanova-che (us-east-2) - Limit is 150Mi
  

@alexeykazakov how could we request a cluster wide tenant update in order to apply the same quotas for all namespaces? Looks like the config is different for different namespaces

In the meantime, I will provide a PR with a workaround on devfile end

[ibuziuk]

PR with a workaround - https://github.com/eclipse/che-devfile-registry/pull/101
@Katka92 could you please try it on Hosted Che with kkanova account and provide your approval ?

[alexeykazakov]

@alexeykazakov how could we request a cluster wide tenant update in order to apply the same quotas for all namespaces? Looks like the config is different for different namespaces

It would be best if you create a JIRA for that in https://jira.coreos.com/browse/CRT
cc: @MatousJobanek

[ibuziuk]

@alexeykazakov @MatousJobanek issue created - https://jira.coreos.com/browse/CRT-270
Current rh-che issue is going to be closed once Hosted Che is updated to the latest 7.2.0 release

[alexeykazakov]

Current rh-che issue is going to be closed once Hosted Che is updated to the latest 7.2.0 release

When is it going to happen?

[ibuziuk]

@alexeykazakov early October, but we already applied workaround on devfile end

[MatousJobanek]

@ibuziuk @Katka92 The cluster-wide update should be finished, could you please check that everything is set correctly now?

[Katka92]

@MatousJobanek Seem like the quota was changed on my account kkanova [1] and I was able to start Apache Camel workspace correctly. The same for osio-ci-e2e-004.

[1]

[ibuziuk]

@MatousJobanek great job. Closing