Che: Implement Airgapped Configuration Mechanism

[tsmaeder]

This is about designing and implementing one or more mechanisms for configuring build systems int he "air-gapped" case. We need to solve two distinct problems:

• Override the settings from the source code (e.g. pom.xml) and default settings (e.g. maven central)
• Make tools accept self-signed certificates for secure connections (e.g. https).

There are various desirable properties for any solution:

• The solution should work for commands define in the devfile as well as using regular tools in the workspace (e.g. "mvn clean install" from the command line should work)
  
  
  
  • Ideally, an administrator could override settings in existing devfiles per che installation via some mechanism (I believe there is a cr.yaml file in CRW)
  
  
  • If there are relevant settings in the devfile, they should take precedence over those set per installation
  
  
  • If the user makes changes to any of the settings (for example, by editing a settings.xml in ~/.m2), they take precedence over system/devfile settings
  
  

[tsmaeder]

We discussed using init containers for transforming configuration values into files on in user.home. How exactly this would work needs to be designed.

[slemeur]

@tsmaeder Before going into the implementation of new mechanisms, we recently merged https://github.com/eclipse/che/pull/15435 which should allow to override settings.xml file more easily.
Documentation on how to achieve that and leverage the existing work should be step 0

[tsmaeder]

@slemeur not really...it's a prerequisite, but we still need mechanisms for creating those files. Environment variables cannot be used in settings.xml

[tsmaeder]

The idea is to collect the various ways we can configure language support for airgapped for different tools. Once we have that, we can design a proper config mechanism.

[l0rd]

@tsmaeder what about using an init container as described in https://github.com/eclipse/che/issues/14857#issuecomment-559078390. It would be one distinct init container for every dependency manager we support but the mechanism would be the same (env var + init container).

[tsmaeder]

One mechanism that seems to tick all the boxes I've come up with:

1. Add a "files" property to che components (any) in the devfile like so:

- id: redhat/java11/latest
  type: chePlugin
  files:
    -  path: ~/.m2/settings.xml
       referenceContent: "some xml here $INTERNAL_REPO_URL more xml"

1. In the container init script we read those "files" and replace the env variable references (allow for escaping if we want put env variables in the container).

2. Have a mechanism to set/override environment variables from the "resource.yml" that's use for system-wide config overrides in CRW.

[tsmaeder]

@l0rd @slemeur @skabashnyuk wdyt?

[nickboldt]

can we also have either:

       referenceContent: "some URL to the file we want injected here"

or

       referenceContentURL: "some URL to the file we want injected here"

[tsmaeder]

We can add the same configuration mechanism to the meta.yml in the plugin. Also, we could make the creation of each file depend on the presence of a particular env variable.

[tsmaeder]

We might not cover all cases and combinations, but at least we have 90% covered.

[sunix]

would part of this be covered with enabling kubernetes secrets ?
https://github.com/eclipse/che/issues/14680

[tsmaeder]

Well, the plan includes a provision for evaluating env vars in the file content, so I guess you can use stuff from there.

[nickboldt]

FYI downstream request for this feature in https://issues.redhat.com/browse/CRW-610

Something we can do in 7.9?

[tsmaeder]

To be done here:

• [ ] Write out a description of a design and discuss on che-dev list
• [ ] Implement (or document) mechanism to override repositories (for example) in global config (resource.yaml).
• [ ] Implement Syntax in devfile
• [ ] Implement Syntax in plugin meta.yaml
• [ ] Implement mechanism to write files via container startup script or init container.

[tsmaeder]

@nickboldt can you tell how the mechanism to override stuff in crw ("resource.yaml") functions or do you know who to ask?

[tsmaeder]

I've started a write up of the problem in a google doc: https://docs.google.com/document/d/1rE7MFUPnP3pq57vGM_DqyuFvlCbjYuqe-xFXgDiXSEM/edit#heading=h.3lz6pxl55uha

[slemeur]

Would be better in a document that anybody could be able to open.

On Wed, Feb 12, 2020 at 4:27 PM Thomas Mäder [e-mail]
wrote:

I've started a write up of the problem in a google doc:
https://docs.google.com/document/d/1rE7MFUPnP3pq57vGM_DqyuFvlCbjYuqe-xFXgDiXSEM/edit#heading=h.3lz6pxl55uha

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/eclipse/che/issues/15518?email_source=notifications&email_token=AAMPTIJI2Q6KNECS7LLFZQ3RCQIO5A5CNFSM4J4HDXE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELRFXLI#issuecomment-585259949,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAMPTIOX2ZHG3LYSYANTBQTRCQIO5ANCNFSM4J4HDXEQ
.

--
Stévan LeMeur // Product Manager // Developer Tools // [phone]

[tsmaeder]

I'll make it public when it's ready for general review.

[sunix]

I couldn't access to the doc,
as mentionned in https://github.com/eclipse/che/issues/14680#issuecomment-589075364

We should try to do something similar to https://docs.openshift.com/container-platform/4.3/builds/creating-build-inputs.html#builds-adding-input-secrets-configmaps_creating-build-inputs

The way I see it, is a section in the Dashboard where you could define these configmaps or secrets. We could also have configmaps or secret per organisation. and we could have a way to set these configmap or secrets through chectl.

I definitely do not see these values/files defined in the devfile. But references to secrets and configmap.
These secrets and config maps could be used by any workspaces by just referencing them.

We could have default entries for configmaps and secrets that we use in our default devfiles. These ones could be overriden by the user.

[tsmaeder]

Here's my proposal for implementing the air-gapped configuration mechanism: it consists of a general way of generating preferences, env vars and config files based on configuration values:

[tsmaeder]

Tooling Configuration in Che

This document describes an approach to configure development tools in Che (VS Code extensions, maven, go compiler, etc.). The concrete requirement for this comes from the practice of air-gapping, where (usually large) companies don’t allow developers access to artifact repositories on the internet and use curated copies in of those repositories on the company intranet instead. The processes for air-gapping with various technologies is documented in the issues hanging off Document and QE Usage of Dependencies Repositories in Airgap Scenario · Issue #15351 · eclipse/che · GitHub

Ways to configure tools

The configuration mechanisms we’ve seen seem to fall into the following categories:

• Extension preferences
  For example, the startup parameters of the Java language server can be configured via the VSCode-Java preferences. We have a mechanism in place that allows us to provide preferences in the devfile. 

• Environment variables
  As an example, the proxy for Go package resolution can be set via an environment variable. There is a mechanism to set environment variables in containers belonging to editors, plugins and other components

• Configuration files
  These are usually text files, like ~/.m2/settings.xml for maven, but also binaries like keystores. The tendency for them is to be small in size.

• Commands
  Some systems provide commands to do configuration, for example the npm repository can be set with npm-config. However, so far all those commands we’ve seen write some configuration files (e.g. .npmrc) and as such would be covered if we could write config files.

Goals

1. easy to configure (use templates provided by Che org for common use cases)

2. configure concrete values centrally per Che installation by admins (where the repositories are, etc.)

3. configuration overridable per devfile

4. Allow for use of defaults, e.g. use the default maven repositories

5. generality: should work for third-party technologies

Non-Goals

1. configuration in UI

2. configuration other than per install, i.e. per user, role

Approach

We already have a way to set environment variables in devfiles. We also can set preferences in devfiles. What is missing is the possibility to create configuration files in the appropriate places. It seems we can cover the case of commands with configuration files. So our approach will be to use environment variables and creation of configuration files in order to fulfil the goals defined above.

Case: Environment Variables

We already have a way to define environment variables in devfiles. However, we cannot set their values in terms of a central configuration done by the admins of the Che installation. To solve this, I propose these enhancements:

1. Introduce a configuration mechanism (via config maps, maybe?), that allows to define constants that will be available in every devfile on this Che installation for use as outlined below. The set of variables that can be defined must be open-ended and no values must be ‘well-known’ by the system.

2. We provide a templating mechanism for definitions of environment variable values that uses the variables from step 1. So in in a devfile we could define the following env variable for a component:

    env:  
       - name: ‘GOPROXY’  
         value: ‘${internal_proxy_url}’
   

   where ‘internal_proxy_url’ is a configuration constant.

This alone would allow us to fulfil goal 1, 2, 3 and 5, but in order to fulfil goal 4 (defaults), we need to be able to turn off the definition of the env variable:

1. We make environment variables conditional on the presence of configuration constants. Concretely, we would add an additional property (let’s call it ‘onlyIf’) to the definition of env variables in devfiles:

   env  
       - name: ‘GOPROXY’  
         value: ‘${internal_proxy_url}’  
         onlyIf: ‘internal_proxy_url’  
   

   Concretely, this means that the env variable ‘GOPROXY’ is only defined if the admin configured the constant ‘internal_proxy_url’

Case: Theia Preferences

I believe we can use the exact same approach as for environment variables. However, in order to be able to make the setting of values conditional on configuration, we’d need to align the syntax for defining preference values with the one for env variable, so instead of writing 

preferences:

  - clangd.path: /usr/bin/clangd

we’d have to write

preferences:

 - name: ‘clangd.path’

   value: ‘/usr/bin/clangd’

   onlyIf: ‘some_config_value`

Case: Config Files

Again, the approach for creating config files should largely be the same as for the env-vars and preferences. What I would propose is a syntax in the devfile that allow to describe a config file in a component. Let’s look at an example:

components:
  - id: redhat/java11/latest
    type: chePlugin
    config
      path: ‘/home/theia/.m2/settings.xml’
      referenceContent: ‘https://some.domain.com/content/userSettings.xml’
      onlyIf: ‘maven_snaphot_repository, maven_release_repository`

So this would tell the system to create a file inside the container of component redhat/java11/latest at the path /home/theia.m2/settings.xml, but only if the configuration variable maven_repo_url is set.

The content of the referenced file could look like this:

<settings>
  <profiles>
    <profile>
      <id>my-nexus</id>
      <repositories>
        <repository>
          <id>my-nexus-snapshots</id>
          <releases>
            <enabled>false</enabled>
          </releases>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
          <url>${maven_snapshot_repository}</url>
        </repository>
        <repository>
          <id>my-nexus-releases</id>
          <releases>
            <enabled>true</enabled>
          </releases>
          <snapshots>
          <enabled>false</enabled>
          </snapshots>
          <url>${maven_release_repository}</url>
        </repository>
      </repositories>
    </profile>
  </profiles>
  <activeProfiles>
    <activeProfile>my-nexus</activeProfile>
  </activeProfiles>
</settings>

Of course, in addition to supporting referenceContent with an url, we should also support a content attribute where the content of the file can be provided inline as a yaml-encoded string.

Note that instead of depending on a single config variable, this configuration depends on multiple variables. It would be allowed to have multiple definitions for the same path, with the entry that contains the most variables taking precedence. If two configuration entries with the same path match the same number of variables, it is considered an error.

File creation

File creation should be done by the system, through a mechanism like oc exec. This works in all cases. File creation through init containers, for example, would be restricted to locations that are mapped to volumes. (I believe you can only mount volumes to existing directories, as well, right?). Config files should only be created if they do not exist already. (Maybe this should be overridable in the devfile, as we may want to override stuff in plugin containers that already have config files present).

Config provided by plugins themselves

It has been suggested that plugins themselves should provide the configuration for things like air-gapping. It is true that we could provide all the same mechanisms I have outlined in meta.yaml as well as in devfiles.

However, I don’t think that is a useful feature for these reasons

1. It would not solve the problem of the dev container: The only description we have of the dev container (for example the ‘maven’ container in the built-in maven devfile) is the devfile. Therefore we would have to do the configuration as described in the chapters above in the devfile anyway. Having just one configuration mechanism for everything is simpler than having two.

2. The devfile is visible to the user. Therefore, he can see what configuration variables are used and what configuration will be written. The meta.yaml of a plugin is not shown anywhere in the system. Therefore, we would have to provide complete documentation on the variables used and what content they will produce somewhere. Since we cannot be responsible for all plugins in the long run, we would need some mechanism for users to access that plugin-owned documentation somehow.

3. It does not really provide anything beyond the configuration in the devfile: we can already provide an out of the box experience in our built-in devfiles that only requires an admin to set a config variable. The set of config variables can readily be discovered by looking at the built-in devfiles.

Use of secrets

I propose that we could reference kubernets secrets by name in the same way as config variables. In this way, we could write them to files in the workspace, inject them into env variables, etc. That last feature is useful for authenticating to repositories and has been requested as a separate feature, as well: https://github.com/eclipse/che/issues/14680

[tsmaeder]

@tolusha @skabashnyuk a lot of the implementation of the above concepts would fall in your domain, so could you please comment? Also @l0rd .

[l0rd]

I have some concerns with org/team/user configurations specified in devfiles. A devfile describes a development workspace (sources, dev tools, runtimes and commands) and may be used across multiple org/team/users.

We should provide a mechanism to configure workspaces components without explicitly specify the config bits in the devfile.

For example for environment variables:

If I am an admin I would like to specify some env variables at the instance level:

• che.workspaces.components.all.env_vars="GOPROXY=<proxy-setting>,..." <-- injected in all components of every workspaces
• che.workspaces.components.che_plugins.all.env_vars="GOPROXY=<proxy-setting>,..." <-- injected in chePlugin components of every workspaces
• che.workspaces.components.che_plugins.ms_vscode.go.env_vars="GOPROXY=<proxy-setting>,..." <-- injected in every chePlugin ms-vscode/go container

If I am a user I will want to specify some env variables for one or multiple devfile components as above but in this case in my user preferences:
json "che.workspaces.components.chePlugins.all.envVars": { "GOPROXY": "<proxy-setting>", }

cc @metlos @mshaposhnik @amisevsk @sleshchenko @davidfestal @benoitf

[sunix]

@l0rd https://github.com/eclipse/che/issues/14680 is about setting env variables for one or multiple devfile components in a kind of "user preferences". Could we make that happen ? not only for env variable, as settings.xml would be hard to customise through env variables, that's why secrets would make sense.

[tsmaeder]

@sunix to make that clear: you're suggesting to create a kubernetes secret containing the text contents of "settings.xml" and mounting that with the key "settings.xml" and the path "~/.m2 in all containers where it applies, correct?

[tsmaeder]

That would take care of how to create the config files. But how we specify that and override it in various places (install configuration/user-level) is still not to be discussed.

[sunix]

We could give the ability to the user to store his own (or organisation) m2_settings secret that would contain the settings.xml file. We could provide a default value for that file:

<settings />

in the devfile we would just do something like in kubernetes for a dockerimage or plugin, something like:

  components:
    - type: dockerimage
      ...
      volumes:
         secretName: m2_settings
         mountPath: "/user/.m2/settings.xml"

or something similar to https://docs.openshift.com/container-platform/4.3/builds/creating-build-inputs.html#builds-adding-input-secrets-configmaps_creating-build-inputs

for users and organizations that would like to use their own settings.xml (with credentials to maven repos, maven repos to use, proxies, etc ...) they would be able to change that in the default values in the orgianization level or let the user change that in his own secrets space like we would do for preferences.

Users would be able to use the devfile as it is and it could stay generic. We could then also remove the ssh hack and use the same mechanism everywhere

[l0rd]

@sunix yes I think that we should include #14680 in next sprint. The only thing I would suggest is avoiding to mention the secret in a devfile (the same devfile may work even without a particular settings.xml) but leverage secret's annotations and labels to specify that it should be mounted in a che workspaces.

[sunix]

so you mean, if a secret is not available, don't mount or don't expose env variable, log a message mentioning that and continue... maybe

[tsmaeder]

@sunix we would still have to have a mechanism that either does or doesn't create the secrets based on central settings or absence of those settings. We'd still have to have a way to inject central settings inside the contents of those secrets.

[tsmaeder]

But we could use the secrets mechanism to handle the "by hand" configuration cases where we need a file (and can't work with env variables).

[l0rd]

@sunix if the a secret is not available that may be perfectly fine. The same devfile may be shared among 2 users: one need to set a specific settings.xml but not the other. If we decouple the settings from a devfile that would be possible. Otherwise different orgs would need to use different devfiles (starting with the devfiles in our devfile registry)

@tsmaeder secrets can be created:

1. manually by the users before starting a workspace (first iteration)
2. automatically by whoever creates the users namespaces (second iteration)

[sunix]

I think that if we take the example

   components:
    - type: dockerimage
      alias: maven
      image: quay.io/eclipse/che-java11-maven:nightly
      env:
        - name: MAVEN_CONFIG
          value: ""
        - name: MAVEN_OPTS
          value: "-XX:MaxRAMPercentage=50 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
            -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
            -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom
            -Duser.home=/home/user"
        - name: JAVA_OPTS
          value: "-XX:MaxRAMPercentage=50 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
            -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
            -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom"
      memoryLimit: 512Mi
      mountSources: true
      volumes:
        - name: m2repo
          containerPath: /home/user/.m2/repo   
        - secretName: m2settings
          mandatory: false
          containerPath: "/user/.m2/settings.xml"

It can be shared and generic enough.
We could make that secret mounted if it does exist. the name here is important so the user would know which secret name to use to set a value or file to mount.

To me it's very important that the destination path is set in the devfile because it is impossible to predict where a file should be in a particular container. And that would avoid one settings.xml being used correctly in one container but not in the other.

So in all my devfiles or my organization devfiles, when a container need to use a maven and that we want to add the ability to add a custom or organization level settings.xml, just add the right secretName and the right containerPath

   components:
    - type: dockerimage
      alias: maven
      ...
      volumes:
        - secretName: m2settings
          mandatory: false
          containerPath: "/user.home/ofthecontainer/.m2/settings.xml"

and we could have some secretName that are commonly used in all our default devfiles. Then, if a team is choosing to use other secrets for other things to their devfiles, it would also work.

[tsmaeder]

@l0rd

automatically by whoever creates the users namespaces (second iteration)

The question is _when_ this can happen: since configuration can change, we would _at least_ have to create/delete the secrets when we create a workspace. But better still to pick up change at each workspace start. Do you think that "lifecycle" is possible with secrets?

[sunix]

let's say that a user have the previous devfile and has even created and started a workspace out of it. If he figures out it is not working (cannot access to maven central for instance), he could

• back to the dashboard (or anyother UI), create the secret "m2settings"
• upload his settings.xml with the right configuration to that secret "m2settings"
• restart his workspace and ... it will work :)

And let's say he is starting another workspace from another devfile that is using maven, it would work, taking the right settings.xml

[l0rd]

@tsmaeder I would like Che to not be responsible for creating and updating those secrets. To allow git with SSH flow we currently persist users SSH key pairs in postgres, expose a Che API to manage them and mount them in workspace Pods in the form of Kubernetes secrets. That's something I would like us to stop doing. Instead, when provisioning a new Che user, an administrator should setup the user namespace, including the configmaps and secrets. When needed, the admin should update namespace secrets and configmaps (and any namespace object). Developers can update configmaps and secrets as well but they should not need to deal with infrastructure configs if they do not want to. cc @skabashnyuk

@sunix what I am suggesting is a kind of dependency injection. It will simplify the devfile and keep it separate from infrastructure configurations. But you are right, this mechanism requires the agreement on some conventions (i.e. predictable settings.xml path in ${maven.home}/conf/ or ${user.home}/.m2/).

[sunix]

@l0rd Yes, what I am suggesting is a dependency injection by name: in this container, inject m2settings in /home/user/.m2/settings.xml . This would also be what user familiar to kubernetes would expect no?

How would you see the usage of secret's annotations and labels ?

[tsmaeder]

@l0rd If I understand your suggestion correctly, I see some drawbacks to your suggestion:

1. The setup task has to be repeated by the admin for every user
2. If a setting changes (let's say a new repo server is set up), the admin needs to update the secrets for every user.
3. If I take https://github.com/eclipse-theia/theia/issues/7516 as a guide, we need to mount the secrets per container, so Che _will_ be involved, even if it doesn't provide the values of the secrets. If we have more than one secret we need in a workspace, we'll need to provide a volume mount per secret (becuase it will need to be mounted in a different place.
4. In order to mount secrets, we need a mount point, which as I understand it a) needs to be present in the image and b) will be replaced with a directory containing only the secrets "files" specified in the volume. This won't really work for ~/.m2/settings.xml, since maven will want to write it's local artifact cache to~/.m2`.

[tsmaeder]

@nickboldt since air-gapped requirements are largely driven by CRW customers, I'd be interested in your input.

[l0rd]

@l0rd Yes, what I am suggesting is a dependency injection by name: in this container, inject m2settings in /home/user/.m2/settings.xml . This would also be what user familiar to kubernetes would expect no?

@sunix I think we are converging :-) We can use (optional) devfile annotations to specify injection properties like the path. Something like:

devfile.yaml
---
...
components:
 - type: dockerimage
   annotations:
      mavenSettingsPath: /my/m2/folder/  # <--- optional, overrides secret's defaultPath
   alias: maven
   image: maven:3.11
...

maven-settings-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: mvn-settings-secret
  labels:
    app: che                # <--- mandatory: all secrets labeled `app: che` will be mounted
    targetContainer: maven  # <--- optional, if omitted will be mounted on all containers
  annotations:
    mavenSettingsPath: /home/user/.m2/
data:
  settings.xml:  ...

@tsmaeder for 1) and 2) provision/updating users configs is about applying a template. An admin will have a template for che namespaces and will apply that every time a new Che user needs to be provisioned. That's how it works on che.openshift.io for instance. For 3) and 4) the sample above should clarify how a container and a path would be specified.

[tsmaeder]

@l0rd what is "maven-settings-secret.yml"? And I don't understand how the above addresses my point 4) from above: ~/.m2 will be broken for use with maven if you mount the secrets there.

[tsmaeder]

If I am a user I will want to specify some env variables for one or multiple devfile components as above but in this case in my user preferences:

@l0rd I'm not objecting to having per user settings, as I stated in my original proposal, I just did not consider it something that was a stated requirement.

[tsmaeder]

@tsmaeder I would like Che to not be responsible for creating and updating those secrets. To allow git with SSH flow we currently persist users SSH key pairs in postgres, expose a Che API to manage them and mount them in workspace Pods in the form of Kubernetes secrets.

Che is responsible for translating a devfile into a set of Kubernetes configurations (a pod, basically). That translation will use some central configuration values. There is no need for an API to manage anything, as far as I can see.

[sunix]

@l0rd ok as long as it works. It also has to work with env variables. Some secrets are mounted as files, others would be be exposed as env variables.

[skabashnyuk]

@l0rd about https://github.com/eclipse/che/issues/15518#issuecomment-610624721 . What are the rules to match :

 annotations:
      mvnSettingsSecretPath

and

annotations:
    defaultPath: /home/user/.m2/

[tsmaeder]

@l0rd @sunix @skabashnyuk I fear if my concerns from https://github.com/eclipse/che/issues/15518#issuecomment-610807237 are not considered, we're building something that will be quite useless.

[l0rd]

@tsmaeder let me try to answer but I am not sure that I got your questions so let's do a call if those answers don't help.

what is "maven-settings-secret.yml"

It's the specification of a secret that contains maven settings. Secret can be mounted as files in containers as specified here. This secret should be created in the developer namespace. For example I should have the secret created on my mloriedo-che namespace on che.osio. The secret can be created by the administrator that creates the namespaces or by the developers.

~/.m2 will be broken for use with maven if you mount the secrets there

True. Some possible workaround: mounting settings.xml in ${maven.home}/config, specifying <localRepository/> in settings.xml, using volumes subpath to mount the local repo or using an init container that copies settings.xml to ~.m2?

[l0rd]

@skabashnyuk you are right. I think that the name of the annotation should match.

[sunix]

how could .m2 be broken? @l0rd @tsmaeder? maven writes local artifacts in .m2/repository that could be a volume and not interfering with .m2/settings.xml

[l0rd]

@skabashnyuk I have update the name of the annotation in the secret and the devfile to be identical ( mavenSettingsPath)

[l0rd]

how could .m2 be broken? @l0rd @tsmaeder? maven writes local artifacts in .m2/repository that could be a volume and not interfering with .m2/settings.xml

@sunix I haven't tested but if settings.xml is mounted in .m2 it should override the folder completely. From the docs "Specify (...) mountPath to an unused directory name where you would like the secrets to appear".