Operator-sdk: [ansible] CR variables converted to snake_case when passed to playbook

[jmazzitelli]

Bug Report

What did you do?

I have a CR yaml that contains camelCase key names. When the ansible playbook is called, all keys are converted to snake_case.

What did you expect to see?

I expect all variable key names for all CR settings to retain the exact spelling. I do not expect them to be converted to snake_case if they are camelCase.

What did you see instead? Under which circumstances?

All camelCase key names are converted to snake_case.

Environment

• operator-sdk version: 0.9.0

• Kubernetes version information:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:23:09Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+6569b4f", GitCommit:"6569b4f", GitTreeState:"clean", BuildDate:"2019-07-10T19:31:33Z", GoVersion:"go1.11.6", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

OpenShift 4.1

• Are you writing your operator in ansible, helm, or go?

Ansible

Possible Solution

shrug

Additional context

Create a CR with this in the spec:

  deployment:
    version: "1.0"
    affinity:
      pod:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: security
                operator: In
                values:
                - S2
            topologyKey: failure-domain.beta.kubernetes.io/zone

Now have the ansible script look at the deployment dict - e.g. print it out in a debug msg:

- debug:
    msg: "STARTING OPTEST: deployment is: {{ deployment }}"

Notice all the camelCase keys are converted to snake_case:

    "msg": "STARTING OPTEST: deployment is: {u'version': u'1.0', u'affinity': {u'pod': {u'preferred_during_scheduling_ignored_during_execution': [{u'pod_affinity_term': {u'label_selector': {u'match_expressions': [{u'operator': u'In', u'values': [u'S2'], u'key': u'security'}]}, u'topology_key': u'failure-domain.beta.kubernetes.io/zone'}, u'weight': 100}]}}}"
}

Replication Procedures

ATTACHMENT: optest.zip

I have attached a .zip file containing a very simple/small test operator. It shows this problem. Here is how I tested. I am running with a CRC VM running OpenShift 4.1.6 - you will notice the push-operator.sh script will push to the CRC image registry. If you run this test, you will probably need to change push-operator.sh to push to your own registry and make sure you change deploy/operator.yaml so the image references pull from your registry.

The steps are as follows - I assume the cwd is in the optest directory after you unzip the attachment.

1. ./build-operator.sh -- this builds the operator image
2. ./push-operator.sh -- this pushes the operator image to an image registry in CRC VM
3. ./create-operator.sh -- this creates the operator resources in the cluster
4. oc get deployment optest -n optestns -- run this to confirm the operator is up and running
5. ./create-cr.sh -- create the CR which contains camelCase key names
6. oc logs deployment/optest -n optestns -c ansible -- after the operator runs, get its logs to see the camelCase has been converted to snake_case.

The operator's playbook has only a 2-line role that looks like this (see roles/optest/tasks/main.yml):

- debug:
    msg: "STARTING OPTEST: deployment is: {{ deployment }}"

When step 5 above creates a CR that looks like this (see deploy/crds/optest_v1alpha1_optest_cr.yaml):

apiVersion: optest.example.com/v1alpha1
kind: Optest
metadata:
  name: example-optest
spec:
  deployment:
    version: "1.0"
    affinity:
      pod:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: security
                operator: In
                values:
                - S2
            topologyKey: failure-domain.beta.kubernetes.io/zone

The playbook outputs the following in the logs - which you see when running step 6:

$ oc logs deployment/optest -n optestns -c ansible
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/tmp/ansible-operator/runner/optest.example.com/v1alpha1/Optest/optestns/example-optest/artifacts/5679351375816964720//stdout
ansible-playbook 2.7.10
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/usr/share/ansible/openshift']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.5 (default, Oct 30 2018, 23:45:53) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
Using /etc/ansible/ansible.cfg as config file

/tmp/ansible-operator/runner/optest.example.com/v1alpha1/Optest/optestns/example-optest/inventory/hosts did not meet host_list requirements, check plugin documentation if this is unexpected
/tmp/ansible-operator/runner/optest.example.com/v1alpha1/Optest/optestns/example-optest/inventory/hosts did not meet script requirements, check plugin documentation if this is unexpected

/tmp/ansible-operator/runner/optest.example.com/v1alpha1/Optest/optestns/example-optest/inventory/hosts did not meet script requirements, check plugin documentation if this is unexpected

PLAYBOOK: 5f13ba22c7b749339cd785f127d423d1 *************************************
1 plays in /tmp/ansible-operator/runner/optest.example.com/v1alpha1/Optest/optestns/example-optest/project/5f13ba22c7b749339cd785f127d423d1

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************


ok: [localhost]
META: ran handlers

TASK [optest : debug] **********************************************************
task path: /opt/ansible/roles/optest/tasks/main.yml:1
ok: [localhost] => {
    "msg": "STARTING OPTEST: deployment is: {u'version': u'1.0', u'affinity': {u'pod': {u'preferred_during_scheduling_ignored_during_execution': [{u'pod_affinity_term': {u'label_selector': {u'match_expressions': [{u'operator': u'In', u'values': [u'S2'], u'key': u'security'}]}, u'topology_key': u'failure-domain.beta.kubernetes.io/zone'}, u'weight': 100}]}}}"
}
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0   

Notice the deployment dict now has keys like preferred_during_scheduling_ignored_during_execution and pod_affinity_term when they are really camelCase in the CR such as preferredDuringSchedulingIgnoredDuringExecution and podAffinityTerm

[jmazzitelli]

https://docs.openshift.com/container-platform/4.1/applications/operator_sdk/osdk-ansible.html#osdk-building-ansible-operator_osdk-ansible

The Ansible Operator passes all key-value pairs listed in the CR spec field
along to Ansible as variables. The names of all variables in the spec field
are converted to snake case (lowercase with an underscore) by the Operator
before running Ansible. For example, serviceAccount in the spec becomes
service_account in Ansible.

[jmazzitelli]

Workaround seems to be that I will need to load in the CR myself (using meta.name and meta.namespace as data I will need in order to know which CR to read) and replace the dicts passed into the ansible playbook with the dicts I read in directly.

[jmazzitelli]

Looks like 0.8.0 has a hidden variable that can be used:

https://github.com/operator-framework/operator-sdk/blob/v0.8.x/pkg/ansible/runner/runner.go#L405-L406

I did not see this documented anywhere, but this is exactly what I need.

I'm closing this issue, sorry for the noise (BTW: if this isn't documented, it should be :)

[Oblynx]

@jmazzitelli I had the same issue, thanks a lot for pointing to that variable. I can't find documentation for it, so I believe this issue should be reopened until it's documented.

[geerlingguy]

Seconding the reopening of this issue until docs are added for this feature.

[fabianvf]

We should both add more documentation for this, and add a way to disable it to the watches.yaml

[Reamer]

I have the same issue. I doesn't understand the workaround with the hidden variable. Can someone explain more.

[camilamacedo86]

Hi @jmazzitelli, @Oblynx, @geerlingguy

The same code shows valid for 0.12. This issue is labelled to make clear that it is about doc for ansible-operator. Also, please feel free to contribute with by raising a PR for the docs. Contributions with the project are very welcome :-)

What needs to be doc is: `How to use variable key names for all CR settings to retain the exact spelling and not converted them to snake_case if they are camelCase." For further info see here

[vinzent]

Actually I expected only first level of keys to be snake-cased. It was quite a surprise, that this is snake-case modification is recursive for the complete structure.

[TristanCacqueray]

@Reamer for a resource named:

- version: v1alpha1
  group: softwarefactory-project.io
  kind: Zuul

You can access the unmodified spec with this ansible var: _softwarefactory-project_io_zuul_spec, at least with quay.io/operator-framework/ansible-operator:v0.13.0

[vinzent]

I'm aware of that.

At least we're using the role with an ansible operator AND additionally also with "traditional" inventory. So I need to check if that specific key exists or not and use set_fact to create a temporary fact that stores the actual data to be used.

[estroz]

Person to help triage this issue: @fabianvf

[jmazzitelli]

Did the rules change as to what is considered camelCase in 0.17.0?

We had a setting "iter8" that worked fine, but now its being converted to "iter_8" - and this didn't happen before.

[vinzent]

@jmazzitelli That behaviour i already faced with 0.10 (first version i've used).

[jmazzitelli]

We had iter8 working ok (not converted to iter_8) up until 0.15.0 (and possibly 0.16.0). At least, that is the report I'm getting.

[vinzent]

At least it was on 0.10/11 that way, that was the time I changed to read from the unmodified "automagic" key.

I personally tink the auto conversion is doing more harm than it helps. It is cool to have at the beginning but soon you will be surprised about things you didn't thought of.

[jmazzitelli]

@vinzent I'm specifically asking about variables with numbers in them. Yes, the snake_case conversion has been there for a long time.. but SPECIFICALLY for an alphanumeric string followed by a digit - is that now considered camelCase where before it was not?

[vinzent]

yes, i had exactly that for me unexpected converting of a name with a number at the end. bla1 ended up with bla_1.

[aneeshkp]

I am working on converting helm charts into ansible template.

In the process of porting templates, the helm charts .values are transferred into roles/x/defaults/main.yml and by default, they are all camelCase.

Now when I want to override the values via CR, they won't work unless I convert all the variables in roles/x/defaults/main.yml and in all the templates into snake_case. Is there any easy solution to tackle this?
I read something about the hidden variable above in the discussion but couldn't understand much. I am trying to avoid parsing syntax tree to find all variables and replace these variable names as snake_case.