Ansible: 'recursive loop detected in template string' error with variables in name

[inderpreet99]

Issue Type:

Bug Report

Ansible Version:

1.6.10 and 1.7

Environment:

Mac OSX 10.9.4

Summary:

I've found a weird bug with ansible playbooks (or maybe my code/setup). When I use a tasks playbook using an include, the variables passed in can't be used in the 'name' of a module.

Steps To Reproduce:

• Create a tasks playbook
• Create a runner playbook that includes tasks playbook
• Add a task to tasks playbook that takes a parameter from the runner playbook, and in it's name: include the variable.
• Run the task runner playbook
• You will receive a recursive loop detected error.

Here's the problem example:
task_runner.yml

---
- name: Disable {{ host }}
  hosts: all
  gather_facts: no # for debug purposes

  tasks:
    - name: Run task
      include: tasks.yml
      vars:
        host: "{{ host }}"

tasks.yml

- name: "ping {{ host }} twice"
  shell: ping -c 2 {{ host }}

It should be noted that if I change name: "ping {{ host }} twice" to name: "ping host twice", I will not receive the error (that I'm receiving in "Actual Results" below).

Expected Results:

No error message.

Here's a working example first done in a single playbook:
task_playbook.yml

---
- name: Disable {{ host }}
  hosts: all
  gather_facts: no # for debug purposes

  tasks:
    - name: ping {{ host }} twice
      shell: ping -c 2 {{ host }}

Here's how it runs:

(venv) isingh$ ansible-playbook -c local task_playbook.yml -i hosts --extra-vars host=google.com

PLAY [Disable google.com] *****************************************************

TASK: [ping google.com twice] *************************************************
changed: [fionn]

PLAY RECAP ********************************************************************
fionn                      : ok=1    changed=1    unreachable=0    failed=0

Actual Results:

(venv) isingh$ ansible-playbook -c local tasks_runner.yml -i hosts --extra-vars host=google.com

PLAY [Disable google.com] *****************************************************
ERROR: recursive loop detected in template string: {{host}}

Thanks.

[mpdehaan]

Hi!

So this seems to be working exactly as designed for me, because you're defining a variable named host who's value is host.

   vars:
        host: "{{ host }}"

I would recommend _NOT_ doing that.

It's kind of a "Doctor, My Arm Hurts When I Do This" kind of thing, if that joke translates :)

I'm going to close this one as working as designed, but feel free to stop by ansible-project if you'd like to discuss further. We're open to it.

[thom-nic]

I'm encountering the same error, although with a different configuration. This is enough to reproduce it in a playbook:

 vars:
    app:
      user:    rails
      home:    "/home/{{ app.user }}"

If I flatten the vars to app_user and app_home everything is peachy but I thought it was nice keep the hierarchy, especially when I have a lot of app-prefixed vars. Using ansible 1.8.3.

[bn0]

I'm in the same boat as thom-nic. Although after thinking about it, it does make sense. We're trying to reference a variable before it has been assigned.

Try the same thing in python:

>>> app = {"user": "rails", "home": "/home/" + app['user']}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'app' is not defined

It's a shame, because it does make for nice neat configs.
My workaround is to define re-used variables separately, i.e. in your case:

app_user: "rails"
app:
   user: "{{ app_user }}"
   home: /home/{{ app_user }}

Bit ugly, but so far it's a trade-off I'm ok with.

[octalthorpe]

Having something similar after moving from 1.6.X to 1.8.4. It was working before for us, now we run into the recursion issue.

We have a playbook as follows:

- role: rsyslog
      tags: ['rsyslog']
      rsyslog:
        reception:
          tcp:
            enabled: True
        transmission:
          tcp:
            enabled: True
            hosts: "{{rsyslog.producers}}"

Then in a group_vars for a specific environment we define the rsyslog:

rsyslog:
  producers: "{% set hosts = ['localhost:5544'] %}{% for host in groups['logmaster'] %}{% do hosts.append(host + ':5140') %}{% endfor %}

Once the rsyslog role is run we get the error:
=> recursive loop detected in template string: {{rsyslog.producers}}

I really hope this was not a bug that was fixed and now we cannot take advantage of it. My gut feel it is a ordering change, in the way variables are built up. Note we have forced the hash_behaviour to merge in our ansible configuration.

Any thoughts or suggestions would be welcomed before I dig in deeper or rework how we define things.

[sebas5384]

same problem here :-1:

[jaytaylor]

Same issue, worked around it by just moving and renaming the "conflicting" variable.

[leafpeak]

Same here...because you're aloud to pass "variables" as extra arguments there are times where I need to use the passed in value otherwise use a default value.

ansible-playbook playbook.yml -e "my_var=1"

# playbook.yml
vars:
  my_var: "{{ my_var | default(my_default_var) }}"

[michaeljs1990]

Just encountered what @thom-nic did. I am not entirely sure why that results in an error though. Looking at it it looks like something that should work perfectly fine.

[majidaldo]

yup. when i nest role dependencies, this happens.

[boyangwang]

Same issue here. Wanted to use the nicer format so bad

[conrado]

Software can solve all sorts of issues, and certainly this could be solved too. However the format for a dictionary calling on itself while it is being defined is in its own domain. A year ago someone asked this question, where a format is proposed, it could be a way to do it:

https://stackoverflow.com/questions/22989401/using-a-relative-path-to-reference-another-element-in-a-multi-dimensional-array

[anirudh-wa]

I am also running into the same error. Can someone tell me how do I go about it. I am using the below variable and I get recursive loop error.

maxHeapSizeAnsi: "{{(hostvars[inventory_hostname]['ansible_memtotal_mb'] * 3 / 4)|int}}"

[Hubbitus]

I also fall in it. Is it will be fixed? Should it be reopened?

[dan-petty]

+1

[jmcollin78]

+1

[ivan7farre]

+1

[penguinjournals]

+1

[d3vil-st]

+1

[wenerme]

+1

[mburz]

+1

[matthewhartstonge]

Seems dumb, yet it makes sense as to why this shouldn't work due to the dictionary trying to recursively ask itself for a key when the dictionary itself hasn't been fully initialized yet.
This is how I managed to get around it with a role:

Broken w/ recursion error

This is what I was trying to do before hand to have only one place to alter the version.

application:
  params:
    state: present
    version: "0.6.3"
    download_url: "https://releases.hashicorp.com/consul/{{ consul.params.version }}/consul_{{ consul.params.version }}_linux_amd64.zip"

  agent:
    ...

Fix

I decided to make use of the defaults folder and stuff the random things in there as keeping the vars flat forces them to be initialized first. Then, I could pull them into my 'beautiful' structure in vars/main.yml

   .
   |-roles
   |---Application
   |-----defaults
   |-------main.yml
   |-----files
   |-----handlers
   |-----meta
   |-----tasks
   |-----templates
   |-----vars
   |-------main.yml

In defaults/main.yml i have:

_version: "0.6.3"
_download_url: "https://releases.hashicorp.com/consul/{{ _version }}/consul_{{ _version }}_linux_amd64.zip"

In defaults/vars.yml i have:

consul:
  params:
    state: present
    version: "{{ _version }}"
    download_url: "{{ _download_url }}"

  is_agent: false
  agent:
    ...

This way the variables can still be altered higher up, but also allows a single source of where the 'default vars' can be changed from ;)

Edit

After testing, I changed default vars to have an underscore due to denoting them this way (via convention) as 'private'. Also, this way you know exactly what file the variables are being read from. heh.

[mamoit]

+1

[oupala]

+1

I would love to be able to use such configuration:

myapp:
  version: 1.6.3
  download_url: http://example.org/download/myapp-{{ myapp.version }}.tgz

[netkgk]

I would recommend NOT doing that.

It's kind of a "Doctor, My Arm Hurts When I Do This" kind of thing, if that joke translates :)

@mpdehaan, imagine I've just installed another module from the ansible galaxy in replacement for that I've used before. Imagine it has set default variable named, let's say mysql_version. Imagine I have the variable of the exact same name in my module, which has dependency in meta to the module I've just installed.

I don't want to change the variable name inside my module. As it's mnemonic by itself, and all the variables are named following the same rule, i.e. service_name_version and I don't want to stage my team up with something like mysql_version_another_stupid_name_because_ansible_cannot_do_that_properly. A special variable name is another nuance which a team member supposed to remember, hence more chance for an error. And I can't bring this from the inventory, as the module supposed to be sharable.

What would be your advice for that pretty much common use case?

[jeliser]

+1

[kravvcu]

+1

Not having this prevents having a nice, clean variable hierarchy, since we have to do it the old-fashioned prefixed way. Consider:

mysql:
  version: 5.5
  install_path: "/var/lib/mysql/{{ mysql.version }}"
  user: mysql
  group: mysql

vs

mysql_version: 5.5
mysql_install_path: "/var/lib/mysql/{{ mysql_version }}"
mysql_user: mysql
mysql_group: mysql

Although this is just an example not a real world case of (at least for me), it presents what the problem is. IMO, the latter definition is just nasty. The former applies some kind of DRY'ing out.

[kbechler]

+1

[rqelibari]

Referring to @bn0 answer above:

Although after thinking about it, it does make sense. We're trying to reference a variable before it has been assigned.

And also referring to all who say, that this feature would be logically nonsense:

This feature is actually well-defined in terms of meaning exactly one thing. The syntax and semantics (in terms of logic) are both well defined for this kind of "back reference".
The problem here really refers only on how an implementation would look like.

1. Implementation Problem

But wait: how can a hash map value refer to the hash map which is still not defined?

The assumption here is partly wrong. Because we are handling two syntaxes here. YAML and ansible variables. One must know, that ansibles notion of variables {{ some_var_name_here }} has no meaning in YAML (see section 2 of this) . It's actually ansible itself, which gives meaning to that string. And thus ansible is free on how to implement the parser to handle that. Let's see a quick example, given the following YAML settings document:

1. hash_map:
2.    version: ab
3.    path: "/somepath/{{hash_map.version}}"

When is the variable hash_map defined? Already on _line 1_ or only after _line 3_?

Two different implementations

Let's see how the implementation would be affected by this decision.

Notice: Those examples are really simplified, in reality all this would be handled by a parser differently.

Implementation#1 implies that the data structure is only defined after _line 3_ in the example above:

variables = {}
variables["hash_map"] = {"version": "ab", "path": "/somepath/{{hash_map.version}}" }
# We cannot do this!!!
variables["hash_map"] = {"version": "ab", "path": "/somepath/%s" % variables["hash_map"]["version"] }

Implementation#2 implies that _line 1_ above is totally sufficient to allocate the data structure.

variables = {}
variables["hash_map"] = {}
variables["hash_map"]["version"] = "ab"
# This is totally possible
variables["hash_map"]["path"] = "/somepath/%s" % variables["hash_map"]["version"]

Both represent the same, but as you can see the latter is somehow a bit more powerful.

2. Ok, so where is the problem then?

What does the standard say?

Let's see what we can achieve with plain YAML.

Representation of YAML according to the standard

The YAML standard describes the semantics of YAML as a rooted directed graph. Nodes are described under section 3.2.1.1. Nodes.

According to the _standard_ values are represented as nodes too. But - and here is the point - there is no node type as far as I know, which can semantically represent the feature above correctly (as in the way we want it), as one node. The only real option here would be using a sequence node (aka list in YAML).

Consequences of this

The YAML example above would be represented by the following graph according to my understanding of the standard:

As we want to give meaning to {{hash_map.version}} we would have to use anchor and alias nodes like that:

Note: This feature is described in the 6.2.9. Node Anchors section

1. hash_map:
2.    version: &anchor "ab"  #setting the anchor
3.    path:
4.        - "/somepath/"
5.        - *anchor          # referring to the anchor, yields ab

Verify this with this YAML online parser

yielding the following graph:

What follows of this?

Above we can see that we cannot combine those two nodes. As such there is no way to achieve this feature using YAML only syntax.
But we can also see, that {{ some_var_name_here }} is only an ansible notation and has no meaning in YAML. As such, we are free to define the semantics of that as we want!

TLDR

YAML is a markup language (despite of what it's acronym may suppose) and a markup language just answers the "what" question not "how". And as it is totally clear what the reference means, it is totally possible to implement this feature.

Keep in mind, that this feature is not conform with the current YAML standard v.1.2 (latest release according to Wikipedia), but so isn't our notion of ansible variables {{ some_var_name_here }}.

EDITS:

• 2018-04-15 21:11:
  
  
  
  • Reformulate the intro.
  
  
  • Correct misspelled word (now -> know).
  
  

[Hubbitus]

And so what the conclusion? As {{ some_var_name_here }} is just ansible notation and its semantic may be changed to handle "deffer resolving" do you plan implement something similar (like closure binding for example in groovy)? Issue still closed.

[rqelibari]

I would really love to implement this feature! It is not that easy though. As this feature is of greater impact, I will need some time, if I do it on my own. Preferably there would be one or two other contributors willing to help, in which case things would speed up.

[michaelkrupp]

To me this behavior was really unexpected and feels kind of weird. There are many possible scenarios where the current behavior is not very pleasant to work with:

~~~

• set_fact:
  host: "some.host.com"

• include: task.yml
  vars:
  host: "{{ host }}"
  ~~~

This potentially happens every time you pass data from one role/task to another. Especially with externally developed Roles (Galaxy) this can be a major PITA. The work-around of setting a intermediary fact or using a intermediary dummy-task is quite ugly:

~~~

• set_fact:
  not_host: "{{ host }}"

• include: task.yml
  vars:
  host: "{{ not_host }}"
  ~~~

[hugollm]

This is a very unexpected behavior. Any decent language allows you to set a dict key equal to a variable of same name. In python for instance:

{'host': host}

It would be real trouble if one had to hack the variable name around just to pass it forward:

{'host': _host}

Currently, the best workaround I've found is to prefix the variables with the role name. e.g.:

nginx_domain: '{{ myapp_domain }}'

It works but it's kinda troublesome, don't you think?

[shitalm]

+1

[Erouan50]

+1

[pierrefevrier]

+1

[tehfink]

+1

[wujie1993]

+1

[florres]

+1

[alexander-balashov]

+1

[cobyrne09]

+1

[Alwaysc0d3]

+1

[yaroslav-shabalin]

+1

[cars10]

+1

[duanshiqiang]

+1

[jn-bedag]

+1

[cmpunches]

+1

This is a YAML implementation bug.

[acjohnson]

+1

[JasonSheedy]

+1

[knowshan]

+1

[BennyHarv3]

+1

[martindines]

+1

[daMupfel]

I also hit this. Can you please think about reopening this?

[Alwaysc0d3]

@everybody, and i mean literally everybody :-).

This did hit me before, the only thing you can do is restructure your variables. Note you can't refer to a variable that's not already definded (see variable precedence in the ansible docs. Note also that if you refer to a variable you define later on in the file, you will get an error.).

The real 'problem' is not in Ansible but in Jinja2. If you think you can fix this, be my guest. However, i don't think its easy to fix, if not impossible within the current jinja2 architecture / project.

When you have something like this (credits to @rqelibari for the example):

hash_map:
  version: ab
  path: "/somepath/{{hash_map.version}}"

And also wants a clean and nice ansible variable file and / or directory. Consider another architecture, you could add variables in other files, you're not limited to the main.yml file or the all.yml file. Ansible is flexible, so if your'e smart, move and play with it :-D.

For example, you could create a file in the ' all ' directory of the group_vars named 'hash_map' and create the variables you need in this file.

Yes it should be nice if it's possible to refer to the key you noted in the same key-store, currently it is'nt possible. I don't see it happen within now and a few months either...

Work around it, be creative and don't make your ansible scripts a mess. Personally i did like to make everything configurable with the ansible-variables. The only reward i got was an unmanageable playbook and a fuzzy variable structure :-).

Alright, dont +1 this on ansible but fix this at the jinja2 project.

[cmpunches]

This issue is a direct result of ansibles use of a yaml parser's
interpolation feature instead of loading the yaml object and doing a second
pass for interpolation. This is not just a jinja bug this is an
implementation bug in ansible. Please inspect. Loading as raw string and
then processing the initialised object members in a second pass should fix
this.

[iron77]

+1

[bsberry]

+1

[ghost]

+1

[liubog2008]

+1