St2: Mistral not rendering multiple Jinja expressions properly

Created on 16 Aug 2017 · 7Comments · Source: StackStorm/st2

Problem

I have a Mistral workflow (below) that has a line of Jinja with multiple expressions in it. This is 100% valid Jinja that renders properly as show in code below. In Mistral i receive the following error:

Error

2017-08-16 13:27:27.700 935 ERROR mistral.engine.task_handler [-] Failed to handle action completion [error=Can not evaluate Jinja expression [expression= _.directory }}/{{ _.filename , error=chunk after expression, data={}], wf=examples.broken_jinja_example, task=create_path_jinja, action=std.noop]:
Traceback (most recent call last):
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 105, in _on_action_complete
    task.on_action_complete(action_ex)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py", line 264, in on_action_complete
    self.complete(state, state_info)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py", line 163, in complete
    data_flow.publish_variables(self.task_ex, self.task_spec)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/workflow/data_flow.py", line 211, in publish_variables
    task_ex.published = expr.evaluate_recursively(data, expr_ctx)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 96, in evaluate_recursively
    data[key] = _evaluate_item(data[key], context)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 79, in _evaluate_item
    return evaluate(item, context)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 71, in evaluate
    return evaluator.evaluate(expression, context)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/jinja_expression.py", line 140, in evaluate
    result = JinjaEvaluator.evaluate(patterns[0][1], data_context)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/jinja_expression.py", line 89, in evaluate
    ", data=%s]" % (expression, str(e), data_context)
JinjaEvaluationException: Can not evaluate Jinja expression [expression= _.directory }}/{{ _.filename , error=chunk after expression, data={}]
: JinjaEvaluationException: Can not evaluate Jinja expression [expression= _.directory }}/{{ _.filename , error=chunk after expression, data={}]
2017-08-16 13:27:27.700 935 INFO workflow_trace [-] Task 'create_path_jinja' (b7758c32-e182-4dfa-8f5e-4b0437473372) [SUCCESS -> ERROR, msg=Failed to handle action completion [error=Can not evaluate Jinja expression [expression= _.directory }}/{{ _.filename , error=chunk after expression, data={}], wf=examples.broken_jinja_example, task=create_path_jinja, action=std.noop]:
Traceback (most recent call last):
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 105, in _on_action_complete
    task.on_action_complete(action_ex)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py", line 264, in on_action_complete
    self.complete(state, state_info)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py", line 163, in complete
    data_flow.publish_variables(self.task_ex, self.task_spec)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/workflow/data_flow.py", line 211, in publish_variables
    task_ex.published = expr.evaluate_recursively(data, expr_ctx)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 96, in evaluate_recursively
    data[key] = _evaluate_item(data[key], context)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 79, in _evaluate_item
    return evaluate(item, context)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 71, in evaluate
    return evaluator.evaluate(expression, context)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/jinja_expression.py", line 140, in evaluate
    result = JinjaEvaluator.evaluate(patterns[0][1], data_context)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/jinja_expression.py", line 89, in evaluate
    ", data=%s]" % (expression, str(e), data_context)
JinjaEvaluationException: Can not evaluate Jinja expression [expression= _.directory }}/{{ _.filename , error=chunk after expression, data={}]
] (execution_id=bf923a97-430b-4c4d-90ec-b5d67ac3b4a5): JinjaEvaluationException: Can not evaluate Jinja expression [expression= _.directory }}/{{ _.filename , error=chunk after expression, data={}]
2017-08-16 13:27:27.704 935 INFO workflow_trace [-] Workflow 'examples.broken_jinja_example' [RUNNING -> ERROR, msg=Failed to handle action completion [error=Can not evaluate Jinja expression [expression= _.directory }}/{{ _.filename , error=chunk after expression, data={}], wf=examples.broken_jinja_example, task=create_path_jinja, action=std.noop]:
Traceback (most recent call last):
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 105, in _on_action_complete
    task.on_action_complete(action_ex)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py", line 264, in on_action_complete
    self.complete(state, state_info)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/engine/tasks.py", line 163, in complete
    data_flow.publish_variables(self.task_ex, self.task_spec)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/workflow/data_flow.py", line 211, in publish_variables
    task_ex.published = expr.evaluate_recursively(data, expr_ctx)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 96, in evaluate_recursively
    data[key] = _evaluate_item(data[key], context)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 79, in _evaluate_item
    return evaluate(item, context)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/__init__.py", line 71, in evaluate
    return evaluator.evaluate(expression, context)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/jinja_expression.py", line 140, in evaluate
    result = JinjaEvaluator.evaluate(patterns[0][1], data_context)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/expressions/jinja_expression.py", line 89, in evaluate
    ", data=%s]" % (expression, str(e), data_context)
JinjaEvaluationException: Can not evaluate Jinja expression [expression= _.directory }}/{{ _.filename , error=chunk after expression, data={}]
] (execution_id=bf923a97-430b-4c4d-90ec-b5d67ac3b4a5): JinjaEvaluationException: Can not evaluate Jinja expression [expression= _.directory }}/{{ _.filename , error=chunk after expression, data={}]

Action

---
description: Workflow that reproduces a problem in Mistral with Jinja expressions
enabled: true
runner_type: mistral-v2
entry_point: workflows/broken_jinja_example.yaml
name: broken_jinja_example
pack: examples
parameters:
  directory:
    default: /opt/stackstorm/configs
    type: string
  filename:
    default: test.yaml
    type: string

Workflow

version: '2.0'

examples.broken_jinja_example:
  description: Workflow that reproduces a problem in Mistral with Jinja expressions
  type: direct
  input:
    - directory
    - filename

  output:
    yaql_path: "{{ _.yaql_path }}"
    jinja_path: "{{ _.jinja_path }}"

  tasks:

    create_path_yaql:
      action: std.noop
      publish:
        yaql_path: <% $.directory %>/<% $.filename %>
      on-success:
        - create_path_jinja

    create_path_jinja:
      action: std.noop
      publish:
        jinja_path: "{{ _.directory }}/{{ _.filename }}"

Python code that renders the Jinja properly

#!/usr/bin/env python
import jinja2
import os


class JinjaUtils:

    @staticmethod
    def render_str(jinja_template_str, context):
        env = jinja2.Environment()
        tmpl = env.from_string(jinja_template_str)
        return tmpl.render(context)


if __name__ == "__main__":

    context = {'_': {'directory': '/opt/stackstorm/configs',
                     'filename': 'test.yaml'} }
    template = "{{ _.directory }}/{{ _.filename }}"
    print JinjaUtils.render_str(template, context)

bug

Source

nmaludy

Most helpful comment

@humblearner already on it https://github.com/StackStorm/mistral/pull/32 :smile:

nmaludy on 8 Sep 2017

🎉2

All 7 comments

So, i think i figured out what is going on here... but haven't figured out how to fix it yet.

Looking at the mistral jinja expression code here:

https://github.com/StackStorm/mistral/blob/master/mistral/expressions/jinja_expression.py#L123-L126

It's trying to match the regex `({{(.*)}})' to the pattern.

https://github.com/StackStorm/mistral/blob/master/mistral/expressions/jinja_expression.py#L30

Well according to the following regex breakdown:

https://regex101.com/r/05FFax/1

The regex is matching out the outside {{ }} instead of parsing out each group individually. This regex matching, or the logic here needs to change. I'm not sure what it needs to change to yet.

nmaludy on 8 Sep 2017

After doing some regex experimentation i think we want to change the regex from
({{(.*)}})
to
({{(.*?)}})

This makes the "match everything" .* non-greedy and stop at the first }} it finds, instead of greedily consuming the rest of the string at stopping at the last }}.

nmaludy on 8 Sep 2017

Yup, that seems to fix the issue. Working in v2.4. Care to open a PR @nmaludy with a test : https://github.com/StackStorm/mistral/blob/master/mistral/tests/unit/expressions/test_jinja_expression.py. Thanks! 🙂

humblearner on 8 Sep 2017

@humblearner already on it https://github.com/StackStorm/mistral/pull/32 :smile:

nmaludy on 8 Sep 2017

🎉2

Submitted to OpenStack Mistral upstream for review https://review.openstack.org/502198

nmaludy on 9 Sep 2017

@m4dcoder This got merged into Mistral and is available on GitHub https://github.com/openstack/mistral/commit/2b985feb0b37a6f9fbe506250885a36305bfabaf

What's the process for getting this merged into st2mistral? Should i create a merge PR over in that project? Should i just let you guys handle it?

nmaludy on 14 Sep 2017

Fix verified (using w/f in description):

vagrant@u16:/opt/stackstorm/packs/default/actions$ st2 run default.broken_jinja_example
...
id: 5a00d1c5de59a42542829b99
action.ref: default.broken_jinja_example
parameters: None
status: succeeded
start_timestamp: 2017-11-06T21:19:01.136671Z
end_timestamp: 2017-11-06T21:19:06.980924Z
result:
  extra:
    state: SUCCESS
    state_info: null
  jinja_path: /opt/stackstorm/configs/test.yaml
  tasks:
  - created_at: '2017-11-06 21:19:01'
    id: 204c5084-daa7-44bd-8187-601cfe69836a
    input: null
    name: create_path_yaql
    published:
      yaql_path: /opt/stackstorm/configs/test.yaml
    result: null
    state: SUCCESS
    state_info: null
    updated_at: '2017-11-06 21:19:01'
    workflow_execution_id: 5aed6d2c-970b-4e69-a80a-ba6f1323fada
    workflow_name: default.broken_jinja_example
  - created_at: '2017-11-06 21:19:01'
    id: 64bc89c1-3b3e-4e1a-8714-4e5e1e8c841b
    input: null
    name: create_path_jinja
    published:
      jinja_path: /opt/stackstorm/configs/test.yaml
    result: null
    state: SUCCESS
    state_info: null
    updated_at: '2017-11-06 21:19:01'
    workflow_execution_id: 5aed6d2c-970b-4e69-a80a-ba6f1323fada
    workflow_name: default.broken_jinja_example
  yaql_path: /opt/stackstorm/configs/test.yaml
vagrant@u16:/opt/stackstorm/packs/default/actions$ st2 --version
st2 2.5.0

humblearner on 6 Nov 2017

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Orquesta with-items bug

trstruth · 3Comments

Orquesta should allow publishing of all the intermediate task results and published variables

Kami · 8Comments

Pack Install Doesn't Honor System user

bigmstone · 8Comments

st2 --debug outputs URLs that are not accessible from remote hosts

nmaludy · 5Comments

st2 execution list on large database of executions results in "Message length is larger than server max message size"

trstruth · 5Comments