Robotframework: [RF 3.2.1] Inline Python evaluation issue using python module in expression part of list comprehension

[marcinmath]

Inline Python evaluation in RF 3.2 seems to not work for me in the given example:

1. Version information:
   Robot Framework: 3.2.1
   Python interpreter type: pyenv Python 3.6.9
   OS: Ubuntu 18.04

2. Steps to reproduce the problem:

    *** Variables ***
    # message in my real test is data loaded from file stored as string
    @{message}    {"foo": {"bar": "5.0.0"}}    {"foo1": {"bar1": "5.0.1"}}

    *** Test Cases ***
    Old Syntax Test Working
        ${final_message}    Evaluate    [json.loads(x) for x in ${message}]    modules=json

    Test New RF32 Inline Python Evaluation Syntax Not Working
        ${final_message}    Set Variable    ${{[json.loads(x) for x in $message]}}

Output
```

Test

Old Syntax Test Working | PASS |

Test New RF32 Inline Python Evaluation Syntax Not Working | FAIL |

Resolving variable '${{[json.loads(x) for x in $message]}}' failed: Evaluating expression '[json.loads(x) for x in $message]' failed: NameError: name 'json' is not defined

Test | FAIL |
2 critical tests, 1 passed, 1 failed

2 tests total, 1 passed, 1 failed

```

Am I doing wrong something here or it's RF issue?

[LukaszKosc]

I confirm that issue.
My system:
Microsoft Windows 10 Version 1809 (OS build 17763.1217)
robotframework==3.2
Python 3.6.5

My example below:

*** Test Cases ***
Test Math Old Syntax
Test Math RF3.2 Syntax

*** Keywords ***
Test Math RF3.2 Syntax
    @{output}    Set Variable   ${{[math.pi+float(x) for x in [1,2,3]]}}
    log    ${output}   console=True

Test Math Old Syntax
    @{output}    Evaluate   [math.pi+float(x) for x in [1,2,3]]    modules=math
    log    ${output}   console=True

Output:

==============================================================================
Test :: Suite description
==============================================================================
Test title                                                            [4.141592653589793, 5.141592653589793, 6.141592653589793]
Test title                                                            | FAIL |
Resolving variable '${{[math.pi+float(x) for x in [1,2,3]]}}' failed: Evaluating expression '[math.pi+float(x) for x in [1,2,3]]' failed: NameError: name 'math' is not defined
------------------------------------------------------------------------------
Test :: Suite description                                             | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================

[pekkaklarck]

I can reproduce this issue with Python 3 as well and was to some extend aware something like this could happen. Why the problem occurs is pretty complicated:

• Python has nested namespaces for variables. Most of the time you are only working with local and global scopes, but, for example, nested functions create also other namespaces. These "other scopes" are the reason Python 3 added the nonlocal statement.

• When using eval(), you can only pass it global and local namespace. In addition to that, the global namespace must be a real dictionary, not a custom mapping. As the docs explain elsewhere eval() doesn't have the full environment for resolving names.

• In Python 3 list comprehensions are implemented internally so that they use function namespaces which crates one additional scope. In Python 2 list comprehensions are implemented as for loops and that avoids this problem altogether.

• Our custom evaluation logic uses eval() so that it passes explicit modules (can be given only with the Evaluate keyword) as a global namespace and uses a custom object as a local namespace. This custom local namespace is responsible on handling the $variable syntax as well as automatically importing modules used in the expression.

• Due to the aforementioned changes to how list comprehensions are implemented, Python 3 consults only the global namespace when evaluating the json.loads(x) part of an expression like [json.loads(x) for x in $message]. When using Evaluate with explicit modules=json the module is in the global namespace but when using the inline evaluation syntax it isn't. Because Python 3 doesn't query json from our custom local namespace object, we cannot automatically import it.

This example demonstrates that you can encounter this issue also with plain Python:

>>> def example():
...     import json
...     return eval("[json.loads(i) for i in ['1', '2']]")
... 
>>> example()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in example
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <listcomp>
NameError: name 'json' is not defined

As the following example shows, the problem doesn't occur without eval() because in that case the list comprehension sees the intermediate namespace where json is imported:

>>> def example():
...     import json
...     return [json.loads(i) for i in ['1', '2']]
... 
>>> example()
[1, 2]

The problem doesn't occur if json is in the global namespace either:

>>> import json
>>> def example():
...     return [json.loads(i) for i in ['1', '2']]
... 
>>> example()
[1, 2]

In my opinion this is a bug in Python 3 caused by the change to list comprehension implementation. It has been reported several times already in different forms, but apparently fixing is it hard and there doesn't seem to be that much interest on it.

Unfortunately I don't have any good ideas how to fix or workaround this on Robot side. We could try catching NameErrors and importing modules mentioned in them but that feels pretty hackish and complicated.

[pekkaklarck]

Here's a hackish workaround for the problem:

--- a/src/robot/variables/evaluation.py
+++ b/src/robot/variables/evaluation.py
@@ -13,6 +13,7 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.

+import re
 from tokenize import generate_tokens, untokenize
 import token

@@ -44,12 +45,21 @@ def evaluate_expression(expression, variable_store, modules=None,
                         % (expression, get_error_message()))


-def _evaluate(expression, variable_store, modules=None, namespace=None):
+def _evaluate(expression, variable_store, modules=None, namespace=None, retry=True):
     if '$' in expression:
         expression = _decorate_variables(expression, variable_store)
     global_ns = _import_modules(modules) if modules else {}
     local_ns = EvaluationNamespace(variable_store, namespace)
-    return eval(expression, global_ns, local_ns)
+    try:
+        return eval(expression, global_ns, local_ns)
+    except NameError as err:
+        if retry and not PY2:
+            match = re.match("^name '(.*)' is not defined$", str(err))
+            if match:
+                modules = '%s,%s' % (modules or '', match.group(1))
+                return _evaluate(expression, variable_store, modules, namespace, 
+                                 retry=False)
+        raise

It's not as ugly as I expected but there are two reasons I still don't like it too much:

• Relaying on Python exception message details isn't that great. Adding a test for this fix would help detecting if the message changes, but this is such a detail that could change even in a minor Python version.

• Evaluating the expression twice can have strange results. For example, a generator could possibly be exhausted so that in the second run there's nothing to iterate anymore.

An alternative is just documenting this behavior in the User Guide.

[marcinmath]

Thank you for providing nice workaround.
Hope this is minor issue since we can still use old Evaluate syntax.