Wemake-python-styleguide: Eval math operations

[sobolevn]

We need to add a tool to eval pure math and bool operations.

For example, we can check that 1 + 2 is 3 and "a" * 2 is "aa"
We need it because sometimes we restrict to use several datatypes in some places: like float in dict keys. But, people can actually write float numbers as math expressions: 0.1 + 5 * 10 ^ 2. We have to parse this information and return result nodes.

Algorithm:

• if all elements are constants - return the evaluated result
• if anything fails - return the wrapping node itself

Refs #861

[orsinium]

Also actual for #739

[sobolevn]

And #977

[Jrryy]

I could try and take care of this myself. My question is, is this supposed to become a new rule or it would rather modify the code adding the evaluated operation as a new node?

[sobolevn]

it would rather modify the code adding the evaluated operation as a new node?

Something like this. But! I am not sure how this should work. I am open to ideas!

[Jrryy]

It is a complicated subject. People usually do not code such operations, but in certain situations it makes the code clearer. For example, if you want a variable to hold an amount of days in seconds, you could write days = x*24*60*60. There are constant operations, but you can clearly see what is stored in this variable.

I suppose I could explore the usage of some of the modules in the project for the purpose of auto improving the code, and use noqas in the cases in which the user would not want the code to be modified.

[sobolevn]

Thanks a lot, @Jrryy! Please, feel free to ask any questions. I am here to help!

[orsinium]

we restrict to use several datatypes in some places

This one sounds like something that should be moved into "typed linter" when we have one, it's about types, not values.

What about evaluating values, I in deal use astroid for it. It's a poorly documented, slow, buggy library but it does its job. At least, yuo can use it for inspiration on how to evaluate things.

Also, because infer has 4 different behaviors for uninferrable things, I use this small wrapper around it: https://github.com/life4/deal/blob/master/deal/linter/_extractors/common.py#L121-L129

[Jrryy]

It looks quite interesting, I will definitely take a look at it, although according to your explanation it might be even better that I myself make the evaluator.

[sobolevn]

I don't think that we would need anything except ast.literal_eval for this job. It is good enough for this task.
The main question is how can we store both original and evaluated nodes in the tree?

[orsinium]

I don't think that we would need anything except ast.literal_eval for this job.

AFAIK, literal_eval doesn't execute operations:

ast.literal_eval('1+2')
# ValueError: malformed node or string: <_ast.BinOp object at 0x7f5b50cce9b0>

[Jrryy]

Maybe it depends on the python version? In python 3.6.9:

>>> import ast
>>> ast.literal_eval('1 + 2')
3

However, I believe we still might need to use other modules since ast doesn't have any functionalities to modify or replace nodes in the AST, iirc.

[sobolevn]

doesn't have any functionalities to modify or replace nodes in the AST

See NodeTransformer https://docs.python.org/3/library/ast.html#ast.NodeTransformer

[orsinium]

Maybe it depends on the python version?

You're right, it works in 3.6 but doesn't work in all later python versions:

[sobolevn]

We have https://github.com/wemake-services/wemake-python-styleguide/blob/master/wemake_python_styleguide/logic/safe_eval.py 😆

[Jrryy]

See NodeTransformer https://docs.python.org/3/library/ast.html#ast.NodeTransformer

Completely forgot about that. You're right.

We have https://github.com/wemake-services/wemake-python-styleguide/blob/master/wemake_python_styleguide/logic/safe_eval.py 😆

All right, all of this is very helpful, this will solve many problems.

[Jrryy]

Hi. Sorry for the big delay, I had to take a break and I have been able to start coding on my own again pretty recently.

I have just started working on this issue and since I don't think we came up with a solution a month ago, after thinking about it for a while I wanted to propose one now.

According to the original post and several referenced issues, what we want is a piece of code that helped WPS evaluate expressions involving only constants and store the result, so that the visitors could easily visit the node and throw the corresponding violations.

My proposal is adding another step in the pipeline for transforming the AST tree (here) in which, for every binary operator, it would try to recursively evaluate the operations inside of it and store the result in an attribute inside the BinOp node itself, the same way it is done with wps_if_chained and other attributes of the sort. The values of that attribute would be:

• If the operation and all the operations inside of it are all made of constants, simply store the evaluated result. For example, if we had x = 5*"a", the ast.Mult node would have an attribute which equals to "aaaaa".
• If there was anything that was not a constant in the operation, None would be stored in the node. This would happen in, for example, x = 2*3 + y.
• If evaluating one of the operations threw an Exception, something else would have to be stored (any ideas are welcome). This would happen, for example, in x = 2 + 0/0.

If this sounds okay, I will proceed and implement this. Otherwise I will gladly accept any other ideas.

[sobolevn]

Sounds like a good idea. Let's try it!