Yapf: slow on large files?

[hayd]

E.g. this (3300 line) file takes yapf over 10 minutes https://github.com/pydata/pandas/blob/master/pandas/tseries/tests/test_offsets.py.

 % time yapf --diff pandas/tseries/tests/test_offsets.py > /dev/null
yapf --diff pandas/tseries/tests/test_offsets.py > /dev/null  712.20s user 5.37s system 84% cpu 14:04.86 total
 % time autopep8 --diff pandas/tseries/tests/test_offsets.py > /dev/null
autopep8 --diff pandas/tseries/tests/test_offsets.py > /dev/null  13.46s user 0.33s system 85% cpu 16.191 total

_(related: #39 ?)_

[eliben]

Yes, this is very likely related to #39

[gwelymernans]

Agreed. There are a lot of large data literals in there. My guess is that autopep8 isn't looking at them, because they aren't violating PEP 8 style, maybe?

When you do get the output, does it look good? Are there any areas where we could improve (mostly with the data literals)?

[hayd]

The autopep8 diff is mostly spaces around operators, a bit of lining up indentation, and breaking a few lines. Here's a gist of the respective diffs.

yapf's output looks pretty reasonable, comment indentation seems a bit off... a couple of possibly "new" issues:

• precedence? should prefer to split the entire function call (also here)
• comment references a specific line (kinda annoying to push it to the next...)

---

Note: This file has had autopep8 applied to it in the (distant) past so it's not a fair fight, in the sense that autopep8 may have to do fewer passes than it would if the file was "cold"... that said 712s is a long time.

...autopep8 gets (slightly) faster with fewer/no pep8 infractions, IIUC yapf is constant (to the ast).

[gwelymernans]

The "splitting the entire function call" situation is something that I've seen in the past and would love to have a solution to. I have a suspicion that it will come down to increasing the connectedness of the function call if it can fit on a single line, etc. I have a rough idea, involving overlapping strongly connected regions. But I'm not sure how well it will work...

The comment references is a bug. :-( Please file a PR for it.

[hayd]

Will see about a PR :), for the moment it's #124! I think "splitting the entire call" is equivalent to #123.

[joshburkart]

Could YAPF be compiled with Grumpy? I really like how clang-format is always nearly instantaneous even on rather large files...

[garymm]

I still find YAPF to be terribly slow on large files.

[sebastien-rosset]

I have done a comparison of yapf versus black for a large python file which is generated by the OpenAPITools code generator.

• time yapf -i iam_api.py completes in 1m50.350s
• time black iam_api.py completes in 0.786s
• time black --fast iam_api.py completes in 0m0.641s

So yapf is about 139 slower compared to black for that particular file.