Pycodestyle: Escaped character in regular expression falsely triggers W605 "invalid escape sequence"

[benguyer]

Here's a simple example of a python script that falsely triggers this warning. It matches the literal character "?"

import re

regex = re.compile("\?")
non_matching_string = "test"
matching_string = "test?"

if regex.search(non_matching_string):
    print("This won't be printed")
if regex.search(matching_string):
    print("This will be printed")

I am using:
pycodestyle version: 2.4.0
python version: 3.5.2

[sigmavirus24]

With the following

import re

regex = re.compile(r"\?")
non_matching_string = "test"
matching_string = "test?"

if regex.search(non_matching_string):
    print("This won't be printed")
if regex.search(matching_string):
    print("This will be printed")

The only difference is

regex = re.compile(r"\?")

The behaviour does not change and pycodestyle is happy.

~/s/p/pycodestyle ❯❯❯ ./pycodestyle.py t.py
~/s/p/pycodestyle ❯❯❯ python t.py
This will be printed
~/s/p/pycodestyle ❯❯❯ python3 t.py
This will be printed

I'm wondering if there's something wrong with that such that you can't use it.

[benguyer]

@sigmavirus24 Thanks, that works perfectly. Sorry, I had missed that when reading the python regular expression documentation.

[thorade]

autopep8 started adding the r prefix for several strings, is this due to this issue here? I do not really understand, is it recommend/required to have the r?

[FichteFoll]

Depends on what you are or were doing. If you use invalid escape sequences, previously they were expanded to the literal backslash and the following char, but because this will become a syntax error in the future, you need to either use a raw string or escape the backslashes. You may decide on which you want to use, but in most cases people probably forgot to add a r for regular expressions or file paths, in which case turning those into raw strings is the best fix.

[thorade]

Thanks for clarification. Yes, it was a regex.

[rask]

The following works but triggers pylint warning:

subprocess.run(['foobar | grep -E \'\s+?(0-9)+\''], text=True)

The following breaks but does not trigger pylint warning:

subprocess.run([r'foobar | grep -E \'\s+?(0-9)+\''], text=True)

[sigmavirus24]

@rask This is not the pylint issue tracker and your comment doesn't contribute to the conversation. If you're seeking help, StackOverflow would be a good place to ask for it.

[asottile]

@rask blinding adding r isn't always the correct solution (as there are also correct escape sequences in your string which will be converted to raw backslashes)

fortunately, pyupgrade knows how to make the correct transformation

$ cat t.py
subprocess.run(['foobar | grep -E \'\s+?(0-9)+\''], text=True)
$ pyupgrade  t.py
Rewriting t.py
$ cat t.py
subprocess.run(['foobar | grep -E \'\\s+?(0-9)+\''], text=True)

[rask]

Sorry, meant pycodestyle, running version 2.5.* here.

What my problem is that making a fix to make the tool pass produces broken code, which is probably what is not wanted.

[rask]

@asottile In the meantime I fixed the problem by converting \s to [ \t], which is not optimal but gets the job done.

[asottile]

@rask right, your fix was not the correct one in the presence of other valid escape sequences -- notice the pyupgrade output: the correct fix is to escape just the backslash in front of the s and not to make it a raw string