Bfg-repo-cleaner: ***REMOVED*** printed throughout repo when using --replace-text on specific files

[warren-oneill]

After trying to replace the text on four files with sensitive data I found that ***REMOVED*** was placed all over my repo. Some examples were replacing docstrings, comments and curly brackets.

[rtyley]

Did you use any regex in your 'replacement text' file? This is what you would see if you accidentally used a regex that also matched features of computer code - like the } above.

[rtyley]

By default, the text-to-replace is parsed as a string literal, rather than a regex, so you would have had to prefix them with regex: in order for them to have been interpreted like that.

[warren-oneill]

Ah I think I know the problem. One the files I wanted to protect was a python file and it search my repo for any lines contained in the .py file (e.g. }, """) and replaced them with ***REMOVE***.

[warren-oneill]

after applying bfg --replace-text file_name.py I used git reflog expire --expire=now --all && git gc --prune=now --aggressive. Is there anything I could have changed to make sure it only affected the file I specified?

[warren-oneill]

perhaps I should have just used --delete-files

[rtyley]

bfg --replace-text file_name.py

Unfortunately that's not how the --replace-text flag is used - I'm curious to know if that was just a mistake on your part, or did you read an instruction telling you to do it that way somewhere? If so, where?

This the (admittedly not perfect) usage text for --replace-text:

--replace-text <expressions-file>
        filter content of files, replacing matched text. Match expressions should be
        listed in the file, one expression per line - by default, each expression is
        treated as a literal, but 'regex:' & 'glob:' prefixes are supported, with '==>' 

So you create a file called, e.g. replacements.txt, and put stuff like this in it - one line per password, credential, etc:

PASSWORD1 # Replace literal string 'PASSWORD1' with '***REMOVED***' (default)
PASSWORD2==>examplePass         # replace with 'examplePass' instead
PASSWORD3==>                    # replace with the empty string
regex:password=\w+==>password=  # Replace, using a regex

...then you run the BFG, specifying the file of replacements:

$ java -jar bfg.jar  --replace-text replacements.txt  my-repo.git

When you used bfg --replace-text file_name.py, you effectively told the BFG "here is a file of X passwords that I want you to remove from all files in this repo's Git history" (where X is just the number of lines of python code in the file).

Is there anything I could have changed to make sure it only affected the file I specified?

If you were only specifying a password in your replacements file, there usually wouldn't be any need to do that, because you really would want that password gone, wherever it was in your repo history. There are include/exclude expressions for filtering though:

--filter-content-including <glob>
        do file-content filtering on files that match the specified expression (eg '*.{txt,properties}')
--filter-content-excluding <glob>
        don't do file-content filtering on files that match the specified expression (eg '*.{xml,pdf}')

[warren-oneill]

Ah ok now I get the proper usage. I think deleting would have been better for me since I wanted to get rid of the whole file. Is there anyway to undo the replacements?

[rtyley]

I'm afraid the only recovery method is to use the backup you made before you started the the process:

[warren-oneill]

I have a backup. how exactly do I use it to rewrite the history?

[simonh1000]

I need the instructions above to understand how to use this program. I wish that its essence made it into the README