Restic: excluding $RECYCLE.BIN with an exclude file doesn't work

[kontakm]

Output of restic version

restic 0.9.6 (v0.9.6-0-gb723ca3d) compiled with go1.10.4 on linux/amd64

How did you run restic exactly?

I have an NTFS partition shared between a dual-boot Windows 10 and Ubuntu 18.04, which I backup with restic from Ubuntu. As you probably know, Windows 10 creates a directory $RECYCLE.BIN in the root directory of the partition to hold files that are moved to the recycle bin. I do want to backup the shared partition, but not the $RECYCLE.BIN directory.

Further below I have provided a detailed description on how to reproduce what I am observing. Long story short: Using --exclude="\$RECYCLE.BIN" with the backup command does exclude the directory, putting "$RECYCLE.BIN" (or any other variant I could come up with, see below) into an --exclude-file does not exclude the directory.

What backend/server/service did you use to store the repository?

I observe the behavior both with a local repository (under Ubuntu 18.04) and with a remote repository via SFTP (backup run also on Ubuntu 18.04).

Expected behavior

The folder $RECYCLE.BIN is excluded from the backup when added to an exclude file.

Actual behavior

The folder $RECYCLE.BIN is included in the backup although added to an exclude file.

Steps to reproduce the behavior

Inside a temporary test directory, issue the following commands in a bash:

# prepare the directory to be backed up
mkdir "data";
mkdir "data/\$RECYCLE.BIN";
touch "data/backup_this";
touch "data/\$RECYCLE.BIN"/ignore_this";

# init repo
mkdir "repo";
restic init --repo=repo/

# prepare exclude file
echo "\$RECYCLE.BIN" > exclude.file

Now issuing restic --repo=repo/ backup --verbose --verbose data/ yields:

open repository
enter password for repository: 
repository 837fc3f7 opened successfully, password is correct
lock repository
load index files
start scan on [data/]
start backup on [data/]
scan finished in 0.257s: 2 files, 0 B
new       /data/$RECYCLE.BIN/ignore_this, saved in 0.004s (0 B added)
new       /data/backup_this, saved in 0.000s (0 B added)

Files:           2 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Data Blobs:      0 new
Tree Blobs:      1 new
Added to the repo: 372 B

processed 2 files, 0 B in 0:00
snapshot 27c00c6b saved

as expected.

Issuing restic --repo=repo/ backup --verbose --verbose --exclude="\$RECYCLE.BIN" data/ yields

open repository
enter password for repository: 
repository 837fc3f7 opened successfully, password is correct
lock repository
load index files
using parent snapshot 27c00c6b
start scan on [data/]
start backup on [data/]
scan finished in 0.257s: 1 files, 0 B
unchanged /data/backup_this

Files:           0 new,     0 changed,     1 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Data Blobs:      0 new
Tree Blobs:      1 new
Added to the repo: 372 B

processed 1 files, 0 B in 0:00
snapshot 61c27a16 saved

which is also expected.

However, issuing restic --repo=repo/ backup --verbose --verbose --exclude-file=exclude.file data/, where

 $ cat exclude.file 
$RECYCLE.BIN

results in

open repository
enter password for repository: 
repository 837fc3f7 opened successfully, password is correct
lock repository
load index files
using parent snapshot 61c27a16
start scan on [data/]
start backup on [data/]
scan finished in 0.211s: 2 files, 0 B
unchanged /data/backup_this
new       /data/$RECYCLE.BIN/ignore_this, saved in 0.003s (0 B added)

Files:           1 new,     0 changed,     1 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Data Blobs:      0 new
Tree Blobs:      0 new
Added to the repo: 0 B  

processed 2 files, 0 B in 0:00
snapshot 9a62470f saved

That is, the directory is not excluded from the backup although it is present in the exclude file.

I did also try different variants of the exclude file (escaping the dollar sign, adding quotes, adding a newline at the end of the file), but nothing made restic to exclude the directory.

Do you have any idea what may have caused this?

I assume that there must be some error when dealing with file names from an exclude file that involve nasty characters (like the dollar sign).

Do you have an idea how to solve the issue?

Unfortunately, I have no clue about go (I'm only fluent in C++), but there must be something wrong in the interaction with the OS and/or the filesystem when dealing with file names including strange characters.

Did restic help you today? Did it make you happy in any way?

I use restic extensively both for local and remote backups. Keep up the good work!
If I can be of any additional help in resolving this issue, let me know.

[dionorgua]

Not sure about --exclude, but in exclude files restic expands environment variables. Use $$RECYCLE.BIN to avoid this.

See: https://restic.readthedocs.io/en/latest/040_backup.html#excluding-files

[kontakm]

Well, thanks, it seems that I did overlook exactly this single paragraph in the documentation.

It does seem to work out:

 $ cat exclude.file 
$$RECYCLE.BIN

and issuing restic --repo=repo/ backup --verbose --verbose --exclude-file=exclude.file data/ again yields

open repository
enter password for repository: 
repository 837fc3f7 opened successfully, password is correct
lock repository
load index files
using parent snapshot 96bdd658
start scan on [data/]
start backup on [data/]
scan finished in 0.211s: 1 files, 0 B
unchanged /data/backup_this

Files:           0 new,     0 changed,     1 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Data Blobs:      0 new
Tree Blobs:      0 new
Added to the repo: 0 B  

processed 1 files, 0 B in 0:00
snapshot 5c475b96 saved

as expected.

However, in now over 10 years of Linux experience I never encountered having to escape a $ by $$ instead of \$, but as the documentation states, this seems to be standard for os.ExpandEnv, so this is not specific to restic.

However, restic --repo=repo/ backup --verbose --verbose --exclude="$$RECYCLE.BIN" data/ yields

open repository
enter password for repository: 
repository 837fc3f7 opened successfully, password is correct
lock repository
load index files
using parent snapshot 5c475b96
start scan on [data/]
start backup on [data/]
scan finished in 0.204s: 2 files, 0 B
new       /data/$RECYCLE.BIN/ignore_this, saved in 0.002s (0 B added)
unchanged /data/backup_this

Files:           1 new,     0 changed,     1 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Data Blobs:      0 new
Tree Blobs:      0 new
Added to the repo: 0 B  

processed 2 files, 0 B in 0:00
snapshot ed44b219 saved

which seems to be quite inconsistent and counter-intuitive.

[chrisrd]

The double quotes in --exclude="$$RECYCLE.BIN" mean that the shell expands $$ to the current PID. Try it with single quotes: --exclude='$$RECYCLE.BIN'

(E.g. look at echo "$$RECYCLE.BIN" vs echo '$$RECYCLE.BIN')

[MichaelEischer]

@kontakm The issue looks like it's resolved to me. Please close it if that's the case.