Restic: Exclude patterns not being applied

[mholt]

Output of restic version

restic 0.6.1 (v0.6.1-126-gb0fb95df)
compiled with go1.8.3 on darwin/amd64

How did you start restic exactly? (Include the complete command line)

restic backup --exclude-file ~/exclude.txt ~

What backend/server/service did you use?

sftp to LAN

Expected behavior

Skip the files matching the pattern in exclude.txt

Actual behavior

Files were included unless I drastically loosened the pattern to the point where the pattern is too ambiguous.

Steps to reproduce the behavior

Using an exclude file like this:

.[^.]*
/matt/Applications
/matt/Dropbox
/matt/Downloads
/matt/VirtualBox VMs
*.vmdk

My goal is to skip dotfiles and a few specific folders in my $HOME directory (named matt) (but not every folder named "Applications" for example). Oh, and any .vmdk files (they can be huge!)

When I relaxed the patterns to this:

.[^.]*
Applications
Dropbox
Downloads
VirtualBox VMs
*.vmdk

it worked (meaning it skipped the folders) but this is too ambiguous, as I don't want to skip all folders named Downloads, for example.

Do you have any idea what may have caused this?

Probably me still not understanding exclude patterns correctly. :smile: I did read #1005 and I guess I'm confused if absolute paths are required for --exclude or if repo-absolute paths are required like with --include.

[mholt]

Update: Okay, replacing /matt/ with $HOME/ seemed to work, although I find the inconsistency between --exclude and --include absolute paths a little odd/surprising.

[ibib]

Usually $HOME/blabla is /home/matt/blabla or did you change something on your system?

[fd0]

Sigh, yes, you are absolutely right. The exclude patterns are absolute in regards to what you backup:
The pattern /home/matt/*.go matches all Go files in your home when you run restic backup /home/matt. Due to the way the archiver works (see #549, bad design decision we need to correct), the /home/ prefix is dropped and the top-level path component in the snapshot is /matt. So for restore, you'd need to use an include pattern of /matt/*.go to match the same files. I know it's bad, I'll correct it eventually :)

So, this is the corrected exclude file list for your use case:

.[^.]*
/home/matt/Applications
/home/matt/Dropbox
/home/matt/Downloads
/home/matt/VirtualBox VMs
*.vmdk

[mholt]

@ibib On my Mac, my home directory is /Users/matt, but since --include uses a repository-absolute path (like restic ls shows), I figured the proper absolute path would be /matt/....

@fd0 Ah! That makes more sense, thanks for explaining. So --exclude when used with restic backup is absolute to the file system, and --include (used with restic restore) is absolute to the repository. In other words, the patterns are absolute to where the files are coming from.

Thanks so much for the explanation. Now I understand better why 549 is an issue you want to fix. :smile: