Borg: How to back up only certain files (e.g. *.jpg)

[jim-collier]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Question

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.1.13

Operating system (distribution) and version.

Ubuntu 18.04.4

Hardware / network configuration, and filesystems used.

Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
16 GB RAM
ZFS 5 x 3-way mirrors, 22 TiB usable

How much data is handled by borg?

15 TiB

Question

Is this possible? I can't find a reference to a command-line flag that will do this, or related FAQ, doc, or issue.

The reason I'm doing this, is to back up my most important files first, which I can get most optimally "close enough" with wildcard matching.

I understand I can generate a separate "includes" file but if there's a way to keep it native, that would be better for my use case.

Many thanks.

Update: I realize there actually is no --files-from flag or similar functionality (and --patterns-from is marked experiments - and seems a little dicey to include millions of files).

[ThomasWaldmann]

borg help patterns

[jim-collier]

Sadly, as mentioned, I'm not comfortable with the "experimental" status particularly on production data, but thanks for the suggestion.

Edit: Also, it appears that includes in --patterns-from would take priority over a lengthy list of regexes I've specified in --exclude_from, which would mean that I would need to refactor those regexes into --patterns-from, and get the order of include/exclude correct. Which, without knowing why it's marked "experimental", seems like it might further push the boundaries of complexity and increase the odds of something going wrong.

And finally, the documentation on --patterns-from is on the dicier side, understandably with experimental status. For example, the sum total of help on the syntax specific to --patterns-from, above and beyond --pattern, apparently anywhere that Google appears to be able to see, consists of this:

The patterns.lst file could look like that:

# "sh:" pattern style is the default, so the following line is not needed:
P sh
R /
# can be rebuild
- /home/*/.cache

Which leaves to inference, what P and R mean. It would seem that:

• P is some kind of preprocessor command that means "treat everything that follows as if prepended by "sh:", for shell-style matching. But this implies other questions that aren't answered, such as:
  
  
  
  • Presumably the other styles would also be valid?
  
  
  • Can you use this P keyword on any line, to change the matching style for all subsequent lines until another one is encountered?
  
  
  • Instead of P, can you specify the matching style for every line? If so, how would that syntax look like in the context of, and relative to, leading - and +?
  
  
• R seems to mean "Root", or "Relative". (Or either synonymously.) Meaning, everything sepcified below is relative to whatever R says. But similarly, unanswered questions:
  
  
  
  • Why then is "/home/*/.cache" specified with a leading "/"? Does it override P (for example if P was set to, say, /etc)? Is that only because R just means where to start scanning from?
  
  

Etc. Sure, I could test these theories and answer each question through experimental trial and error, but it would be much more assuring if there were canonical documentation on it.

(And yes I know this is floss and I could improve the docs myself. Unfortunately I invest time on other projects and it would be impossible for anyone to contribute to every open source project they use. For those who contribute, we still have to be OK being an average consumer for 99.99% of floss we rely on. Ditto with borg contributors. And so we evaluate the pros and cons of competing projects for a given solution, through that lens.)

[ThomasWaldmann]

Maybe really read borg help patterns before assuming things?

I'ld like to fix anything unclear there, but would expect the discussion starting from what is already documented.

[jim-collier]

@ThomasWaldmann Seems like we're miscommunicating. Maybe my previous reply was too long.

I wrote at length, the shortcomings, as I see them, specifically of the output of borg help patterns. Literally, "starting from what is already documented". I'm not sure why you are having heartburn over that.

Rather than saying I'm "assuming things", maybe try to understand the shortcomings that I saw as an average user, and why - which I went to some effort to think through clearly, and tried to present logically - specifically from the output of borg help patterns. And in fact, that output does not seem to be elaborated on in any other documentation that I've been able to find, e.g. Borg Documentation.

You may understand perfectly well the specific syntax and grammar expected by --patterns-from, as very lightly hinted at in the tiny example file as output in borg help patterns. But without that syntax and grammar being documented for users anywhere that I've been able to find, that output is opaque and not very useful.

Just because I don't understand the syntax & grammar expected in the file read by --patterns-from, and tried my best to infer/reverse-engineer the syntax & grammar, doesn't mean I'm "assuming things". It means I'm obviously struggling to understand how to use the unclear and incomplete output of borg help patterns. I'd much rather not have to resort to what you seem to view as "assuming things".

I understand the feature is experimental. That's all good. I don't expect it (or anything) to be perfect, bulletproof, and well-documented. My only suggestion is to think twice about repeatedly blaming the user, for doing exactly what you suggested (as I already had when you seemed to reflexively suggest it without actually understanding my concerns or that I obviously already did), when s/he finds the output of said suggestion insufficient. My biased feedback which you can consider or ignore.

[ThomasWaldmann]

It clearly says "full path" at quite some places and also explains what that means.

It nowhere says that the R recursion root specification makes patterns defined below it relative. Or that R means relative.

[ThomasWaldmann]

But I agree, there are some shortcomings in that part of the docs:

• usage of leading / in patterns - bug in docs or ignored by the code?
• ability of using a pattern type prefix directly in a pattern line
• scope of a P definition line
• explicitly saying that R means recursion root (starting point of recursion)
• extend the example pattern file to cover more of this

[ThomasWaldmann]

borg 1.2 can also read paths from stdin (e.g. like find ... | borg ...).