Restic: Add '--files-from-raw' flag that treats every line literally, processes in exactly that order, and with no comments or globbing

[jim-collier]

What should restic do differently? Which functionality do you think we should add?

restic --files-from <file> does not work like most other similar flags in other programs - notably, "industry-standard" rsync, or rclone.

(That "problem" is already documented in other issues, including #3005 which I opened. But that issue is contentious, not going to happen, is a losing battle, and can/should be closed.)

restic --files-from <file> does not treat <file> as a raw dump of file specifications as typically generated by other utilities (e.g. chaining in more powerful find, sed, awk, and/or grep into a scripted workflow).

Instead restic --files-from <file>, treats each line as potentially hand-crafted globbing specifications. In other words, it's literally the functionality of what many other programs provide a --patterns-from or --includes-from option for. (Including rsync, rclone, and borg.)

This has already caused numerous issues to be opened or tengentially mentioned (enumerated below), e.g. because brackets in filenames causing problems. (The solution being, ironically, to chain together a scripted workflow which first escapes all globbing characters from the filenames in <file>.)

However, as others have pointed out, changing this logic now might horribly break existing user scripts that depend on the globbing functionality of --files-from.

So it seems that the only solution is to build a --files-from-raw <file> option, that operates _exactly_ as [rsync](https://linux.die.net/man/1/rsync) --files-from does.

Rclone provides both --files-from and --files-from-raw flags, so there is some industry precedence.

(In rclone's case though, --files-from behaves much more like rsync's --files-from, than restic's. It just allows comments and non-filename spaces, which their --files-from-raw option does not.)

Rclone's --files-from-raw behaves exactly as rsync's --files-from does, as restic --files-from-raw should also behave:

• Every line is _an exact 1:1 file specification_. No globbing.
• Leading and trailing spaces, tabs, etc. are considered part of the filename.
• Files are processed (backed up) exactly in the order they appear in <file>
  
  
  
  • This is a very important part of this feature, as the file order may be very particular and exact, e.g. already sorted by the user on date, size, etc.
  
  
• Comments or empty lines aren't "allowed".
  
  
  
  • More specifically, it just might not exist as an exact, literal file path - in which case it would be ignored, with a warning emitted.
  
  
  • But it might also unintentionally point to an explicit filename, e.g. ## my supposed coment but is actually a filename.
    
    
    
    
    
    • Which is why it should be stated in the docs that "comments aren't allowed".
    
    
    
  
  
• File path specifications are delimited with newline.
  
  
  
  • It should be noted for completeness, that newlines _can_ be in linux filenames as an extreme edge case.
  
  
  • But this extreme edge case is also not handled by any other utility that I'm aware of;
    
    
    
    
    
    • _Explicitly_ including rsync and rclone. In those cases, newlines in filenames must be dealt with in other ways.
    
    
    
  
  
  • In all such cases (including proposed behavior for --files-from-raw), newlines in filenames might not match anything, or worse, match something unintended. (Because only the individual parts of the filename would be looked at).
  
  
  • Anecdotally, newlines in filenames are almost always accidentally or due to program bugs.
  
  
  • It's easy enough to find a bash one-liner on stackexchange that changes newlines in filenames to something else.
  
  
  • I think this "problem" already affects most of the rest of restic anyway. Almost universally across time and space, the solution is to remove newlines from filenames before processing files.
  
  

Additionally, restic --files-from-raw <file> should default to refer to the previous snapshot as parent, if no other snapshot-related arguments or tags are specified. (Note: I don't know how tags work so that part may not make any sense.) Because:

• The current behavior of restic --files-from <file> (#2246, #3004) isn't appropriate for --files-from-raw <file>. Specifically, the behavior of there being no parent snapshot, if the contents of <file> changed since last run.
• Because the whole point of --files-from-raw <file> is that _the contents of <file> are likely to be different from run to run_. Arguably that's how --files-from should work too, but at least that can and should be fixed with --files-from-raw.

How is this any different than similar issues?

• Don't treat entries in --files-from as glob definitions #3005:
  
  
  
  • I opened it. It's a losing battle and isn't going to happen. (For what I recognize as good and valid reasons.)
  
  
• Paths in --files-from files that include brackets aren't backed up #2448:
  
  
  
  • Similar to #3005 (or the other way around).
    
    
    
    
    
    • Proposes fixing --files-from, rather than a new feature of --files-from-raw.
    
    
    
  
  
• Support backing up user-provided list of files #2944:
  
  
  
  • A broader superset with some overlap with #3005, also complains about the behavior of --files-from.
  
  
  • Also asks for support of a JSON array, and null-terminated filenames.
  
  
  • Doesn't ask for a new feature specifically to behave _exactly_ like rsync --files-from <file> nor in as much specifically detailed behavior.
  
  
• --files-from: "pattern ... does not match any files, skipping" / "syntax error in pattern" #2276:
  
  
  
  • Also complains about behavior of --files-from.
  
  

What are you trying to do? What problem would this solve?

Perhaps millions or surely at least thousands of system admins and power users rely on other utilities to generate a raw list of files, that is then fed into bulk-ey batch-ey file processing utilities such as rsync. As part of scripted batch solutions.

Or at least:

• This is the way I personally deal with processing a large number of files that need advanced include/exclude rules, e.g. for local backups, and how I need to work with restic.
• This is how I operated as a sysadmin back in the day.
• This is how many of my peers operated.

With this new feature - that also doesn't disconnect itself from parent snapshots - I would finally have the One Backup Solution To Rule Them All, and could sleep easy at night, and could use it for our specific business use case! ;-)

Did restic help you today? Did it make you happy in any way?

I feel like Jerry Maguire's girlfriend: Restic is SO CLOSE to being what it is capable of! :-D

And either way, great dev and user community.

[rawtaz]

Jim, with all due respect, please lower the signal to noise ratio. Look at the size of your post above. It is huge, way beyond what is reasonable, and contains a lot of commentary that is simply not needed to convey the core of the suggestion at hand. There is no need to write this much text, instead please keep it short and concise and to the point. I think one way to rephrase what I'm saying is that you don't have to elaborate every single bit of your thoughts - we hear you, and will also ask if there's anything we think is unclear, as we try to make sure we understand what's being communicated :) Thank you.

[jim-collier]

@rawtaz I couldn't disagree more.

• I closed #3005 which covered the same ground but in a losing attempt to change an existing feature. This is a new feature.
• I'm trying to be very explicit in what the feature should do. I have very specific functionality in mind, and although I'm not implementing it, nor have the experience in Go to do so, now is my only change to have input on that process, rather than complaining about it after the fact.
• From your template asking detailed questions, it seems as if that is valued over vague, incomplete, and poorly researched issues.
• This specifically has been an ongoing problem with at least half a dozen related issues and spanning many months - but none proposing a specific solution (other than changing existing behavior of --files-from). Seems to me at least, to be a pretty important concern in the community that deserves a detailed, fine-grained, very explicit proposed solution. And in this case, based on the feedback I've read from restic devs in other issues, if not detailed very carefully in the request, would likely propagate many of the same peripheral complaints that have also tangentially accompanied --files-from (e.g. no snapshot parent, not following same listed order, etc.). There may be perfectly good reasons for that, but I feel they at least need to be pointed out so that the decisions to do it differently are thought about, and purposely decided on, rather than just left to inherit some existing behavior. (I've also been a programmer for over a decade. I get that.)
• I'm a product designer, as part of my job. Trust me, this is _shorthand_ for feature specifications. You're probably aware of this as well. If you prefer, I could submit an actual specification. Something tells me you'd like that even less.
• When I've commented specifically on --files-from, people seem to take it personally and reject suggestions outright, or get sidetracked on unrelated gripes. (But not changing an existing feature is perfectly legit. Hence new feature.) I'm trying to head all that off by addressing objections ahead of time. Maybe you work better with the same amount of text but in back 'n forth format. Unfortunately personally, I don't have enough time for that throughout the day, only in blocks.
• I'd like to use restic for the business that employs me, and for whom I make such decisions. Currently I can't. This feature, and whenever prune speedup makes it into master, it will be The One. You and many others may not care, but I think that would still be a small symbolic win for the restic community. (And certainly I would rest easier.)

So help me out: what parts, exactly, do you specifically find to be "noise"? I'm happy to delete words/paragraphs if you feel it's redundant with itself, or already well understood by the community and doesn't need to be relitigated. But most of the word count is defining feature specification and behaviors. That said, it's also fairly common industry practice for feature requests and bug reports to be self-contained and complete, if it's not explicitly building off of something else.

[ProactiveServices]

Perhaps the forum is a better venue for in-depth discussion.

[jim-collier]

@ProactiveServices

Perhaps the forum is a better venue for in-depth discussion.

I'm not clear on what part of this thread so far you're addressing. Are you suggesting the initial feature request shouldn't point out in detail what --files-from is perceived to get wrong, and what behavior, in detail, the new feature request should embody? Or are you addressing the follow-on comments? Thanks.

[rawtaz]

@jim-collier I wrote up a suggestion of how the first post in this issue could look instead (see below). It's much shorter but I have kept pretty much everything from your post that seemed relevant (to me of course, it's all individual), and trust me when I say I read through all of what you wrote in order not to miss any part of it.

I understand and value your thinking when you're just trying to be specific and detailed with the best of intentions, I totally understand that. I'm just commenting because as you can see there is a lot of text, and I truly think that we hear your request even if you don't go into detail. FWIW I think everyone understands the problem and the proposed solution by now. All I'm saying is that it would just be great if you could try to think about keeping the amount of text down a bit in general (not just here). If there's anything unclear about the suggestion/issue we can always address that in the comments 👍

Since this is off-topic in terms of the issue, I'll go ahead and hide both this and the previous related comments in this issue as off-topic, which will make them hidden but still viewable if one wants to.

---

What should restic do differently? Which functionality do you think we should add?

restic backup should accept an option --files-from-raw <file> which works like the current --files-from <file> but instead of globbing the lines in the specified file parses them exactly as they are written, verbatim.

What are you trying to do? What problem would this solve?

It's very common, often as part of automated/scripted setups, to have tools generating a raw list of files to back up, which is then given to restic backup using the --files-from option, which applies globbing to each line to resolve the filename to back up. When one or more filenames in this list contains characters that have a special meaning in the globbing, e.g. brackets, dollar signs or starting/ending with a space, these files might end up not being included in the backup because the result of the globbing no longer matches the actual file path/name on disk.

This problem has already been mentioned or discussed in #2276, #2448, #3005 and #2944 (which is perhaps closest to a suggested specification besides this issue). While it's possible to perform pre-processing of the list of files that will be fed to restic backup using --files-from, this is error-prone (one could easily miss some corner-case) and unnecessary work, when there could instead be a --files-from-raw option that parses the list of files verbatim.

Since changing the current behavior of --files-from would break backwards compatiblity, I instead propose adding the --files-from-raw version, like rclone already did and like rsync's --files-from work. Please see the suggested specification below. For my environments and use cases, this feature would be the key to making restic the perfect tool for backing up my data.

Suggested specification

I suggest that --files-from-raw reads and resolves the lines in the specified file in the following way:

• Every line is _an exact 1:1 file specification_ - No globbing.
• Leading and trailing spaces, tabs, etc. are considered part of the filename.
• Files are processed (backed up) exactly in the order they appear in <file> (this is very important, as the order may be very intentional, e.g. already sorted by the user on date, size, etc).
• Comments or empty lines aren't "allowed" - A comment sign such as # will be interpreted as part of the filename, and empty lines ignored (possibly with a warning emitted).
• File path specifications are delimited with newline - In case a filename actually contains a newline, this is one exception that --files-from-raw won't accommodate and the filename will have to be corrected on disk or those files included in other ways.

Additionally, restic --files-from-raw should default to refer to the previous snapshot as parent, if no other snapshot-related arguments or tags are specified, in order to let the user help restic figure out which files don't need to be scanned again.

Did restic help you today? Did it make you happy in any way?

I feel like Jerry Maguire's girlfriend: Restic is SO CLOSE to being what it is capable of! :-D

And either way, great dev and user community. I just wish I could have this feature in restic so I can integrate it with my environment.

[rawtaz]

I think this is a feature that is clearly needed. I totally don't think that people should have newlines in their filenames, but the other things like special/globbing characters in the filenames, as well as starting and ending spaces/tabs, is something we simply need to deal with at some point.

I have a lot of users that just can't properly name files, to the extent that filenames are a complete mess, so even though I think that starting/ending spaces should be cleaned up, and certain other characters don't belong in filenames, that's not how reality works. For this reason I support this feature request.

[jim-collier]

@rawtaz

I wrote up a suggestion of how the first post in this issue could look instead (see below)

Thanks for the constructive effort in providing a better example. Point taken.

[greatroar]

Disallowing newlines is not a universal convention. GNU find and xargs are equipped with -print0 and -0 flags to deal with newlines in path names: they NUL-terminate them, knowing that the NUL byte is the only byte not allowed in Unix path names. The POSIX standard xargs allows quoted path names in its input for pretty much the same reason. Git uses NUL-terminated strings in its tree format.

If the purpose is to backup from a tool-generated list, the universal solution for Unix would be to accept find -print0 output.

Telling the user to clean up their path names is missing the point: you want to run a backup beforehand, because automated cleanup is error-prone.

[rawtaz]

Sure. But at the same time; What's an actual valid use case for having newlines in your filenames? And does anyone ever actually have that in their filenames? I mean, seriously.

[greatroar]

I've seen filenames with newlines a few times. They were always the result of bugs (lack of input validation, using the wrong field to generate a filename). The thing is, such filenames do occur and I don't like it when they break my tools.

[rawtaz]

I'd say that if you end up with such filenames you'd notice it and fix it as part of whatever process you're doing that produce these files, otherwise you have way bigger problems with your processes than a backup software not taking those filenames into account. Not to mention that if you have this problem, you're unlikely to successfully produce a proper include file in the end anyway. But i digress.

If this is something we want to support, then I suggest that we simply make --files-from-raw use only the NULL byte as line separator, which effectively makes it 100% purely treating the entire file fed to it verbatim and raw - then there's no question at all about what it does, how it processes the lines, and there's no way it can fail to process a properly formatted include file. This should cover every possible use case I suppose.

[pvgoran]

I'd propose to introduce both --files-from-raw (newline separators) and --files-from-raw0 (null byte separators). This should satisfy everyone, sound familiar to find users, and would probably not cost much in terms of code complexity.

[smlx]

The other option here is to add secondary flags that affect the behaviour of --files-from.

--files-from --from-noglob
--files-from --from-0

rsync has a similar feature for null terminated filenames:

--from0, -0
       This tells rsync that the rules/filenames it reads from a file are ter‐
       minated by a null ('\0') character, not a NL, CR, or CR+LF.   This  af‐
       fects  --exclude-from,  --include-from,  --files-from,  and  any merged
       files specified in a --filter rule.  It does not  affect  --cvs-exclude
       (since all names read from a .cvsignore file are split on whitespace).

[rawtaz]

Please let's not complicate this and add a bunch of new flags. This new feature is supposed to be designed for automated/scripted data, and if that's what one is doing then one should have no problem at all separating the filenames with a NULL byte instead of a regular newline. Hence, there's no point in supporting a non-NULL version of it.

[jim-collier]

If this is something we want to support, then I suggest that we simply make --files-from-raw use only the NULL byte as line separator

I'm not a fan of this idea. I agree with your earlier statement of

I'd say that if you end up with such filenames you'd notice it and fix it as part of whatever process you're doing that produce these files, otherwise you have way bigger problems with your processes than a backup software not taking those filenames into account.

However: If null byte delimiters are what it takes to see this feature make it, I can deal with a null delimiter! And since that is in fact only one of two characters that are impossible in every *nix I'm aware of, it would at least placate those that either want newlines in their filenames (unlikely), or don't want to deal with it beforehand. It could even be argued that it would make this option more "correct".

(At minimum this debate has been enlightening for me personally, because I've never once considered or dealt with newlines in filenames, in my utilities. It wouldn't break a single one, at worst result in the set of files dealt with not exactly matching what was expected.)

[greatroar]

I'm working on a PR. May I suggest putting the 0 in the flag name to be perfectly explicit? --from-files-raw0? --from-files0?

[rawtaz]

It makes no sense to put a 0 in the option name to indicate that it uses NULL to separate filenames. An option should be named after what it does, not how it does it. In this case, the feature is about reading files in a "raw" or "verbatim" way, it's not about reading filenames with a NULL character as the separator. That's why --files-from-raw (what it does) is more appropriate than --files-from0 (how it does it). I suggest just keeping it simple and naming the option as discussed before. It's up to the description of the option and documentation for the software to explain how it does things.

[greatroar]

This is not an implementation detail. This is specifying which file format the user has to supply. It's also consistent with find, xargs, rclone, rsync and Git (except git ls-files spells zero "-z").

[rawtaz]

It's not needed. What makes you think it's needed? Do you think anyone will use --files-from-raw without first having read what it actually does (and in that noted that filesnames should be separated by NULL)? Besides, it sure is an implementation detail, especially when you remove the "raw" part which is arguably a much more prominent and important part of describing what the option does.

[pvgoran]

In context of restic, --files-from-raw is better than --files-from0. Because the latter is perceived as a modification of --files-from "with zero byte separetor", whereas in fact it would be a significantly different functionality (verbatim filenames vs. glob patterns), so a more prominent difference in naming is needed.

Then again, I already suggested using --files-from-raw0, which would both sufficiently differentiate the new option from --files-from, and give a hint on null byte separators. :)

[greatroar]

@pvgoran Good point.

@rawtaz I'm not saying it's needed. It's just more similar to what other tools do and it's self-documenting, while --files-from-raw suggests the same behavior as rclone, but rclone wants newlines. Do you object against the 0, or against not including -raw?

[rawtaz]

Do you object against the 0, or against not including -raw?

If there's only one --files-from-raw then I object against adding a 0 onto it, because details such as how separation is done in the files fed to restic through that option is not something that should be described by the naming of the option. Regarding not including -raw, that's not even an option (no pun intended :P) for the reasons mentioned by @pvgoran.

We have some thoughts on how to proceed with this, stay tuned ;)

[rawtaz]

Alright so @fd0 and @MichaelEischer and I have agreed on what we think is the way forward with this feature request and its implementation. We'd like to do the following:

• Implement a new --files-from-raw <file> option that reads file paths 100% verbatim, without any globbing or parsing, from <file>. With this option, the file paths must be separated only by a regular \n newline character.

• Implement a new --files-from-raw0 <file> option that reads file paths 100% verbatim, without any globbing or parsing, from <file>. With this option, the file paths must be separated only by a \0 NULL byte.

By 100% verbatim we mean that every single byte/character listed between the separators will be included and used in the file path, even things like #, whitespace (besides the separator), etc, regardless of where in the file path it's located. The only part that will be parsed and not included in the file paths are the separators (only one per option).

We believe that this is the only solution that will truly accommodate every possible use case of this type while at the same time allowing for the newline/NULL flexibility that seems to be needed in the CLI. With these two options, one should be able to feed any file path to restic successfully.

Does anyone object to this, and if so with what rationale? Also, if I missed anything in the description/specification above, please let me know :)

@greatroar Can you update your PR to match the above two points? There's some comments regarding the description of these options that I'll provide when you've done so :+1:

[greatroar]

@rawtaz I have two rather subtle remaining points.

You ask for separators, but I implemented terminators. The distinction is that between "foo\x00bar" (separated) and "foo\x00bar\x00" (terminated). The latter option has one less corner case (missing final NUL), matches what find -print0 produces and allows us to give a clear error message for when the user passes the wrong file ("foo\nbar\n" does not contain any NUL characters). I guess that for newlines, this matters less.

Regarding newlines, do we allow "\r\n" too, for the convenience of Windows users?

[DRON-666]

Regarding newlines, do we allow "\r\n" too, for the convenience of Windows users?

--exclude-file and --files-from allow "\r\n" and i don't see any reason why --files-from-raw must work differently.

[greatroar]

Those options do a lot of things differently from the proposed ones. But TBH, the reason I'm asking is because it takes more code to handle CRLF.

[DRON-666]

But TBH, the reason I'm asking is because it takes more code to handle CRLF.

This code will be identical to readLinesFromFile, but without comment/space trimming.

[fd0]

Regarding newlines, do we allow "\r\n" too, for the convenience of Windows users?

The code for --files-from uses bufio.ScanLines here, which supports \r\n as well as \n. It should be the same for the new --files-from-raw. :)

The distinction is that between "foo\x00bar" (separated) and "foo\x00bar\x00" (terminated). The latter option has one less corner case (missing final NUL), matches what find -print0 produces and allows us to give a clear error message for when the user passes the wrong file ("foo\nbar\n" does not contain any NUL characters). I guess that for newlines, this matters less.

I'd like to return an error message when no Null bytes are present, so I'd go for the "terminator" approach here.

[rawtaz]

This would mean that -raw0 requires an ending \0 on the very last line, whereas -raw does not require an ending newline on the very last line. That's inconsistent.

[DRON-666]

It's inconsistent only because you decided to consider non terminated lines as error.
rclone uses same simple function for both --files-from-raw and --files-from.
Even rsync uses same code (if (eol_nulls? !ch : (ch == '\n' || ch == '\r'))) for all line ending types.

[smlx]

It's inconsistent only because you decided to consider non terminated lines as error.

POSIX also considers this an error: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_206

[greatroar]

I've now implemented CR/LF endings but no check for final newline in #3017. Adding that check wouldn't be hard and I don't care either way, so please decide :)

[rawtaz]

The distinction is that between "foo\x00bar" (separated) and "foo\x00bar\x00" (terminated). The latter option has one less corner case (missing final NUL), matches what find -print0 produces

It's amendable to match that, but we don't need to, because if -raw0 uses separators instead of terminators, and is fed a file from -print0 produce, so the data is e.g. foo\x00bar\x00, then -raw0 will see three "entries", namely foo, bar, and `. We should obviously silently ignore empty (zero length) entries, so the last one which is due to the terminating \0 from-print0, will just be ignored. Hence there's no need or value in matching the type of output-print0` produces, since there's no negative impact in not doing so.

and allows us to give a clear error message for when the user passes the wrong file ("foo\nbar\n" does not contain any NUL characters). I guess that for newlines, this matters less.

I'd like to return an error message when no Null bytes are present, so I'd go for the "terminator" approach here.

Just to be clear; If we make -raw0 require terminators, will it give an error message when it's fed a file like foo\x00bar\x00star? Also, will it give an error message when fed something like foo?

[greatroar]

will it give an error message when it's fed a file like foo\x00bar\x00star? Also, will it give an error message when fed something like foo?

Yes, on both counts. More importantly, it gives an error message for foo\nbar\n, so it can catch a file in -raw format early. Empty filenames (\x00\x00) are treated as an error as well, because no system permits them and they may point to bugs in the script that generated the names. Vice versa, --files-from-raw checks for zero bytes and issues an error messages suggesting --files-from-raw0.

[rawtaz]

What I don't like about this (-raw requiring only separators but -raw0 requiring terminators) is that the only practical reason/use for/of it is to give a better error message than the user would get if they fed a newline-separated file into -raw0 (where restic would tell them a potentially long filename couldn't be found). That's really the only reason we introduce inconsistency in the options, and I think that noone will ever not notice their files not being backed up properly when they accidentally give a file to -raw0 instead of -raw. So I think this to be extremely limited practical use, hence it doesn't feel like it's worth the cost of an inconsistent CLI.

But I think it is what it is and we have to do it anyway. Also @fd0 and @MichaelEischer think it's needed. We'll just have to really designate the -raw0 version as an option for very specifically formatted/generated data, whereas -raw is more allowing and for potentially sloppy hand-crafting.

Does your code for -raw currently silently ignore empty entries (e.g. from foo\n\nbar) or does it give an error and refuse those? I think that if we want this option to be the allowing type it should silently ignore them, right?

[greatroar]

-raw also errors out when seeing an empty line in the PR.

[rawtaz]

I don't see the point of that, am I missing some good reason for it? It's one thing to not silently ignore empty lines/entries in -raw0 because it's expected to be 100% perfect as generated by a script or similar, but since we know that an empty string will never be a valid filename, why not just ignore it in -raw.

[greatroar]

Consistency, but I guess that argument is out the window :smile:

Allowing empty lines makes it easier to hand-edit file lists because files can be grouped. Not allowing them makes it easier to debug automatically generated lists. Your call.

[rawtaz]

@greatroar We will go with --files-from-raw silently ignoring empty lines/entires (e.g. foo\n\nbar will not complain about anything). Thanks.

[greatroar]

Ok! I've updated #3017, please review there.