Nixpkgs: gnutar: unpacks from remote if filename contains colon / --force-local should be default

[andir]

Issue description

gnutar treats filenames with colons as remote filenames unless the
--force-local flag has been set. This is probably a longer (well?)
known problem with tar that could trick users into downloading remote files
instead of unpacking local (potentially verified) files when they contain a
colon.

The conversation was brought up on IRC by @ivan that linked to an Debian bug
from 2005 [1].

Recently there was the very same question on the help-tar mailing list [2]
which didn't get much/any attention as to why this behavior is still the case.

I have created a patch (for nixpkgs & gnutar) [3] that swaps the flag from
force-local to allow-remote.

An alternative to swapping the flag is removing/disabling the feature entirely
(and potentially provide a gnutarRemote package with the standard behavior).
Another approach would be using an Open/FreeBSD flavored tar but that would
cause UX breakage...

Steps to reproduce the issue can be found at the end of the help-tar post [2].

The next steps will be deciding if anything at all should actually be done and
if so what.

This issue is here to track progress and opinions.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=290435
[2] https://lists.gnu.org/archive/html/help-tar/2018-03/msg00000.html
[3] https://github.com/andir/nixpkgs/commit/23e297be8b9e861f532e1faaa8a115c57f0b7274

[ivan]

I believe the problem with tar is serious and that this kind of patch may be the least painful way to resolve it. Very few people know about tar's rmt path behavior or --force-local, which is almost never used. There are no usages of --force-local in nixpkgs, for example, even in places where the filenames are not obviously safe. Turning off rmt paths by default and inverting the option to --allow-remote will protect users from tar's rmt path behavior, while adding minimal burden to the smaller number of remote magtape protocol users. Such users, for example, can add a tar wrapper script to their PATH that adds an --allow-remote argument. I realize this may be an annoying surprise for a few, but the tar behavior is bad enough that it makes sense to protect everyone else by default.

[xeji]

Swapping the flag makes a lot of sense. But then I don't really like the idea of a nixpkgs-specific patch altering a crucial tool's behavior in a non-standard way. Any chance we can get upstream to fix this?

[andir]

IIRC upstream considers this a feature but we csn try anyway.

On Mon, 30 Jul 2018, 20:13 xeji, [e-mail] wrote:

Swapping the flag makes a lot of sense. But then I don't really like the
idea of a nixpkgs-specific patch altering a crucial tool's behavior in a
non-standard way. Any chance we can get upstream to fix this?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/44213#issuecomment-408958996,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAm_dMqjDf7MOvoEC4rh46AlDyP-4a0Wks5uL0y7gaJpZM4VllFT
.

[ckauhaus]

I'd propose to include the patch and provide a gnutarRemote attr which carries the old behaviour, so people can put that into PATH if they really need it.

But this change should get included only on master and a release note for 19.03 should be published.

[ivan]

The patch also needs to be updated to keep (and no-op) the --force-local option.

[ivan]

I looked at this again and the hard part seems to be updating all of the documentation (with the approach of "just patch it out entirely unless gnutarRemote = true"). There are things in the info pages (generated from tar.texi) that refer to remote magtape servers, remote files, rmt, rsh, and HOSTNAME:file syntax. I assume the --rmt-command= and --rsh-command= options would also need to be removed, because it looks like the : syntax is the only way to specify a remote host.

If we want to stay with GNU Tar, andir's original --allow-remote approach would make the documentation updates much more manageable, because the remote server functionality would remain in tar.

In any case, updating the --help would break the translations that come with tar and thus show some English text in non-English locales for parts of --help.

[ivan]

From some discussion in #nixos-security a while ago, there was some agreement that replacing stdenv and system-path's gnutar with libarchive's bsdtar would be worth looking into. libarchive bsdtar seems to make efforts to be gnutar-compatible, though I would really like to know about cases where that is not so. If this were implemented, I guess some new package would point a tar at libarchive's bsdtar. I would like to test a whole-nixpkgs rebuild against libarchive bsdtar at some point, but I do not have a Hydra right now.

[andir]

@ivan Can you prepare that change? I suspect we might have to move bsdtar somewhere to avoid recursive dependencies (just a feeling).

I also not have a hydra but I often run "full" rebuilds to test changes I make. We can probably ask someone with hydra access to add a new jobset once we have a PR & branch ready.

[andir]

On the case of using it in stdenv (primarily during compilation): What is the benefit besides ease of mind? We are running in a sandbox already so the network "feature" should not be an issue there. I just grep'ed through nixpkgs again and we aren't using the --force-local flag anywhere so if someone runs without a sandbox this might be a valid issue.

[ivan]

I will be travelling very soon and cannot prepare the change for at least a month.

re: stdenv, the sandbox is not enabled everywhere; I noticed it was off by default on macOS and while it could be enabled on 10.13, it was broken on 10.14 when I last checked.

[riking]

Instead of changing the name to allowRemote, how about switching it to requiring --force-local=false to enable the old behavior? This should be slightly more backwards compatible.

[ivan]

Changing the behavior of --force-local in gnutar is very involved, I started down that path a while ago but it has a lot of documentation that would need to be updated as well. I think doing that to gnutar would only make sense if upstream wanted to accept it.

[zimbatm]

I would be ready to eat my socks if any nixpkgs user is relying on that feature. Is it possible to break it on purpose with an error message like "rmt has been disabled in nixpkgs".