Nixpkgs: switch-to-configuration not interpreted using perl

[eadwu]

Issue description

Running it as an executable returns a bad interpreter error while running it explicitly with perl works fine. Running with bash also replicates the following error.

/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 3: use: command not found
/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 4: use: command not found
/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 5: use: command not found
/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 6: use: command not found
/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 7: use: command not found
/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 8: syntax error near unexpected token `('
/nix/store/v0ba874xw18j47f3hisk4130qll3wfrx-nixos-system-nixos-19.03.git.824a927/bin/switch-to-configuration: line 8: `use Sys::Syslog qw(:standard :macros);'

Steps to reproduce

nixos-rebuild switch

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.0.0-rc1, NixOS, 19.03.git.824a927 (Koi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.2pre6526_9f99d624`
 - channels(root): `"nixos-19.03pre165037.eebd1a92637"`
 - nixpkgs: `$HOME/Downloads/nixpkgs`

[dtzWill]

I encountered the same when trying out the new 5.0.0-rc1 (!!) kernel, and it appears to have been resolved when reverting.

Does that work for you? If so, then we need to sort out why it's acting this way (config? dunno!)....

[eadwu]

Seems like it is a problem with 5.0.0-rc1, reverted to 4.20.1 and the error doesn't show anymore.

[matthewbauer]

Please report this upstream! I think https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/binfmt_script.c?id=c2315c187fa0d3ab363fdebe22718170b40473e3 is the culprit but don't know enough about the kernel to fix it.

If someone is able to, verify that these cases work:

#!/usr/bin/env perl
use constant;
print 'ok';

and

#! /usr/bin/env perl
use constant;
print 'ok'

[eadwu]

Both don't work if you run them through bash, running them through perl and works though. I'll see if I can compile 5.0-rc2 fast enough before I can goto bed.

Using chmod +x beforehand also seems to make it work.

~/Downloads/tests
➜ ls
test1  test2

~/Downloads/tests
➜ cat test1
#!/usr/bin/env perl
use constant;
print 'ok';

~/Downloads/tests
➜ cat test2
#! /usr/bin/env perl
use constant;
print 'ok'

~/Downloads/tests
➜ bash test1
test1: line 2: use: command not found
test1: line 3: print: command not found

~/Downloads/tests
➜ bash test2
test2: line 2: use: command not found
test2: line 3: print: command not found

~/Downloads/tests
➜ perl test1
ok%

~/Downloads/tests
➜ perl test2
ok%

~/Downloads/tests
➜ chmod +x test1 test2

~/Downloads/tests
➜ ./test1
ok%

~/Downloads/tests
➜ ./test2
ok%

~/Downloads/tests
➜ uname -a
Linux nixos 5.0.0-rc1 #1-NixOS SMP Mon Jan 7 01:08:20 UTC 2019 x86_64 GNU/Linux

Still seems to be a problem on 5.0.0-rc1, using nixos-rebuild switch.

Still occuring on 5.0.0-rc3.

[L-as]

This problem also applies to the command-not-found script. Also, for me running the scripts directly from fish (i.e. /path/to/script and not fish /path/to/script), it will give a more accurate error about the operating system not being able to execute the files. I will post the exact error I get once I can.

[L-as]

This is what happens if I try to run the file directly from a fish shell:

Failed to execute process '/nix/store/8n8cvy1kbqzs13i5b8vnl55q50bjvr4h-nixos-system-las-19.03pre166308.626233eee6e/bin/switch-to-configuration'. Reason:
exec: Exec format error
The file '/nix/store/8n8cvy1kbqzs13i5b8vnl55q50bjvr4h-nixos-system-las-19.03pre166308.626233eee6e/bin/switch-to-configuration' is marked as an executable but could not be run by the operating system.

[dorianr666]

I confirm this bug as well, running linux 5.0.0-rc4.

[dorianr666]

So I noticed that the switch-to-configuration script has very long hashbang line and did some testing. I found out that kernel 5.0.0-rc1 - 5.0.0-rc4 (newest at this moment) starts to fail to execute a script if its hashbang line is longer than exactly 128 bytes (newline included).

Reproduction:
Run any kernel in range 5.0.0-rc1 - 5.0.0-rc4.
Create file script containing:
#!/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa (newline included)
Execute:
$ chmod a+x script
$ ./script
You'll get:

Failed to execute process './script'. Reason:
exec: Exec format error
The file './script' is marked as an executable but could not be run by the operating system.

Now remove one a from the hashbang line and run it again; you'll get the expected:

Failed to execute process './script'. Reason:
The file './script' specified the interpreter '/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', which is not an executable command.

Running both variants of the script on kernel 4.20.5 throws only the second, expected, error. So this looks like userspace-breaking kernel regression, if it affects essential NixOS scripts. Not sure what kernel dev to contact so I'll leave that up to someone else.

PS: I'm using fish shell, error messages in bash may differ but the problem remains.

[dtzWill]

Looks like they're enforcing the limit that was supposed to already be applied?

https://github.com/pypa/pip/issues/1773#issue-32376582

And indeed looking at our headers I'm seeing:

/* sizeof(linux_binprm->buf) */
#define BINPRM_BUF_SIZE 128

Scrolling to the end it looks like this is getting attention elsewhere recently as well,
likely others encountering same?

I think the commit linked previously is indeed responsible-- or certainly rather relevant (see the commit message!).

Not sure where this leaves us.... :/

[samueldr]

Has anyone contacted upstream yet? Is there a report (maybe from another project) that is relevant to follow?

Fixing it in the nixos kernels is probably not enough; kernel 5.x from other distros may still have issues with Nixpkgs stuff.

[L-as]

@dtzWill: it doesn't matter that it was a bug, kernel policy is nothing breaks userspace. @samueldr I'll try contacting upstream

[L-as]

Well I made a bug report on bugzilla, hopefully I CC'ed the right guy.

[aszlig]

I've bisected this using this VM test and the commit that causes the regression is https://github.com/torvalds/linux/commit/8099b047ecc431518b9bb6bdbba3549bbecdc343.

[aszlig]

I also think this could break more than just our use case, especially because Perl seems to expect that the hashbang is truncated by BINPRM_BUF_SIZE. It parses the hashbang and execve()s itself, so that multiple arguments are possible and the length limitation doesn't affect it, so I guess it's expected behavior even on other Unices.

So even if our bug report doesn't get noticed, I'd suspect that this change will get reverted eventually, hopefully before 5.0 gets its final release.

[dtzWill]

4.20.8 maybe has this behavior now too, maybe?

Anyway I did the obvious thing and patched BINPRM_BUF_SIZE and things
are good again--at least as long as I don't use non-NixOS machines or
something :P.

https://github.com/dtzWill/nixpkgs/commit/8dc14c61b9c414fdbfea7080a6da0c9221c8b07e
and parent (apologies).

Not suggesting nixpkgs should adopt this but thought I'd share in case
others found things acting strange after a recent update as well ;).

On Sat, 02 Feb 2019 15:30:26 -0800, aszlig [e-mail] wrote:

I also think this could break more than just our use case, because while the hashbang is truncated by BINPRM_BUF_SIZE. There is a reason why Perl parses the hashbang and execve()s itself, so that multiple arguments are possible and the length limitation doesn't affect it, so I guess it's expected behavior even on other Unices.

So even if our bug report doesn't get noticed, I'd suspect that this change will get reverted eventually, hopefully before 5.0 gets its final release.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/NixOS/nixpkgs/issues/53672#issuecomment-460008609Non-text part: text/html

[samueldr]

This might be affecting the current LTS (4.19.21) in addition to 4.20.8 (as reported by @dtzWill) and 5.0-rc*.

Assuming the commit tracked down by @aszlig is the right one:

• 4.19.y
• 4.20.y

Found also in

• 4.14.y (v4.14.99)

[grahamc]

Based on the documentation at https://www.kernel.org/doc/linux/REPORTING-BUGS and some extra courage from @mmlb, we've decided @samueldr will write a message to LKML and Linus. I'll send Oleg (author of that patch) an email now, though.

[samueldr]

Mailing list thread:

• https://lkml.org/lkml/2019/2/13/853

[NeQuissimus]

@dtzWill Could we have a PR with the patch? Even if only as a temporary mediation?

[L-as]

@samueldr I think you may want to CC Andrew Morton [e-mail], since he signed off on the commit. I'd like to CC him myself, but then I'd have to figure out how to send e-mails to the mailing list first.

[L-as]

Actually, one can just send him an e-mail regarding the matter now that I think about it.
Edit: Well I sent him an e-mail.

[samueldr]

We probably won't need #55734

• https://lore.kernel.org/lkml/20190214012754.GA17326@beast/T/#u

[dtzWill]

On Wed, 13 Feb 2019 19:18:49 -0800, Samuel Dionne-Riel [e-mail] wrote:

We probably won't need #55734

Great!

• https://lore.kernel.org/lkml/20190214012754.GA17326@beast/T/#u

\o/

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/NixOS/nixpkgs/issues/53672#issuecomment-463471641Non-text part: text/html

[vcunat]

4.14.y (v4.14.99)

Yes, I had to revert that bump in nixpkgs, for reference: https://github.com/NixOS/nixpkgs/commit/a260b5dbd5cf4bc4060778e09392b01af63c7699

[samueldr]

@vcunat only issue with the revert of 4.14 only is how users of _4_19 and _4_20 (or their aliases) may now be faced with the issue on their system once the channels pass :/

It's probably a tad late to revert the others too before an update, considering small channels updated, thus the full channels probably will update soon.

Ideas, opinions?

[NeQuissimus]

Building with the patch from https://lore.kernel.org/lkml/20190214012754.GA17326@beast/T/#u right now, I'll make a PR if this works fine

[samueldr]

@NeQuissimus you probably can make the PR while it builds; I used the patch within Nixpkgs and used the switch-to-configuration test (with #55744 applied) to validate the fix.

Though didn't think of contributing it as a patch as it's a transient issue, but then again I don't know how often the kernel do a release and how quickly they react to such situations.

[NeQuissimus]

Considering it has to go through the process, you never know :D

[vcunat]

I didn't realize connection to this issue at that time; I agree we would better somehow fix all the kernel versions fast. It might be safer to revert the kernel commit that caused this for now; I'm not sure how much review the patch has got so far. _Right now I can't promise to work on any of this._

[NeQuissimus]

Those are the two options now:

• My PR above (#55763), which we can then pick into stable + I'll remove the patch again with the next round of updates
• Revert the 4.14, 4.19, 4.20 updates

I don't really care either way...

[vcunat]

I actually meant a third option: revert only the upstream commit or its equivalent, for now, until we have a definite result from the discussion upstream. (I'm not sure if it was backported without any modifications, though it seems likely.)

[NeQuissimus]

Oh, I see... Although from what I can tell, Kees' patch is more or less that

[grahamc]

Did any channel bump including these bad kernels?

Sent from my iPhone

On Feb 14, 2019, at 08:33, Tim Steinbach [e-mail] wrote:

Oh, I see... Although from what I can tell, Kees' patch is more or less that

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[NeQuissimus]

Only the small channels have seen a bump (just looking at http://howoldis.herokuapp.com/)

[grahamc]

The nixos-unstable-small and nixos-18.09-small channel URLs are being rolled back.

[NeQuissimus]

I can now confirm that #55763 fixes the immediate issue. Built 4.20.8-hardened and booted into it. nixos-rebuild switch --upgrade -I nixpkgs=... still runs fine.

[NeQuissimus]

Does anybody feel strongly about which fix we apply as long as we have something for the moment?

[grahamc]

I think we should do this multi-pronged:

1. fix our perl shebangs to not be massive: not everybody runs nix on nixos
2. apply what looks like the most likely upstream patch

[L-as]

I think the safest choice should be taken, although a proper fix, i.e. this patch is preferable in the end.

[L-as]

good point @grahamc, but how would you make the shebangs smaller?

[grahamc]

Hydra already does this with buildEnv. Nice perk of this is we get a much faster perl startup time: https://github.com/NixOS/hydra/blob/master/release.nix#L47

[NeQuissimus]

Somebody make an executive decision or I merge #55763 :D
I don't think we should just sit on this for too long...

[grahamc]

I'd like for at least one other person to take a look at #55763 -- kernel upstream or otherwise, but it is my preferred patch.

[grahamc]

Should we leave this one open and pinned for a bit, until channels advance?

[grahamc]

FYI: Our revert patch can be deleted, as upstream has released new, fixed kernels:

• https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.9.158
• https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.14.101
• https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.19.23
• https://cdn.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.20.10

:)

[NeQuissimus]

I am on it!

[samueldr]

Should we leave this one open and pinned for a bit, until channels advance?

I didn't know for sure, and when I saw the title of the issue, once pinned, I got cold feet in re-writing the title and editing the main post to make it more useful for external users. Though, I'm thinking it could be a good idea if done. Though it doesn't necessarily needs to be kept open, only visible?

[NeQuissimus]

master: 50f518c93f4c55ca065573e3ffdcf7192a632f66 7954ec0ffdff541fcc27d89ac9b722d0505931fd 8c14948343fd678cbf639861e1792b2477a8d925 8c14948343fd678cbf639861e1792b2477a8d925

release-18.09: f0ce0f3149cdb7cf083e9a1ce93a68611c3aa13c 6e9074633490c3e8e288d73de1326f332e59344b 6da8b83c5e07a0c6a15055fd3f2ad624721ebd3b 0ba800c4b4b8f9d0cf3fc50ec2fcdf0954f04c9b

[vcunat]

Let me link the upstream summary of this: https://lwn.net/SubscriberLink/779997/11de2bdc8dbc0d69/ (taken from weekly.nixos.org)