Nix: macOS 11 (Big Sur beta 3) further locks down /

Created on 23 Jul 2020 · 34Comments · Source: NixOS/nix

The existing work around to install into /nix via a new APFS volume, and mounting as rw in vifs, no longer appears to work. The volume simply won't mount.

bug darwin

Source

ryanbooker

👀1

Most helpful comment

The drive mounts now.

ryanbooker on 21 Aug 2020

🎉13

All 34 comments

Thanks for testing this.

shepting on 25 Jul 2020

This can be temporarily worked around by disabling SIP "filesystem protections".

Reboot to Recovery OS and execute the following:

csrutil enable --without fs

knownunown on 26 Jul 2020

👎7

In my case, I needed to do: sudo rm /etc/synthetic.conf && echo "nix\tVolumes/Nix" | sudo tee -a /etc/synthetic.conf
and removing the line about nix inside /etc/fstab

sycured on 26 Jul 2020

In my case, I needed to do: sudo rm /etc/synthetic.conf && echo "nix\tVolumes/Nix" | sudo tee -a /etc/synthetic.conf
and removing the line about nix inside /etc/fstab

I originally considered this approach. This turns /nix into a symlink to /Volumes/Nix, which works for accessing things already in your nix store but breaks on adding packages (try nix-shell -p hello --run hello or something similar.) It's mentioned in the docs as an unsupported configuration. Messing with SIP definitely sucks but there are downsides IRT reproducibility when the realpath of the store isn't rooted at /nix.

Alternatively, you can stick with the symlink and set the environment variable NIX_IGNORE_SYMLINK_STORE, but keep the consequences in mind :)

knownunown on 27 Jul 2020

No problem about mounting on Beta 1 and 2, so wait and see for the final release because Apple can revert this change.

Otherwise, next step will be moving back to homebrew because I refuse to degrade SIP if NIX_IGNORE_SYMLINK_STORE doesn't work well but also because nix doesn't provide a way to install it at custom location.

One possible way is what I did, having a macOS Catalina in vm on another computer and syncing the /nix across the network but it's not very elegant.

sycured on 27 Jul 2020

@arianvp said that we might get a firmlink for /nix, which would solve this issue.

Mic92 on 27 Jul 2020

Otherwise it might be time for nix to switch to a different store path on macOS?

Mic92 on 27 Jul 2020

My exact words were that somebody at Apple poked me on the firmlink issue on twitter. I don't know whether they will actually be able to help; but I did sent them a link to this issue.

Lets hope if they can help us =)

arianvp on 27 Jul 2020

Tweet in question for the record: https://twitter.com/ProgrammerDude/status/1285858829155151873

arianvp on 27 Jul 2020

Could someone file an issue at https://feedbackassistant.apple.com with their Apple Developer account on the firmlink issue, and then share with the kind people in this twitter thread? https://twitter.com/rballard/status/1287786882688880651?s=20

arianvp on 27 Jul 2020

Could someone file an issue at https://feedbackassistant.apple.com with their Apple Developer account on the firmlink issue, and then share with the kind people in this twitter thread? https://twitter.com/rballard/status/1287786882688880651?s=20

I have done so, as FB8179445. I doubt you'll be able to see the URL, but it's here:

https://feedbackassistant.apple.com/feedback/8179445

However, the Apple engineer who's been replying has requested that I file a separate Feedback for the Big Sur beta 3 behavior as described in this GitHub issue. I'll try to do that if nobody beats me to it, but I'm not currently running Big Sur, so it might take me a few days.

dhess on 27 Jul 2020

❤1

The fact that mounting a volume using synthetic.conf in Big Sur isn't working seems like a bug, and Apple is aware of it. It remains to be seen whether it will be fixed, but I'm hopeful, as this was one of the two primary use-cases for synthetic.conf and it would be pretty bad for Apple to introduce this mechanism in one release only to break it in the next.

lilyball on 28 Jul 2020

In Big Sur everything on the system volume is protected, paths that are user editable like home, etc, tmp etc, are links out to another volume. Maybe Apple will let this through the new wall, or provide a mechanism to create a link that works the same way as home etc, or maybe it is just a bug in Beta 3, and synthetic.conf mounted volumes should be rw. But it does seem at face value to make sense given the tighter restriction they've added to Big Sur.

ryanbooker on 28 Jul 2020

@ryanbooker What you describe sounds like Catalina. It's unclear to me what's changed in this regard for Big Sur, besides the volume mounting bug.

lilyball on 28 Jul 2020

https://eclecticlight.co/2020/06/25/big-surs-signed-system-volume-added-security-protection/

ryanbooker on 29 Jul 2020

I read through the eclectic light post a few weeks ago, and again now.

I won't try to proclaim, having not tested, but both times it didn't seem to me like the signing change it focuses on should have any impact on the Nix install approach (because it isn't modifying the system volume).

abathur on 29 Jul 2020

I just read that post, and it sounds like the changes here aren't actually related to being unable to mount a volume rw on a path listed in synthetic.conf, as none of this touches the system volume. It definitely just seems like a bug.

lilyball on 29 Jul 2020

I hope you're right and it's just a bug. I was assuming it's because the mount point is on the system volume and nothing on the system volume is allowed to be rw (at least that was my understanding). Whether that's just a mistake in B3 or an intended change is an open question, I suppose.

ryanbooker on 29 Jul 2020

👍1

@ryanbooker do you have an Apple Developer account, and would you want to file an FB for this _specific_ issue and share it here for documentation-sake?

In the FB you can say it's a duplicate of radar 66170165 which is the internal number under which this issue is being tracked at apple. (See https://twitter.com/rballard/status/1287809402385207296)

arianvp on 29 Jul 2020

Yep. I had already filed one, but I've marked it as a dupe. FB8112166

ryanbooker on 3 Aug 2020

👍1

FWIW, the (probable?) regression is still present on Beta 4.

$ csrutil status
System Integrity Protection status: enabled.
$ cat /etc/synthetic.conf /etc/fstab 
nix
run private/var/run
#
# Warning - this file should only be modified with vifs(8)
#
# Failure to do so is unsupported and may be destructive.
#
LABEL=Nix /nix apfs rw,nobrowse
$ diskutil ap unlockVolume disk1s5
Unlocking any cryptographic user on APFS Volume disk1s5
Error unlocking APFS Volume: Couldn't mount disk (-69842)
$ diskutil info disk1s5
...
   Volume Name:               Nix
   Mounted:                   No
...
   FileVault:                 Yes
   Sealed:                    No
   Locked:                    No

knownunown on 5 Aug 2020

👀1

If anyone listening/reading is in the DTK program, I'm curious if there are other reports/discussion about this in the DTK forum that supposedly exists?

abathur on 6 Aug 2020

Note that I and @grahamc have a contact person at Apple's developer relations that can help us approve DTK access requests for companies with nix contributors. If you're interested please reach out to us. Another option is for NixOS organization to request access to MacStadium cloud machines (free of charge)

I think @grahamc also requested one for the NixOS Foundation but I am not sure

The original email that I got:

Firstly, the most expedient way of obtaining a DTK would be for companies who have Nix maintainers and are registered as Apple developers can learn about the Universal Quick Start Program and apply for a DTK here:

https://developer.apple.com/programs/universal/

If this is the way your maintainers want to go, I would be happy to approve the applications to purchase the program. Someone noted on the thread they had applied and not been approved. I have approved then. Please let me know if there are any others.

Secondly, we have a relationship with MacStadium where qualified open source projects can simply get access to a DTK in the cloud. The maintainers of Nix should email [email protected].

The process for qualified OSS developers will be:

Developer signs up for a general MacStadium account so the MSA can be accepted.
MacStadium will manually add the DTK asset to their account at no cost.
Developer will have access to screen sharing on a DTK and can log in right away

arianvp on 6 Aug 2020

The drive mounts now.

ryanbooker on 21 Aug 2020

🎉13

Heise (German Tech newspaper) just reported that apple will only allow signed code. Can that work with binary caches?

(I'm not yet on apple and I think of waiting to go to apple untill the arm is available)

Source: https://www.heise.de/news/Apple-ARM-Macs-fuehren-nur-noch-signierten-Code-aus-4875048.html

Kloenk on 21 Aug 2020

Heise (German Tech newspaper) just reported that apple will only allow signed code. Can that work with binary caches?

(I'm not yet on apple and I think of waiting to go to apple untill the arm is available)

Source: heise.de/news/Apple-ARM-Macs-fuehren-nur-noch-signierten-Code-aus-4875048.html

We likely cannot add signing keys to the nix-build in a reproducible manner. Also the clang used in nixpkgs won't have those keys as well. Everyone on that platform would need to compile packages themselves. What will better work is to run NixOS in their VM similiar to WSL - it may even be faster for certain workloads - macOS system calls are not known to be very performant.

Mic92 on 21 Aug 2020

The drive mounts now.

Do you mean that in the latest beta, this is no longer an issue?

dhess on 21 Aug 2020

Shall we open a new issue for ARM compatibility? It seems to be a bit of a separate discussion

arianvp on 21 Aug 2020

👍5

I also just tested it in beta 5, the volume mounts now just like in Catalina.

printfn on 21 Aug 2020

Note that I and @grahamc have a contact person at Apple's developer relations that can help us approve DTK access requests for companies with nix contributors. If you're interested please reach out to us. Another option is for NixOS organization to request access to MacStadium cloud machines (free of charge)

I think @grahamc also requested one for the NixOS Foundation but I am not sure

The original email that I got:

Firstly, the most expedient way of obtaining a DTK would be for companies who have Nix maintainers and are registered as Apple developers can learn about the Universal Quick Start Program and apply for a DTK here:
https://developer.apple.com/programs/universal/
If this is the way your maintainers want to go, I would be happy to approve the applications to purchase the program. Someone noted on the thread they had applied and not been approved. I have approved then. Please let me know if there are any others.
Secondly, we have a relationship with MacStadium where qualified open source projects can simply get access to a DTK in the cloud. The maintainers of Nix should email [email protected].
The process for qualified OSS developers will be:
Developer signs up for a general MacStadium account so the MSA can be accepted.
MacStadium will manually add the DTK asset to their account at no cost.
Developer will have access to screen sharing on a DTK and can log in right away

.
@arianvp
Note that the NixOS foundation got ahold of a ARM mac mini now (DTK), and I am in the process of arranging some access for @edolstra initially. Let's discuss this further in a new issue. Did you create one already?