Home-manager: paths referenced with `toString` are still copied into the nix store.

[cprussin]

Given a configuration line like the following:

{
  home.file."some-file".source = toString ./some-file-source;
}

I would expect ~/some-file to be symlinked to ./some-file-source (perhaps with some indirection), but not to have ./some-file-source copied into the nix store. However, in practice, I'm seeing ./some-file-source copied into the nix store.

This appears to work as expected with nixos configuration, but not with home-manager. Also, I'm fairly sure this regressed at some point somewhat recently, as I noticed this on a configuration I wrote some time ago and that I know was doing the right thing at the time (I use it to set up a symlink to a data store and when the data is copied into the nix store, it becomes read-only, so I know it used to work because when it fails it's quite loud).

One thing that might be related is that I somewhat recently switched from using a separate home-manager config to using home-manager as a nixos module. However I'm fairly confident this was working for some time after I switched, but I'm not certain.

I'm happy to do more digging/debugging if anyone can point out anything I can do to help provide more info!

[cprussin]

Upon doing a bit of digging, I think that the commit that broke this is likely https://github.com/rycee/home-manager/commit/4bed99c71cc5a671594f879d1aab4a50ae40a3a1. @rycee any advice on the right fix?

[rycee]

If you want to create a link to a file outside the Nix store then for now the correct solution would be to create an activation block along the lines of

home.activation.linkMyFiles = config.lib.dag.entryAfter ["writeBoundary"] ''
  ln -s ${toString ./some-file-source} ~/some-file
'';

Sorry for breaking your configuration! But unfortunately the old behavior had to be sacrificed to support a wider range of file names. Also I would argue that the old behavior was undesirable in that it prevented reproducibility and it was very easy to accidentally make the file not be copied to the Nix store.

At some point it may be nice to add an option that clearly has the purpose of establishing links to files outside the Nix store, but for now I find the activation script trick suitable verbose and discouraging to make one think twice before using it 🙂

[sboosali]

Also I would argue that the old behavior was undesirable in that it prevented reproducibility and it was very easy to accidentally make the file not be copied to the Nix store.

+1

[cprussin]

If you want to create a link to a file outside the Nix store then for now the correct solution would be to create an activation block along the lines of

home.activation.linkMyFiles = config.lib.dag.entryAfter ["writeBoundary"] ''
  ln -s ${toString ./some-file-source} ~/some-file
'';

Thanks for the tip! That's super helpful!

Sorry for breaking your configuration! But unfortunately the old behavior had to be sacrificed to support a wider range of file names. Also I would argue that the old behavior was undesirable in that it prevented reproducibility and it was very easy to accidentally make the file not be copied to the Nix store.

At some point it may be nice to add an option that clearly has the purpose of establishing links to files outside the Nix store, but for now I find the activation script trick suitable verbose and discouraging to make one think twice before using it

I don't disagree that it's a weird piece of nix. And actually I was originally planning to propose exactly what you mention. But giving it some thought, I figured it was probably more important to maintain consistency with nixos than it was to introduce an ideal API to home-manager. What do you think? Does consistency with nixos matter here?

[cprussin]

Also, I tried the activation script workaround you listed @rycee and it doesn't look like that will work because without duplicating a lot of the logic out of https://github.com/rycee/home-manager/blob/f07510e2b63075a5e334e603f666c5c5838563ce/modules/files.nix, that workaround results in attempting to write the symlink on each switch without cleaning up the old symlink. So I think that selecting some blessed API would be important if this is functionality that you think home-manager should support.

[rycee]

Sorry, yeah I guess it should be ln -sf … instead to unconditionally create the link, even if it already exists.

I'm not terribly concerned about maintaining full consistency with the NixOS option. Indeed, even ignoring this case they are already a bit different .

[cprussin]

Right, but even then, you'll have a problem of stale links being left around after switching (when you move stuff around, etc). If you're open to it, I think I'd prefer to work on submitting a PR to implement a more baked in option. What I had in mind is something like:

{
  home.symlink."some-link".target = "/some/link/target";
}

Looking at https://github.com/rycee/home-manager/blob/master/modules/lib/file-type.nix, I'm thinknig the only options that probably make sense to preserve for this new type would be onChange and (maybe) recursive. So I'd make those available as well.

Thoughts?

[stellarhoof]

for now I find the activation script trick suitable verbose and discouraging to make one think twice before using it

@rycee It's thinking twice _after_ discovering that it is actually possible to do so and indeed the recommended way for now.
Is the intention on marking an option internal to hide it and come up with better support for the feature later? If that's the case, I think it'd still be useful to expose this one in particular, perhaps with a comment that it WILL change in the future and it will break stuff. It seems to me that linking files outside the nix store (although discouraged) is still a pretty common usecase to warrant exposing this even if it's not perfect. What do you think?

[oxalica]

Is this issue still active?
I'm also looking for something like home.symlinks."some/path".target = "target"; (designed by @cprussin).

[cprussin]

I'm still happy to write and submit a PR to implement some option if @rycee is open to it. I might implement such an option in my dotfiles repo as a proposal.

[rycee]

I wish I remember where I saw the link but I like the solution @adisbladis uses here: https://github.com/adisbladis/nixconfig/blob/53457d0f20/home-adisbladis-nixpkgs/home.nix#L5

I wish I had thought of that idea before, it is simple solution that seems to solve all the concerns I had:

1. The default case is safe (immutable copy into Nix store).
2. The unsafe case is explicit (having to use a function).
3. It uses the existing, relatively robust, code that creates and removes symlinks inside $HOME.

So my suggestion would be to simply add a function somewhere inside the HM library code that works sort of like mkPersistentLink.

For example,

home.file."some-file".source = lib.hm.mkOutOfStoreSymlink ./some-file-source;

or something.

[adisbladis]

@rycee Me and @talyz are now both using a custom HM module for this that could be incorporated into HM: https://github.com/adisbladis/nixconfig/blob/9d9239e/modules/home-manager/persistence.nix.

[talyz]

Yeah, I wrote it for a pretty specific use case: having tmpfs as the root file system, and figured the general need was therefore pretty low. If there's interest in having it in HM, I'll definitely submit it, though :)

[rycee]

Ah, interesting module. I'm inclined to agree that it probably is not general enough for inclusion in HM, though. The difference from home.file is not obvious and I can imagine getting many questions about when to use home.file and when to use home.persistence.

I'm thinking that the link function is the necessary and sufficient ingredient that would allow the basic functionality. People could then use it to build specialized code for their use case, like the persistence module.

So, if nobody complains I'll add a function config.lib.mkOutOfStoreSymlink (if you have a better name then let me know).

[tshakah]

I noticed mkOutOfStoreSymlink is now in master - how can you use it in a home-manager user config file?

[cole-h]

You can use it like so: https://github.com/cole-h/nixos-config/blob/12b6a8ac398b234918ffbd7555d4420e15cf35ed/modules/rust.nix#L30.

[tshakah]

Thanks! I had that, but clearly had an import not working somewhere, as I kept on getting error: attribute 'file' missing errors. I have restructured the config and now it works.

[cprussin]

If I'm not mistaken I think that this is now resolved as mkOutOfStoreSymlink has landed, so I'm going to close this issue. Thanks @rycee !