Nixpkgs: Cannot find `~/.config/nixpkgs/config.nix` by following the documentation in `python.section.md`

[snenkov]

Describe the bug
I am trying to follow the documentation for Python as described here:
https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md
however, after I run nix-env -if build.nix I do not understand what does it mean that

Environment defined in ~/.config/nixpkgs/config.nix

Such file does not exist in my system.

To Reproduce
Steps to reproduce the behavior:

1. Create a file, e.g. build.nix, with the following expression:

with import <nixpkgs> {};

python35.withPackages (ps: with ps; [ numpy toolz ])

1. and install it in your profile with: nix-env -if build.nix

Expected behavior
I am not sure what I expect, since I am following the guide and trying to learn.
I guess if since the guide says:

Environment defined in ~/.config/nixpkgs/config.nix

I expect to see such as file defining some environment.

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 4.15.0-66-generic, Ubuntu, 18.04.3 LTS (Bionic Beaver)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.4`
 - channels(root): `"nixpkgs-20.09pre221814.10100a97c89"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

P.S. Any lead that can help me solve this issue would be appreciated (thing to read, etc). I already read the Nix Manual and other python related pages. My end goal is to lean how to build a python environment with some packages inside it via nix. Then to enter ("switch on") that environment and work from it.

[bhipple]

For the really basic workflow of "just get a python shell session with an interpreter and some modules available", try the shell section:

https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#temporary-python-environment-with-nix-shell

The manual is letting you know that you could, if you wanted to, create an environment in a file at that location and refer to it for re-use later on, but when getting started it probably makes sense to just use the shell for ad-hoc sessions.

[doronbehar]

@snenkov is this working for you? If so can we close this?

[bhipple]

There might be an actionable item here: a user has reported some confusion with the manual, so we should take a quick scan through the python section of the manual and see if it can be improved. Since it's in simple markdown that's easy to do in emacs :heart_eyes:

In particular, I think we should move the documentation about the ad-hoc python environments with nix-shell up to the first section of the manual, since that's what people will want to start with when playing for the first time. The portions about writing nix expressions are more heavyweight and might scare someone off who's just starting.

[doronbehar]

There might be an actionable item here: a user has reported some confusion with the manual, so we should take a quick scan through the python section of the manual and see if it can be improved. Since it's in simple markdown that's easy to do in emacs heart_eyes

In particular, I think we should move the documentation about the ad-hoc python environments with nix-shell up to the first section of the manual, since that's what people will want to start with when playing for the first time. The portions about writing nix expressions are more heavyweight and might scare someone off who's just starting.

I agree. cc @FRidh ?

[FRidh]

You seem to be mixing up two sections.

In 15.17.1.1.2.1. Environment defined in separate .nix file you define an environment in some file, in this case build.nix, and install it.

In 15.17.1.1.2.2. Environment defined in ~/.config/nixpkgs/config.nix you choose to add that environment to ~/.config/nixpkgs/config.nix, in which case you won't need to point to the file to install it.

There are different way to install, maybe too many.

Environment defined in ~/.config/nixpkgs/config.nix
Such file does not exist in my system.

This purpose of this file is briefly explained in Chapter 2, but definitely deserves a clear section of its own.

To clarify, the file does not exist by default. You can add it, and then Nixpkgs will impurely pick it up.

[FRidh]

In particular, I think we should move the documentation about the ad-hoc python environments with nix-shell up to the first section of the manual, since that's what people will want to start with when playing for the first time

They are at the top? All that is before is some general introduction to Python in Nixpkgs, and how to install a permanent environment instead of a temporary environment. Those two could be swapped but I don't really see the point of that.

What is definitely needed though is updating all references to old Python versions to newer Python versions. It's not nice when following examples you're building a lot of packages.

[bhipple]

What is definitely needed though is updating all references to old Python versions to newer Python versions.

Yeah I was just looking at all the references to python35 and thinking that too! Should we go to python37 or python38?

Those two could be swapped but I don't really see the point of that.

Yeah I'd consider swapping them. Not a huge deal, but at least for me, when I first started on NixOS, I had a lot of "how do I execute this single-file python script that has 2-3 dependencies?" This question comes up naturally, because if you're hacking on a full-fledged project, it tends to be packaged up fully in Nix, but (at least to me a couple years ago) it was unclear how to do the simple stuff on NixOS.

I'd order it like this:

1. Brief mention of nix-shell -p 'python3.withPackages(p: with p; [ requests numpy ])' for the really simple "poke the box, play with it and get a feel for how it works."

2. Then mention the nix-shell shebangs for single-scripts:

#!/usr/bin/env nix-shell
#! nix-shell -i python3 -p "python3.withPackages(ps: with ps; [ requests numpy ])"

1. Then mention how to create a persistent env in a file that you write and save.

2. Then mention how to actually package up a new python module and build it as part of NixPkgs itself.

This feels like it has a more natural progression of: just getting your feet wet playing around -> something a little more complex and persistent but still relatively simple -> something even more complicated and persistent/robust -> fully contributing something to NixPkgs.

I could take a look at sending a PR if you and @jonringer think that's a reasonable idea. It's not actually adding new information, just a (potential) sequencing optimization.

[FRidh]

Should we go to python37 or python38?

python38, it will also become the new default for 20.09.

...proposal...

Seems reasonable to me. I think I originally wrote it with the idea that "installing" seems more natural for users, so let's deal with that first.

[snenkov]

For the really basic workflow of "just get a python shell session with an interpreter and some modules available", try the shell section:

https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#temporary-python-environment-with-nix-shell

The manual is letting you know that you could, if you wanted to, create an environment in a file at that location and refer to it for re-use later on, but when getting started it probably makes sense to just use the shell for ad-hoc sessions.

I managed to load via nix-shell and through a shell.nix file. Admittedly I don't understand all that's happening on the background, but it's a start.

If I may make a suggestion as a new user who is 100% dependent on the documentation.

Taking python as a popular example, it would be nice if there is a part in the documentation, similar to a tutorial, which are a copy/past follow through. Reproducible (just like nix) and with clear deliverable at the end.

For example here I can create a shell.nix and run nix-shell, but then it's not clear what mkShell does. Note that mkShell is mentioned for the first time at this place.
So I cannot really follow along and have a clear deliverable at the end which when I achieve I can feel I have understood. It just shows me what's possible, even though it's not lear what that thing actually is and why it's useful. This brings a lot of confusion to a newbie in nix.

[FRidh]

Clearly we need that. It's been a problem for 5+ years now that we do not really have separate types of documentation. Occasionally there are initiatives but then they die again. The hard part, aside from just writing, is to actually keep it up to date.

Some time ago I started a project using Jupyter Notebooks for tutorials but there is not much content, and what I wrote is mostly for showcasing the concept.

[doronbehar]

Maybe when https://github.com/NixOS/rfcs/pull/64 will get going it'll be easier to brain storm this.

[snenkov]

How / where can I help give feedback to improve the documentation?
Usually my experience is that when the concept is still foreign to someone (me), usually the feedback is most realistic.

[bhipple]

What you're doing here is great. One thing you could do to help is read my updated version of the new-user tutorial top-to-bottom and let me know if you think it flows better. In particular I've tried to re-organize it from simplest-to-more-complex, and I've also tried to delete semi-redundant information along the lines of "You could do X, or you could do Y, which would sorta work, but we strongly discourage it and think you should just do X." Instead it now just says "Do X".

https://github.com/NixOS/nixpkgs/pull/87450

You can see a rendered version here:
https://github.com/NixOS/nixpkgs/blob/a2ea935cd56510af20b7b11325dab2a4d34236e8/doc/languages-frameworks/python.section.md

[snenkov]

Thanks @bhipple . I am working on Nix now, so I will read now and give feedback shortly.

[snenkov]

Now sure where to write notes, so I will write them here.
From top to bottom:

Here follow the code, but my python is 3.8.2. whereas the one I see on the docs is 3.8.1; naturally I ask the question - why; and whether this kind of version change can happen without me realizing.
This does not need to be explained there on the spot. It can be just noted as expected / unexpected and then referred to later on in a more "advanced" section.

A bit later when introducing --run, even though trivial, it will be nice to either not use it or to explain what it does. Small things like this add up and end up confusing to some people.
Same example a bit later with foo.py.

In generally I think it will be nice to mention that the environment is build on a different store path every time, unlike with venv where it's in the same place. This is a huge blocker and confusion point. If I use virtual environment, I have static python path to my scripts, whereas with nix, the path changes each time. I feel it's important to address this change and (maybe later) explain how to can be used or at least compare to venv in order to showcase something relatable to the new nix user.

The place where you discuss pinning nix packages, I am stuck because I don't know where to get the git commit d373d80b1207d52621961b16aa4a3438e4f98167. So I am now blocked. Also when I type https://github.com/NixOS/nixpkgs/archive/ I get 404 error.
Related to this, it will be nice to mention how does one find the numpy package version and if it's possible to use a specific version of it. I have some background nix knowledge here, so I now know how to do that, but it's not natural to a python user as one either uses latest or specific the version (in requirements.txt fore example). This is also one of the nix rabbit holes which a user can take and go down the road of packaging a numpy version, which is fairly complex to a new user and it can be another roadblock.

In the interest of this comment not becoming too long, I am sending it now and will continue in another comment the rest (if you find it useful ofc).

[snenkov]

For me personally, thing start to get confusing from here, onwards.

I manage to make shell.nix with the code provided and when I run nix-shell all works as expected. I am told this is how I can share environment and I can use in CI, so I get excited because that's exactly why I go through the nix learning curve. Great!

We begin with importing the Nix Packages collections.

It would be nice if we can provide a link here - my understanding is that these package are a repo in GitHub, right? Then my understanding makes that small connection which is useful.

I think this

import imports the function, {} calls it and the with statement brings all attributes of nixpkgs in the local scope. These attributes form the main package set.

Is a bit equivalent to doing from some_python_package import * correct? If so, it will be a good addition, because one of the downsides I find in the nix language is that imports like that make attributes hard to track down. It's something I learned about nix today from this article.

I really get confused on mkShell because I don't understand what this does and it's first place where it's introduced.

The other big blocker here in particular is that we just updated the shell.nix file to have a mkShell but we did not do anything with it. This is where I (personally) am stuck in the documentation, because it makes future part of it harder to follow.
Also, when I now nix-shell with this new updated shell.nix file, I get an error:

[nix-shell:~/projects/pytest3]$ nix-shell 
error: undefined variable 'black' at /home/some_user/projects/pytest3/shell.nix:11:5
(use '--show-trace' to show detailed location information)

[nix-shell:~/projects/pytest3]$ 

[stale[bot]]

I marked this as stale due to inactivity. → More info