Nixpkgs: Ceph build broken in unstable

[aij]

Describe the bug
Ceph build fails since staging-next was merged to master in #102739

To Reproduce
Steps to reproduce the behavior:

1. git checkout nixos-unstable (or 3e7fae8eae5)
2. nix-build -A ceph .
3. Alternately, nix-build nixos/tests/ceph-single-node.nix , or nix-shell -p ceph, or anything else that requires the ceph derivation.

Expected behavior
Build should work and tests should pass. :grin:

Screenshots

ivan@tobati:~/nixops/aij/unstable$ git checkout 3e7fae8eae5
HEAD is now at 3e7fae8eae5 Merge pull request #102739 from NixOS/staging-next
ivan@tobati:~/nixops/aij/unstable$ nix-build -A ceph .
these derivations will be built:
  /nix/store/xlw2nd9gj2wn2lli6vlg7j1qhrzqyd67-python3-3.8.6-env.drv
  /nix/store/zgis4kg0hsv0dw0yh1h41srm8qahs5f6-ceph-15.2.5.drv
building '/nix/store/xlw2nd9gj2wn2lli6vlg7j1qhrzqyd67-python3-3.8.6-env.drv'...
collision between `/nix/store/s2hbnqf6ha2ynpkjxhp26m0ds3bimf06-python3.8-jaraco.functools-3.0.1/lib/python3.8/site-packages/jaraco/__pycache__/__init__.cpython-38.pyc' and `/nix/store/vpgacm1mhm9dwmdl9ixbcmhfj6g7dmgi-python3.8-jaraco.collections-3.0.0/lib/python3.8/site-packages/jaraco/__pycache__/__init__.cpython-38.pyc'
builder for '/nix/store/xlw2nd9gj2wn2lli6vlg7j1qhrzqyd67-python3-3.8.6-env.drv' failed with exit code 25
cannot build derivation '/nix/store/zgis4kg0hsv0dw0yh1h41srm8qahs5f6-ceph-15.2.5.drv': 1 dependencies couldn't be built
error: build of '/nix/store/zgis4kg0hsv0dw0yh1h41srm8qahs5f6-ceph-15.2.5.drv' failed

Additional context
I'm not sure why the python package has collisions on jaraco, but I'm guessing it's via transitive python dependencies which I'm not sure how to track.

I've tracked it down to the merge from staging-next in #102739 . Ceph still builds in 3fcfdb9bcc576c3bf8bd8390dcebe4710a086ec2 but not 3e7fae8eae52f9260d2e251d3346f4d36c0b3116 . I have not had much luck in my attempts to use git bisect to find the relevant commit from the staging-next branch.

Notify maintainers
@adevress @ak @johanot @krav

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.73, NixOS, 21.03.git.a25913605b4 (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.8`
 - channels(root): `"nixos-20.03pre208171.2e8fc97dbfa"`
 - nixpkgs: `/etc/nixos/aij/unstable`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
[ ceph ]
# a list of nixos modules affected by the problem
module:
[ ceph ]

[veprbl]

cc @srhb

[srhb]

Thanks for the ping! I'll look into this, probably tomorrow.

[srhb]

Oh, wow, I had forgotten how hacky the py integration is.. Let's say within the week instead :sweat_smile:

[srhb]

bisect appears to indicate 61a04f735af6a3cdbbb6d2d234fd2dfea934fd18 as the cause, probably revealing some bad packaging issues in our tree. Specifically it's reproducible with the expression:

> nix-build -E 'with import ./. { config = {}; overlays = []; }; python3.withPackages (ps: with ps; [ jaraco_functools jaraco_collections])'
this derivation will be built:
  /nix/store/7s07p89snm3r9y9ryiwj2x3vqhjyjwcd-python3-3.8.6-env.drv
building '/nix/store/7s07p89snm3r9y9ryiwj2x3vqhjyjwcd-python3-3.8.6-env.drv'...
collision between `/nix/store/vpgacm1mhm9dwmdl9ixbcmhfj6g7dmgi-python3.8-jaraco.collections-3.0.0/lib/python3.8/site-packages/jaraco/__pycache__/__init__.cpython-38.pyc' and `/nix/store/s2hbnqf6ha2ynpkjxhp26m0ds3bimf06-python3.8-jaraco.functools-3.0.1/lib/python3.8/site-packages/jaraco/__pycache__/__init__.cpython-38.pyc'

I feel like that __pycache__ shouldn't even exist, and that the change in 61a04f735af6a3cdbbb6d2d234fd2dfea934fd18 has just highlighted that issue.

[aij]

FWIW, I just tried I bisect with

nix-build -E 'with import ./. { config = {}; overlays = []; };
 python3.withPackages(ps: [
    ps.sphinx
    ps.flask
    ps.cython
    ps.setuptools
    ps.virtualenv
    # Libraries needed by the python tools
    ps.Mako
    # TODO ceph-common
    ps.cherrypy
    ps.dateutil
    ps.jsonpatch
    ps.pecan
    ps.prettytable
    ps.pyjwt
    ps.webob
    ps.bcrypt
    # scipy > 1.3 breaks diskprediction_local, leading to mgr hang on startup
    # Bump (and get rid of scipy_1_3) once these issues are resolved:
    # https://tracker.ceph.com/issues/42764 https://tracker.ceph.com/issues/45147
    ps.scipy_1_3
    ps.six
    ps.pyyaml
  ])'

and it also led me to 61a04f735af6a3cdbbb6d2d234fd2dfea934fd18.

[FRidh]

bisect appears to indicate 61a04f7 as the cause, probably revealing some bad packaging issues in our tree. Specifically it's reproducible with the expression:

> nix-build -E 'with import ./. { config = {}; overlays = []; }; python3.withPackages (ps: with ps; [ jaraco_functools jaraco_collections])'
this derivation will be built:
  /nix/store/7s07p89snm3r9y9ryiwj2x3vqhjyjwcd-python3-3.8.6-env.drv
building '/nix/store/7s07p89snm3r9y9ryiwj2x3vqhjyjwcd-python3-3.8.6-env.drv'...
collision between `/nix/store/vpgacm1mhm9dwmdl9ixbcmhfj6g7dmgi-python3.8-jaraco.collections-3.0.0/lib/python3.8/site-packages/jaraco/__pycache__/__init__.cpython-38.pyc' and `/nix/store/s2hbnqf6ha2ynpkjxhp26m0ds3bimf06-python3.8-jaraco.functools-3.0.1/lib/python3.8/site-packages/jaraco/__pycache__/__init__.cpython-38.pyc'

I feel like that __pycache__ shouldn't even exist, and that the change in 61a04f7 has just highlighted that issue.

@jonringer shouldn't the pythonNamespacesHook resolve this issue? I suppose we should then set

pythonNamespaces = [ "jaraco" ]

in the jaraco packages.

[jonringer]

I actually ran into this on the python-unstable branch. Unfortunately pythonNamespacesHook is somewhat broken, in that will not expand:

path is set with pycachePath=${constructedPath}/__pycache__/__init__* but, it will never expand to anything when called if [ -f "$pycachePath" ]. So currently, it's only going to remove the __init__.py, but not the related __pycache__ files.

really, I should be doing:

  for pycacheFile in ${constructedPath}/__pycache__/__init__*; do
    echo "removing $pycacheFile"
    rm "$pycacheFille"
  done

instead of

      if [ -f "$pycachePath" ]; then
             echo "Removing $pycachePath"
             rm "$pycachePath"
      fi

This probably wasn't caught with the azure packages, because almost all of them don't have tests, so they aren't generating __pycache__

[jonringer]

I could do a PR to fix it, but it would have to target staging, as it affects buildPythonPackage

[aij]

Do we need to worry about spaces in path names?

I'm trying a ceph build with

        for pycachePath in "${constructedPath}"/__pycache__/__init__*; do
            echo "Removing $pycachePath"
            rm "$pycachePath"
        done

to see if that's the only remaining problem. Even only building ceph, it still has a lot of dependencies to rebuild, so I have no idea how long to expect it to take. My earlier attempts to git bisect were somewhat confounded by other build errors, which may or may not have been resolved already.

Actually, upon closer inspection, I suspect that's not going to make a difference:

ivan@tobati:~$ ls /nix/store/{s2hbnqf6ha2ynpkjxhp26m0ds3bimf06-python3.8-jaraco.functools-3.0.1,vpgacm1mhm9dwmdl9ixbcmhfj6g7dmgi-python3.8-jaraco.collections-3.0.0}/lib/python3.8/site-packages/jaraco/__pycache__/__init__* 
/nix/store/s2hbnqf6ha2ynpkjxhp26m0ds3bimf06-python3.8-jaraco.functools-3.0.1/lib/python3.8/site-packages/jaraco/__pycache__/__init__.cpython-38.pyc
/nix/store/vpgacm1mhm9dwmdl9ixbcmhfj6g7dmgi-python3.8-jaraco.collections-3.0.0/lib/python3.8/site-packages/jaraco/__pycache__/__init__.cpython-38.pyc

The for loop in pythonNamespacesHook should only make a difference if that glob expands to more than one file, right?

FWIW, there are other python packages that do seem affected by that bug. This finds quite a few paths for me:

find /nix/store/ -name __pycache__ | while read d; do echo $d/__init__* | grep -q ' ' && echo $d/__init__*; done

[jonringer]

Yea, that's what I ended up doing for https://github.com/NixOS/nixpkgs/pull/104225 , is just using find