Nixpkgs: pysam build fails: cython uses wrong python version

[scalavision]

Describe the bug
A clear and concise description of what the bug is.

When building pysam library it fails on the tests with the following:

Executing pipInstallPhase
make: Entering directory '/build/source/tests/pysam_data'
samtools faidx ex1.fa
samtools import ex1.fa.fai ex1.sam.gz ex1.bam
make: *** [Makefile:56: ex1.bam] Segmentation fault (core dumped)
make: *** Deleting file 'ex1.bam'
make: Leaving directory '/build/source/tests/pysam_data'
builder for '/nix/store/w8rdvrvs4c3nixvw5sg8wd4rm2l6n7zr-python3.7-pysam-0.15.3.drv' failed with exit code 2

When I see the log files of the build:

Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release!

according to SO :

• https://stackoverflow.com/questions/34603628/how-to-specify-python-3-source-in-cythons-setup-py

It should be enough to add:

 compiler_directives={'language_level' : sys.version_info[0]})

to the meta hashmap defined in setup.py, but I did not get this to work (maybe the hashmap is not able to handle nested objects?).

@unode

could you have a look when you have time? I would like to update truvari and deeptools when this is fixed :-)

To Reproduce
run the tests of pysam during build

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

• system: "x86_64-linux"
• host os: Linux 5.4.11, NixOS, 19.09.2036.c49da6435f3 (Loris)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.2
• channels(root): "nixos-19.09.1774.a3070689aef"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: HTSeq, deeptools, cnvkit, pysaml2, truvari, tebreak
# a list of nixos modules affected by the problem
module:

[unode]

@scalavision I haven't managed to reproduce the issue you are reporting with nix (not on NixOS currently). Do you have a test-case you can share?

When I try to use, for instance, HTSeq it builds without cython.

For instance nix-shell '<nixpkgs_dev>' --pure -p python37Packages.HTSeq --run python where nixpkgs_dev is a checkout of nixpkgs commit c49da6435f3 (after your host os metadata).

pysam builds fine here as well...

[scalavision]

Thanks for looking into this @unode !!

Do you have Docker available?

Maybe this will do:

FROM lnl7/nix:2.2

# To run after image interactively : docker run --rm -it <name of image>

RUN nix-env -iA \
 nixpkgs.git \
 nixpkgs.neovim

RUN mkdir -p /opt/nix && \
 cd /opt/nix && \
 git clone https://github.com/NixOS/nixpkgs.git

RUN cd /opt/nix/nixpkgs && nix-build -I "." -A python37Packages.pysam

It crashes with:

adding 'pysam-0.15.3.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Finished executing setuptoolsBuildPhase
installing
Executing pipInstallPhase
make: Entering directory '/build/source/tests/pysam_data'
samtools faidx ex1.fa
samtools import ex1.fa.fai ex1.sam.gz ex1.bam
make: *** [Makefile:56: ex1.bam] Segmentation fault (core dumped)
make: *** Deleting file 'ex1.bam'
make: Leaving directory '/build/source/tests/pysam_data'
builder for '/nix/store/964xvqxnbbqv0mpklac7gpahz3vb3rap-python3.6-pysam-0.15.3.drv' failed with exit code 2

I have tested with python 37 as well. I have tested it on two different machines besides the docker image, although with quite similar setup. nix-build is supposed to be pure isn't it?

It looks as if the failure is in some test using samtools under the hood. I also tried using pysam 0.15.4, and the same error occurs ...

I am also concerned about cython automatically downgrading to python2 when not finding the language_level property (it's a warning printed to the console while building).

I had the exact same problem with cython downgrading when building svdb in this PR that I am working on, but adding this patch solved it:

• https://github.com/scalavision/nixpkgs/blob/add-svdb-to-biology-pkgs/pkgs/applications/science/biology/svdb/0001-set-language-version.patch#L17

This doesn't have dependency on pysam though (as far as I know).

[scalavision]

@unode I think I've found the problem, htslib and samtools have been upgraded to 1.10.2 and 1.10 respectively.

Building pysam against the old versions of 1.9 is still working.

Is the best way to move on, making hslib and samtools generic? .. allow for multiple versions? Almost the entire bioinformatic universe depends on those libraries ...

Alternatively, we probably have to downgrade the htslib, but the delly package has already started to use the new version of htslib.

I've created a mockup just to test building pysam in this branch:

• https://github.com/scalavision/nixpkgs/tree/add-support-multiple-versions-htslib

@jonringer, @jbedo, @nh2, @mimame what do you think?

Anyone else should be involved in this discussion?

I am still quite new to nix and nixpkgs, so maybe there are even better ways to do this ...

[nh2]

Unfortunately I don't know much about this topic currently. Will leave it to the others.

[unode]

@scalavision sorry I haven't found the time to get to this.

If your diagnostic is correct we might need to bring this issue to the pysam developers. From nixpkgs side, the only safe option is to pin pysam to a specific samtools/htslib version.

If you can try, does updating nixpkgs pysam to use the latest github commit solve the compatibility issues?

As for the cython warning, I feel tempted to also report this to the pysam developers. We can try to fix it on our side but ultimately this should be fixed in pysam's setup.py.

[unode]

Also, a quick search returned: https://github.com/pysam-developers/pysam/pull/875 and https://github.com/pysam-developers/pysam/pull/868 so it might be that pysam 0.15.4 fixes all the issues here.

[unode]

0.15.4 builds fine here. @scalavision can you test if the pull request #80360 fixes it for you?

[unode]

Seems like the pull request #80360 fails with the same error as above, so updating doesn't help.
Next in the test list is latest pysam from git.

[scalavision]

I am sorry, but I am still not able to build it locally using your pull request. I am really puzzled that this actually worked in the ci. Is the ci using samtools 1.9 or 1.10? hm.. by pinning the htslib and samtools versions, is this something I should do before I try to build it locally? Or should this be taken care of in the derivation?

Building pysam 0.15.4 using version 1.9 of samtools and htslib works fine for me locally. I also think upgrading to pysam 0.15.4 probably would work for for truvari and deeptools. I'll try to have a go tomorrrow :-)

I think I'll create a PR for making htslib and samtools generic. I have a lot of tools depending on different versions of these libraries, and I think it will be impossible to make all of these libraries staying up to date (unfortunately). Although, it's amazing that nixpkgs actually makes this possible!

I agree that we should try to make pysam handle the cython downgrade problem!

Thanks a lot so far @unode !! :-)

[unode]

So it turns out this issue is not in pysam but samtools instead, but possibly related with the test files included in pysam_data.

Running the failing commands on the test under gdb results in:

(gdb) run
Starting program: /nix/store/8y3m8rd75si8hz2m4cx3g82l9h10qkja-samtools-1.10/bin/samtools import ex1.fa.fai ex1.sam.gz ex1.bam
/nix/store/vrnxw026hiy8jvdpaaix47x528bkzksk-bash-4.4-p23/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/8g1v3sf0xvf044sz1b4kcrg1i86z9bh7-glibc-2.30/lib/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7ca13c1 in __strlen_avx2 () from /nix/store/8g1v3sf0xvf044sz1b4kcrg1i86z9bh7-glibc-2.30/lib/libc.so.6
(gdb) bt
#0  0x00007ffff7ca13c1 in __strlen_avx2 () from /nix/store/8g1v3sf0xvf044sz1b4kcrg1i86z9bh7-glibc-2.30/lib/libc.so.6
#1  0x00007ffff7f531a1 in stringify_argv () from /nix/store/36s9s0b5vfky4w9jvxi527bjilf257zf-htslib-1.10.2/lib/libhts.so.3
#2  0x00000000004110b4 in main_samview ()
#3  0x0000000000411dab in main_import ()
#4  0x00007ffff7b6ed8b in __libc_start_main () from /nix/store/8g1v3sf0xvf044sz1b4kcrg1i86z9bh7-glibc-2.30/lib/libc.so.6
#5  0x0000000000407b0a in _start ()

Tagging @mimame for help on this one.

[jbedo]

Pysam test also crashes for me. I think this should be reported upstream to pysam as I believe the samtools import command was deprecated years ago. The pysam test should therefore be patched to

samtools view -bt ex1.fa.fai ex1.sam.gz > ex1.bam

[jbedo]

The patch jbedo/pysam@601db642dea31eef7c9a45024d1349816042ef01 does indeed resolve the issue, however there's another one in a later test:

  samtools view -bS example_user_header.sam > example_user_header.bam
  [E::sam_hrecs_error] Header line does not have a two character key at line 4: "@x1    A:2     B:5"
  samtools view: failed to add PG line to the header
  make: *** [Makefile:41: example_user_header.bam] Error 1

Can't see anything obviously wrong with the file but will keep digging.

That said this is clearly a pysam issue and we should resolve it upstream rather than pinning old versions of samtools/htslib.

[unode]

@jbedo samtools 1.10 added more stringent rules about header parsing. See release notes on https://github.com/samtools/samtools/releases. Maybe also:

Examples of problems that will now be rejected include @SQ lines
with no SN: tag, and @RG or @PG lines with no ID: tag.

[rollf]

I stumbled over this issue before and had reported it to the samtools devs: https://github.com/samtools/samtools/issues/1183

I managed to compile pysam without the tests and thus to compile cnvkit:

self: super: {
  # pysam doesn't compile due to issues with `samtools import` during the test phase.
  # See https://github.com/samtools/samtools/issues/1183
  pysam = super.python38Packages.pysam.overrideAttrs (old: rec {
    preInstall = "";
    doInstallCheck = false;
  });
}

and using the overlay cnvkit = python38Packages.cnvkit.override { pysam = pysam; };

[unode]

@rollf Thanks for this reference.

Seems like all the involved parts are patched or on the way to be.
To unblock this we can disable tests for now and re-enable once a new release of pysam comes out.

There is an open issue for the use of import at https://github.com/pysam-developers/pysam/pull/890 . I'll include the patch from @jbedo above in https://github.com/NixOS/nixpkgs/pull/80360. We can remove the patch once pysam makes a new release.

[scalavision]

Thanks to everyone helping out on this!

I've posted an issue in pysam repository, regarding the cython downgrade to python2.

• https://github.com/pysam-developers/pysam/issues/893

[unode]

Was not possible to reopen a closed pull request there's a new one at #80625

Thanks everyone for the help!

[unode]

Seems like just patching for import isn't sufficient. Some of the SAM files used in pysam tests include invalid tags and other issues that appeared due to the header parsing strictness introduced in samtools 1.10.0.

Since these are all test failures due to incorrect test data, I'm instead disabling tests until these issues are fixed in a future release of pysam. There's more on this at: https://github.com/pysam-developers/pysam/pull/890

[scalavision]

I've tested new versions of truvari and deeptools, both of them works so we can close this issue. Thanks everyone for their great work!