Nixpkgs: pycurl tests fail on one machine but work on another

[alexeymuranov]

Issue description

Recently doing

# nixos-rebuild boot --upgrade

on a MacBook with NixOS 18.09, i've got an error:

[...]
===End Flaky Test Report===
=============================== warnings summary ===============================
tests/setopt_lifecycle_test.py::TestString
  cannot collect test class 'TestString' because it has a __new__ constructor

-- Docs: https://docs.pytest.org/en/latest/warnings.html
= 5 failed, 371 passed, 9 skipped, 11 deselected, 1 warnings in 57.69 seconds ==
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 53666)
Traceback (most recent call last):
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 138, in run
    self.finish_response()
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 180, in finish_response
    self.write(data)
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 274, in write
    self.send_headers()
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 332, in send_headers
    self.send_preamble()
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 255, in send_preamble
    ('Date: %s\r\n' % format_date_time(time.time())).encode('iso-8859-1')
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 453, in _write
    result = self.stdout.write(data)
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/socketserver.py", line 803, in write
    self._sock.sendall(b)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 141, in run
    self.handle_error()
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 368, in handle_error
    self.finish_response()
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 180, in finish_response
    self.write(data)
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 274, in write
    self.send_headers()
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 331, in send_headers
    if not self.origin_server or self.client_is_modern():
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 344, in client_is_modern
    return self.environ['SERVER_PROTOCOL'].upper() != 'HTTP/0.9'
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/socketserver.py", line 320, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/socketserver.py", line 351, in process_request
    self.finish_request(request, client_address)
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/socketserver.py", line 364, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/socketserver.py", line 724, in __init__
    self.handle()
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/simple_server.py", line 133, in handle
    handler.run(self.server.get_app())
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/handlers.py", line 144, in run
    self.close()
  File "/nix/store/99p0nyqgayic2ipqx2x4lr4njl8f5nrz-python3-3.6.8/lib/python3.6/wsgiref/simple_server.py", line 35, in close
    self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'
----------------------------------------
part1part2
builder for '/nix/store/wqsyjmqjknv201ng3arfs13v9nq2s1yg-python3.6-pycurl-7.43.0.2.drv' failed with exit code 1
building '/nix/store/xij66sxn2qm9bs93fch2j6797022z7v7-reload-container.drv'...
cannot build derivation '/nix/store/vxh240ix3s0q69fzjjhs6bfnjixnansp-system-config-printer-1.5.11.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/znbj382as1gx9g9gkyf9khy8kv7p5k9s-dbus-1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/7kakv1xiwkkyml19q9h9zs1ravq719an-etc.drv': 1 dependencies couldn't be built
building '/nix/store/wjva8468ml1i6pvldnrymjsbwq3fl36f-stage-1-init.sh.drv'...
building '/nix/store/5nqjkqyvv6407xsrcjly4lrkw6cfijr9-xkb-validated.drv'...
cannot build derivation '/nix/store/780mzphw81j1mz5rc4cyvh8x50ar2llp-nixos-system-Alexey-MacBook-NixOS-18.09.2203.9bd45dddf81.drv': 1 dependencies couldn't be built
error: build of '/nix/store/780mzphw81j1mz5rc4cyvh8x50ar2llp-nixos-system-Alexey-MacBook-NixOS-18.09.2203.9bd45dddf81.drv' failed

Here is the complete output of nixos-rebuild reproduced later.

However, I had no issues on another machine (an Asus) with a similar configuration. I copied over configuration.nix from MacBook, and nixos-rebuild still worked fine.

How can it be?

Steps to reproduce

On the failing machine i can reproduce this with:

# nixos-rebuild boot -I nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/9bd45dddf81.tar.gz

Technical details

Here nixos-rebuild fails:

alexey@Alexey-MacBook-NixOS:~$ nix-shell -p nix-info --run "nix-info -m"
/nix/store/a6rf8a474z5y01w4ld2v54d24kyqcxag-stdenv-linux/setup: line 784: /run/user/1000/env-vars: Permission denied
 - system: `"x86_64-linux"`
 - host os: `Linux 4.19.19, NixOS, 18.09.2098.d4a1e8975a6 (Jellyfish)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.1.3`
 - channels(root): `"nixos-18.09.2203.9bd45dddf81"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Here it works:

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 4.19.23, NixOS, 18.09.2203.9bd45dddf81 (Jellyfish)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.1.3`
 - channels(alexey): `"nixos-17.03-17.03.1949.78e9665b48f, nixos-17.09-17.09.3238.e984f9e48e1"`
 - channels(root): `"nixos-18.09.2203.9bd45dddf81"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[davidak]

Can you reproduce it every time?

The only obvious difference is the hardware but the error don't look like a hardware issue.

[alexeymuranov]

On the failing machine -- yes. Just tried it again.

The failure seems to be related to pycurl: there is a message about "unpacking pycurl", then a lot of build and test output, which ends with a failure. (By the way, I do not remember seeing that much output when upgrading the other machine recently, but maybe i did not pay attention.)

[alexeymuranov]

I also tried

# nix-store --verify --check-contents --repair

but there were no issues to repair.

[jtojnar]

Duplicate of https://github.com/NixOS/nixpkgs/issues/55637?

[alexeymuranov]

Anything i can test to debug this?

I will not have access to the failing machine until Saturday though.

[alexeymuranov]

After downloading the latest 18.09 channel, the upgrade worked. I have now (on the previously failing machine):

alexey@Alexey-MacBook-NixOS:~$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 4.19.24, NixOS, 18.09.2253.753f58d9a42 (Jellyfish)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.1.3`
 - channels(root): `"nixos-18.09.2253.753f58d9a42"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

(Note that the line about "denied permission" is gone now.)

I suppose this can be closed for now, unless someone has any idea of how to debug original problem.

[alexeymuranov]

Just an idea: how about creating a meta-issue or a Wiki page to track all unexplained issues (even closed ones) that contradict (or seem to) the functional and reproducible nature of Nix/NixOS?

[vcunat]

Upstreams often aren't very careful about their test-suites, so they're e.g. sensitive to the machine being overloaded, etc. There's little NixOS-specific about these, except that we run tests a lot, partially thanks to lots of rebuilds. Certainly, if NixOS people feel like fixing these upstream, it won't hurt; otherwise we might be collecting lots of open issues that noone really cares about and that usually belong upstream...

[jtojnar]

nixos-rebuild --upgrade is non-deterministic by design – it is meant as a helper for users that do not require reproducibility. It updates channels before rebuilding and you do not have much control over it – it probably upgraded to some channel revision that was broken and now it upgraded to a fixed one. If you want reproducibility, you need to update channels separately – that way you can see what channel you are using – or even better, call maintain nixpkgs revision manually (e.g. by passing -I nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/19eedaf867d.tar.gz).

[alexeymuranov]

Indeed, I can still reproduce this on my MacBook with

# nixos-rebuild boot -I nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/9bd45dddf81.tar.gz

It is strangely consistent for just some kind of a race condition...

[jtojnar]

Some race conditions appear when building on a machine with certain number of cores more frequently than on others.

[alexeymuranov]

I have just tried the same command on another machine with a different configuration.nix, and there was a similar failure. I was ready to close this issue (as long as the behaviour is consistent, this issue is invalid), but then i tried it second time to capture the output, and it built fine...

[alexeymuranov]

Anyway, feel free to close if you consider that there is nothing to investigate here.

[joepie91]

To add a data point: I just ran into a very similar error as well.

builder for '/nix/store/bqak29ayz6q0b1civmrr9ayh4ac0kjb5-python3.6-pycurl-7.43.0.2.drv' failed with exit code 1; last 10 log lines:
  test_post_with_null_byte passed 1 out of the required 1 times. Success!

  ===End Flaky Test Report===
  =============================== warnings summary ===============================
  tests/setopt_lifecycle_test.py::TestString
    cannot collect test class 'TestString' because it has a __new__ constructor

  -- Docs: https://docs.pytest.org/en/latest/warnings.html
  = 1 failed, 375 passed, 9 skipped, 11 deselected, 1 warnings in 33.14 seconds ==
  part1part2
cannot build derivation '/nix/store/9kd78bm5q7v2p00wl8lzwl15pc8birlv-system-config-printer-1.5.11.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/pq2jzxviiqzb4hn6yz1risskzlcp44fv-dbus-1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/6w7m90x7axwv82rj7dr8dnvw04v61jhz-etc.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/2qr5wghm91ykgxlhjdhg705780i1ra71-nixos-system-x270-18.09.2574.a7e559a5504.drv': 1 dependencies couldn't be built

Another attempt at rebuilding the exact same configuration, immediately afterwards, worked just fine.

[deliciouslytyped]

I got something similar to @joepie91, similarly pulled in by system-config-printer.

builder for '/nix/store/f3k2kb2mwxw2i29i5rxhcil4zfly8yb2-python3.7-pycurl-7.43.0.3.drv' failed with exit code 1; last 10 log lines:
  test_post_single_field passed 1 out of the required 1 times. Success!
  test_post_tuple_of_lists_of_tuples passed 1 out of the required 1 times. Success!
  test_post_tuple_of_tuple_of_lists passed 1 out of the required 1 times. Success!
  test_post_tuple_of_tuples_of_tuples passed 1 out of the required 1 times. Success!
  test_post_unicode_buffer passed 1 out of the required 1 times. Success!
  test_post_with_null_byte passed 1 out of the required 1 times. Success!

  ===End Flaky Test Report===
  ===== 4 failed, 378 passed, 9 skipped, 9 deselected, 8 warnings in 58.52s ======
  part1part2

[joepie91]

The pycurl tests are now reproducibly failing here with this error:

=================================== FAILURES ===================================
___________________ CaCertsTest.test_request_with_verifypeer ___________________

self = <tests.cadata_test.CaCertsTest testMethod=test_request_with_verifypeer>

    @util.only_ssl_backends('openssl')
    def test_request_with_verifypeer(self):
        with open(os.path.join(os.path.dirname(__file__), 'certs', 'ca.crt'), 'rb') as stream:
            cadata = stream.read().decode('ASCII')
        self.curl.setopt(pycurl.URL, 'https://localhost:8384/success')
        sio = util.BytesIO()
        self.curl.set_ca_certs(cadata)
        self.curl.setopt(pycurl.WRITEFUNCTION, sio.write)
        # self signed certificate, but ca cert should be loaded
        self.curl.setopt(pycurl.SSL_VERIFYPEER, 1)
>       self.curl.perform()
E       pycurl.error: (60, 'SSL certificate problem: unable to get local issuer certificate')

tests/cadata_test.py:31: error
----------------------------- Captured stderr call -----------------------------
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/nix/store/zqr5sy6dxnas41s0axyhhvlqnhwj0ywk-python3-3.7.5/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/tmp/nix-build-python3.7-pycurl-7.43.0.3.drv-0/pycurl-7.43.0.3/tests/runwsgi.py", line 67, in run
    bottle.run(self.app, server=self.server, quiet=True)
  File "/nix/store/31x1b2x2qyy918bhvkqfi8z4hid9ifdx-python3.7-bottle-0.12.16/lib/python3.7/site-packages/bottle.py", line 3129, in run
    server.run(app)
  File "/tmp/nix-build-python3.7-pycurl-7.43.0.3.drv-0/pycurl-7.43.0.3/tests/runwsgi.py", line 34, in run
    self.srv = self.make_server(handler)
  File "/tmp/nix-build-python3.7-pycurl-7.43.0.3.drv-0/pycurl-7.43.0.3/tests/runwsgi.py", line 24, in make_server
    srv = make_server(self.host, self.port, handler, **self.options)
  File "/nix/store/zqr5sy6dxnas41s0axyhhvlqnhwj0ywk-python3-3.7.5/lib/python3.7/wsgiref/simple_server.py", line 153, in make_server
    server = server_class((host, port), handler_class)
  File "/nix/store/zqr5sy6dxnas41s0axyhhvlqnhwj0ywk-python3-3.7.5/lib/python3.7/socketserver.py", line 452, in __init__
    self.server_bind()
  File "/nix/store/zqr5sy6dxnas41s0axyhhvlqnhwj0ywk-python3-3.7.5/lib/python3.7/wsgiref/simple_server.py", line 50, in server_bind
    HTTPServer.server_bind(self)
  File "/nix/store/zqr5sy6dxnas41s0axyhhvlqnhwj0ywk-python3-3.7.5/lib/python3.7/http/server.py", line 137, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/nix/store/zqr5sy6dxnas41s0axyhhvlqnhwj0ywk-python3-3.7.5/lib/python3.7/socketserver.py", line 466, in server_bind
    self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use

[alexeymuranov]

@joepie91, see also #71975.

[joepie91]

For those who are in the same situation as I am, where this (sort of) failure is blocking a system rebuild, this workaround in your configuration.nix will disable the install checks:

```nix
{
nixpkgs.overlays = [
(self: super: {
python3 = super.python3.override {
packageOverrides = python-self: python-super: {
pycurl = python-super.pycurl.overrideAttrs (oldAttrs: {
doInstallCheck = false;
});
};
};
})
];
}

[stale[bot]]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[davidak]

this was fixed in the linked pr

https://github.com/NixOS/nixpkgs/blob/ef95956453c5300e02f85d975d72302471092034/pkgs/development/python-modules/pycurl/default.nix#L47