Nixpkgs: GlusterFS python tools all have import errors

[nh2]

Issue description

[root@gluster-cluster-1:~]# gluster-mountbroker 
Traceback (most recent call last):
  File "/run/current-system/sw/bin/gluster-mountbroker", line 5, in <module>
    from gluster.cliutils import (execute, Cmd, node_output_ok,
ImportError: No module named gluster.cliutils

[root@gluster-cluster-1:~]# glusterfind 
Traceback (most recent call last):
  File "/run/current-system/sw/bin/glusterfind", line 14, in <module>
    from glusterfind.main import main
ImportError: No module named glusterfind.main

[root@gluster-cluster-1:~]# gluster-georep-sshkey 
Traceback (most recent call last):
  File "/run/current-system/sw/bin/gluster-georep-sshkey", line 26, in <module>
    from gluster.cliutils import (node_output_ok, execute, execute_in_peers,
ImportError: No module named gluster.cliutils

[root@gluster-cluster-1:~]# gluster-eventsapi 
Traceback (most recent call last):
  File "/run/current-system/sw/bin/gluster-eventsapi", line 22, in <module>
    import requests
ImportError: No module named requests

Steps to reproduce

Install glusterfs; I tried this on nixpkgs commit 250c8900.

Technical details

I think the problem is that these scripts don't have PYTHONPATH set so that it can import gluster's own Python libs (gluster.cliutils) or other required python modules (requests).

These scripts got their #!/usr/bin/env python replaced by nix, but it points directly to a python interpreter binary in /nix/store, not to a wrapper that sets PYTHONPATH.

I'm not sure what the correct way to fix it is. Do we need to wrapProgram --prefix PYTHONPATH each individual script around here? Or is there a way to do it for all of them?

[nh2]

Hmm, looks like other apps like wicd also use (the equivalent of) makeWrapper on every single executable Python script: https://github.com/NixOS/nixpkgs/blob/66d3b9a02d78cddc7ac0623fda44ac5895cd5951/pkgs/tools/networking/wicd/default.nix#L33-L49

[nh2]

Another example: nmap

https://github.com/NixOS/nixpkgs/blob/e3d9f438c357377d524036f88c45e9245c3809fc/pkgs/tools/security/nmap/default.nix#L37-L39

[nh2]

The nixpkgs manual and https://github.com/NixOS/nixpkgs/commit/e760c7bd6a76ed27d156d7df3d493af2a9558f11 suggest that a helper like buildPythonApplication can automatically makeWrapper all python scripts

But that doesn't seem to be the right thing for GlusterFS as it is mainly a C application.

[FRidh]

@nh2 in this case its likely best to use wrapProgram and set PYTHONPATH, not --prefix PYTHONPATH. You could also consider using wrapPythonPrograms.

[FRidh]

cc maintainer @7c6f434c

[nh2]

this case its likely best to use wrapProgram and set PYTHONPATH, not --prefix PYTHONPATH

@FRidh can you elaborate a bit why that is better? I am not very familiar with this.

Also, where can I read about wrapPythonPrograms?

[FRidh]

@nh2 buildPythonpackage and buildPythonApplication are for building Python packages, typically setuptools packages. They build from source/sdist a wheel which is then installed.

They use the shell function wrapPythonPrograms that patches Python scripts to set their name (which is incorrect when Python scripts are wrapped because Python doesn't respect exec -a) and hardcodes their Python dependencies (using site) . The wrapPythonPrograms function is defined in the wrapPython package.

In this case you only have scripts that need a Python interpreter that can find the modules provided by this package. Then its most straightforward, and least error prone, to use wrapProgram. We use set PYTHONPATH instead of --prefix PYTHONPATH because otherwise programs may break because of a leaking PYTHONPATH.

[nh2]

@FRidh Thanks for the explanations, I'll give that a try.

For my reference, wrapPythonPrograms is defined here.

[nh2]

Follow-up on IRC:

nh2: FRidh: should that be `set` or `--set` to makeWrapper?
FRidh: nh2: --set
nh2: FRidh: for some reason, that makes me end up with just `export PYTHONPATH=":"`; `--prefix` doesn't have this problem
nh2: FRidh: I'm using
nh2: wrapProgram $out/bin/gluster-eventsapi --set PYTHONPATH : "$(toPythonPath $out):$(toPythonPath ${python2Packages.requests}):$(toPythonPath ${python2Packages.prettytable}):$PYTHONPATH"
nh2: am I invoking it incorrectly?
nh2: maybe that ` : `?
FRidh: nh2: the colon should indeed be removed in the case of --set
nh2: I guess that's what `separator="${params[$((n + 2))]}";` does for --prefix
nh2: sometimes I have to shiver in presence of the amount of shell scripting in nixpkgs
FRidh: then you're not the only one
nh2: FRidh: I can run `gluster-eventsapi` now without import error. But there's still the problem that it says ".gluster-eventsapi-wrapped: error: too few arguments" instead of "gluster-eventsapi: error: too few arguments" -- the "wrapped" makes it into argv[0]. I think this is related to the `exec -a` you mentioned, and which I can see in the wrapper script. But I didn't understand from your reply: Is there a fix for that?
FRidh: nh2: Unfortunately not. wrapPythonPrograms patches the script to set `sys.argv[0]` so that does solve some issues

[nh2]

I have a solution with the help of @cleverca22: https://gist.github.com/nh2/dbf8eb5801d9d221ef7a51c11c17bb89#file-working-with-tests-nix

I still get the error

# gluster system:: execute georep-sshkey.py node-generate .
Unable to end. Error : Success

but will hopefully fix that too, an then PR the whole thing.

[FRidh]

You could also just open a PR already as it may help in getting feedback.

[nh2]

I finally got what the error with gluster system:: execute georep-sshkey.py node-generate . is.

I did so by instrumenting glusterd-geo-rep.c so that glusterd_op_sys_exec() logs the stderr of programs it invokes, by adding the lines

        runner_redir (&runner, STDERR_FILENO, RUN_PIPE); // nh2

and

        do {
                ptr = fgets(buf, sizeof(buf), runner_chio (&runner, STDERR_FILENO));
                if (ptr) {
                        runner_log (&runner, "nh2 glusterd-geo-rep", GF_LOG_DEBUG, "STDERR ptr OK");
                        runner_log (&runner, "nh2 glusterd-geo-rep STDERR output", GF_LOG_DEBUG, ptr);
                } else {
                        runner_log (&runner, "nh2 glusterd-geo-rep", GF_LOG_DEBUG, "STDERR ptr NULL");
                }
        } while (ptr);

before their respective counterparts for stdout.

In other words, gluster 3.10.1 is discarding the stderr of failing commands in glusterd_op_sys_exec().

The stderr of the failing command (/nix/store/0h89pi4d8wchzhpzklfdlssfg7cmwhl8-glusterfs-3.10.1/libexec/glusterfs/peer_georep-sshkey.py node-generate . in my case) when invoked by gluster is:

Traceback (most recent call last):
  File "/nix/store/wncl4hn1698l0m8r7hc0yg0nf3cn0bmc-glusterfs-3.10.1/libexec/glusterfs/.peer_georep-sshkey.py-wrapped", line 116, in <module>
    runcli()
  File "/nix/store/wncl4hn1698l0m8r7hc0yg0nf3cn0bmc-glusterfs-3.10.1/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 224, in runcli
    cls.run(args)
  File "/nix/store/wncl4hn1698l0m8r7hc0yg0nf3cn0bmc-glusterfs-3.10.1/libexec/glusterfs/.peer_georep-sshkey.py-wrapped", line 75, in run
    node_output_ok(data)
  File "/nix/store/wncl4hn1698l0m8r7hc0yg0nf3cn0bmc-glusterfs-3.10.1/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 61, in node_output_ok
    out = {"ok": True, "nodeid": get_node_uuid(), "output": message}
  File "/nix/store/wncl4hn1698l0m8r7hc0yg0nf3cn0bmc-glusterfs-3.10.1/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 35, in get_node_uuid
    rc, out, err = execute(cmd)
  File "/nix/store/wncl4hn1698l0m8r7hc0yg0nf3cn0bmc-glusterfs-3.10.1/lib/python2.7/site-packages/gluster/cliutils/cliutils.py", line 74, in execute
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/nix/store/3c598gvxavyiv2h0n2lbxgq7sf0rxf1p-python-2.7.13-env/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/nix/store/3c598gvxavyiv2h0n2lbxgq7sf0rxf1p-python-2.7.13-env/lib/python2.7/subprocess.py", line 1024, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

When I invoke it myself, I get the expected output:

# /nix/store/0h89pi4d8wchzhpzklfdlssfg7cmwhl8-glusterfs-3.10.1/libexec/glusterfs/peer_georep-sshkey.py node-generate . 
{"output": {"tar_pub": "command=\"tar ${SSH_ORIGINAL_COMMAND#* }\"  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0YQSaD+iQbQxgWtfaiPvJ+ZKlkImZkRZg7JAQWX4lLJYLiE1GHH7nvCYrJhL1uwmdb49bCWXBjHIvfWA/HzqnK3OwxOJ5kZC8QSVNv8T2MDV60MwLrHYrJLLpoEdiDFncsH+U5jVaIyP+KJxVZzLQvBbXw8T8nnJgmaQMcBjsKINnn9/p0wLteh7njYRS8PkCYvaK3nXtjD2cfFU9PG0xdkHbgMAw2qJKdczAf6Z/uXPt29kN8DvZ1sXWOKSUkZir+Aqq685ARgeBSeZzEmHBXnylw8s1bZwhxzXUw6Li5xu7BALn+nQns0Bw6Pz4Nd3gExphRWcvmgJzPLjuHjkN root@gluster-cluster-1", "default_pub": "command=\"${exec_prefix}/libexec/glusterfs/gsyncd\"  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKM4ekIbV7NYnLkpsAoE70/CXb8SFt2dH6Y63Q0FLgZmG80ltMtZ0HQ+uBaUZwE+q3Eo/CcYC6UvEbbJeAAPQftD13hQQv3Jkxfg38xWYN+PXo7hxt0aDbyt+Mu5MfoVxRqC1GVGZ3ALK5eX8JDLeedKXCu3o7o/lfR8YXhEiFv2JNrt3Dd9rTYo3gyO2EpNKZy9GGu+zUjpzweb65wn3t9gJcP3sW3tvX1GUZqnyA6CqbJH7bVK+wh4erOn0aAtHk3kilIesKjtpz59hsVCqKLGtjacZqPz2WAvLUJlimKd07Oeq9HAt540V0SjXEW+c4beKuf8qgXbudt0AD9P+b root@gluster-cluster-1"}, "ok": true, "nodeid": "c2038af1-6f0f-42ca-a6fd-69d452416992"}

Now just need to find out why it fails when invoked by gluster.

[nh2]

Ah, easy.

https://github.com/gluster/glusterfs/blob/ff5e91a60887d22934fcb5f8a15dd36019d6e09a/extras/cliutils/cliutils.py#L34-L35

    cmd = ["gluster", "system::", "uuid", "get", "--xml"]
    rc, out, err = execute(cmd)

gluster is not on the PATH.

But it is in the PATH of my shell. That makes the difference.

[nh2]

I filed two upstream GlusterFS bugs for this topic:

• errno used incorrectly or misleadingly in error messages (https://bugzilla.redhat.com/show_bug.cgi?id=1449416)
• glusterd-geo-rep.c doesn't log stderr on failure to execute subprocesses (https://bugzilla.redhat.com/show_bug.cgi?id=1449861)

[nh2]

PR: https://github.com/NixOS/nixpkgs/pull/25709

[nh2]

For completeness, the command I ultimately wanted to work (and which works now) is gluster-georep-sshkey generate.

[nh2]

Upstream has indicated that they'd accept a patch that removes the hardcoded paths that I have already patched out with my NixOS patch in my PR.

So I'll try to upstream that patch for some future GlusterFS release.

[nh2]

CC @bachp as you added the glusterfs NixOS service and may be interested in the above.

[nh2]

I found another issue that makes stuff fail at runtime, and it solution. Will update PR tomorrow.

[nh2]

I found another issue that makes stuff fail at runtime, and it solution. Will update PR tomorrow.

gluster checks presence of version too late - https://bugzilla.redhat.com/show_bug.cgi?id=1452705

[nh2]

Will update PR tomorrow.

Done.