Nixpkgs: Mopidy is broken

[rvolosatovs]

Issue description

After upgrading my system to nixpkgs commit cc1d7a358f5, mopidy fails to start

● mopidy.service - mopidy music player daemon
   Loaded: loaded (/nix/store/4jwmkklrip1nkkxsi558cl124f8f0miv-unit-mopidy.service/mopidy.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2017-12-01 16:54:41 CET; 19s ago
  Process: 8885 ExecStart=/nix/store/xxgp8r8m8af3mfzqc0bi0zgvcnc1cm5c-python-2.7.14-env/bin/mopidy --config /nix/store/<hidden>-mopidy.conf (code=exited, status=203/EXEC)
  Process: 8877 ExecStartPre=/nix/store/8zcsgrsj4hj7jgjjlwdpiwhk6hcv7vm5-unit-script/bin/mopidy-pre-start (code=exited, status=0/SUCCESS)
 Main PID: 8885 (code=exited, status=203/EXEC)
      CPU: 21ms

Dec 01 16:54:41 neon systemd[1]: Starting mopidy music player daemon...
Dec 01 16:54:41 neon systemd[1]: Started mopidy music player daemon.
Dec 01 16:54:41 neon systemd[8885]: mopidy.service: Failed at step EXEC spawning /nix/store/xxgp8r8m8af3mfzqc0bi0zgvcnc1cm5c-python-2.7.14-env/bin/mopidy: No such file or directory

Steps to reproduce

checkout cc1d7a358f5 , set services.mopidy.enable = true

Technical details

 - system: `"x86_64-linux"`
 - host os: `Linux 4.9.65, NixOS, 18.03pre121732.cc1d7a358f5 (Impala)`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 1.11.15`
 - channels(root): `"nixos-18.03pre121732.cc1d7a358f5, nixpkgs-18.03pre114739.d0d905668c"`
 - channels(rvolosatovs): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs`

[andir]

@rvolosatovs do you know what commit you upgraded from?

I started bisecting that with a new mopidy test....

[andir]

@rvolosatovs can you try with the following diff applied? I can start mopidy (again). Since I never really used mopidy I can't really test if it works.

--- a/nixos/modules/services/audio/mopidy.nix
+++ b/nixos/modules/services/audio/mopidy.nix
@@ -12,7 +12,7 @@ let
   mopidyConf = writeText "mopidy.conf" cfg.configuration;

   mopidyEnv = python.buildEnv.override {
-    extraLibs = [ mopidy ] ++ cfg.extensionPackages;
+    extraLibs =  cfg.extensionPackages;
   };

 in {
@@ -64,15 +64,18 @@ in {

   ###### implementation

-  config = mkIf cfg.enable {
+  config = let
+    pythonSearchPath = makeSearchPathOutput "lib" pkgs.python.sitePackages [ mopidyEnv ];
+  in mkIf cfg.enable {

     systemd.services.mopidy = {
       wantedBy = [ "multi-user.target" ];
       after = [ "network.target" "sound.target" ];
       description = "mopidy music player daemon";
       preStart = "mkdir -p ${cfg.dataDir} && chown -R mopidy:mopidy  ${cfg.dataDir}";
+      environment.PYTHONPATH = pythonSearchPath;
       serviceConfig = {
-        ExecStart = "${mopidyEnv}/bin/mopidy --config ${concatStringsSep ":" ([mopidyConf] ++ cfg.extraConfigFiles)}";
+        ExecStart = "${mopidy}/bin/mopidy --config ${concatStringsSep ":" ([mopidyConf] ++ cfg.extraConfigFiles)}";
         User = "mopidy";
         PermissionsStartOnly = true;
       };
@@ -81,6 +84,7 @@ in {
     systemd.services.mopidy-scan = {
       description = "mopidy local files scanner";
       preStart = "mkdir -p ${cfg.dataDir} && chown -R mopidy:mopidy  ${cfg.dataDir}";
+      environment.PYTHONPATH = pythonSearchPath;
       serviceConfig = {
         ExecStart = "${mopidyEnv}/bin/mopidy --config ${concatStringsSep ":" ([mopidyConf] ++ cfg.extraConfigFiles)} local scan";
         User = "mopidy";

[andir]

My bisecting lead to 40851a4d26daabf1613c9a79e638aaf9f94b476c. I am currently verifying that by running mopidy on unstable with that commit reverted (it is still building).

Maybe @FRidh could have a look here? I also have the suspicion that #32372 is due to the same commit.

[FRidh]

I think its due to https://github.com/NixOS/nixpkgs/pull/30606 which permits only Python modules inside extraLibs. Mopidy is a Python application, therefore not marked as a package providing Python modules, and as such its not included in the environment.

What are the cfg.extensionPackages for? Are these packages that Mopidy needs to be able to import from? Or do they need to be on PATH?

[andir]

From what I can tell mopidy tries to discover all installed plugins (e.g. mopidy-soundcloud) and imports them: https://github.com/mopidy/mopidy/blob/develop/mopidy/ext.py#L199
Each extension provides an entry point: https://github.com/mopidy/mopidy-soundcloud/blob/master/setup.py#L34

My best guess would be that it just needs them within the PYTHONPATH. No binaries are being installed by them (that are relevant for core behavior as far as I can tell)

[mattfredwill]

Modifying the extensions to use buildPythonPackage seems to work fine with @andir's patch, I've tested mopidy-spotify and mopidy-soundcloud.

I'm experiencing a new separate issue where a gstreamer decoder for mp3 cannot be found, so I cannot play soundcloud or local files, but mopidy-soundcloud is still loaded, and mopidy-spotify works fine.

I don't know if using buildPythonPackage is the most idiomatic solution, but it does work to get the service up and running and it makes sense to me.

[rvolosatovs]

With the patch applied, mopidy starts, but extensions do not get loaded.
I tried to fix the issues here: https://github.com/rvolosatovs/nixpkgs/commit/177ece5795e70d225eaca91d49ecae5997eb9901

Here's what I'm getting on nixos-rebuild:

tests.test_library (unittest.loader.ModuleImportFailure) ... ERROR

======================================================================
ERROR: tests.test_library (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.test_library
Traceback (most recent call last):
  File "/nix/store/pjxvh01jhjm9m62lwm5vbnn0ml5xd35s-python-2.7.14/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/nix/store/pjxvh01jhjm9m62lwm5vbnn0ml5xd35s-python-2.7.14/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/tmp/nix-build-python2.7-mopidy-local-images-1.0.0.drv-0/source/tests/test_library.py", line 13, in <module>
    from mopidy.audio import scan
  File "/nix/store/afx4762bgy6ivlvhrv4g0xqqia2p3wqr-mopidy-2.1.0/lib/python2.7/site-packages/mopidy/audio/__init__.py", line 4, in <module>
    from .actor import Audio
  File "/nix/store/afx4762bgy6ivlvhrv4g0xqqia2p3wqr-mopidy-2.1.0/lib/python2.7/site-packages/mopidy/audio/actor.py", line 10, in <module>
    from mopidy.audio import tags as tags_lib, utils
  File "/nix/store/afx4762bgy6ivlvhrv4g0xqqia2p3wqr-mopidy-2.1.0/lib/python2.7/site-packages/mopidy/audio/tags.py", line 10, in <module>
    from mopidy.internal.gi import GLib, Gst
  File "/nix/store/afx4762bgy6ivlvhrv4g0xqqia2p3wqr-mopidy-2.1.0/lib/python2.7/site-packages/mopidy/internal/gi.py", line 9, in <module>
    gi.require_version('Gst', '1.0')
  File "/nix/store/cbgp2caaq75l2ns703yawmrnalx84y2l-python2.7-pygobject-3.26.1/lib/python2.7/site-packages/gi/__init__.py", line 130, in require_version
    raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace Gst not available


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0>
builder for ‘/nix/store/6cg1qq7h1cznljmh040ws25fwb0i4cj6-python2.7-mopidy-local-images-1.0.0.drv’ failed with exit code 1

Something's wrong with the way gstreamer is linked in mopidy-local-images. If I disable the extension and everything depending on it, it builds, but music from Spotify does not play:

Dec 19 19:29:49 neon mopidy[2536]: ERROR    GStreamer error: gst-resource-error-quark: Failed to connect: Connection refused (1)

[yegortimoshenko]

Hi @rvolosatovs! I think I've fixed mopidy in 85b84527f636c60bd8c0f0567bb471d491fb5a89. Could you check?

[rvolosatovs]

Thanks! Indeed, now mopidy properly starts and I can also access the (web) plugins.
~I still can not play music though:~ nevermind, that was my setup problem.
Awesome!