Nixpkgs: Python 2.7 Full: sqlite3 module doesn't work in virtualenv

[domenkozar]

>>> import sqlite3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/yzj6p5f7iyh247pwxrg97y3klm6d0cni-python-2.7.3/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/nix/store/yzj6p5f7iyh247pwxrg97y3klm6d0cni-python-2.7.3/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: No module named _sqlite3

@chaoflow @garbas can you reproduce?

[domenkozar]

Seems like C compiled module doesn't get propagated.

[cillianderoiste]

IIUC the typical python we use in nix is a wrapper which explicitly adds the paths to some c modules such as readline and sqlite3. virtualenv puts a copy of the unwrapped python into the virtualenv it creates, so it knows nothing about the python c modules.

One way to work with this is to install e.g. python27Packages.sqlite3 into a profile and add the site-packages path in that profile to your PYTHONPATH. From there you should be able to use the virtualenv as normal. I think it's handy to put all the c python modules in the nix profile and plain python stuff in the virtualenv. This is something like the setup described here: https://nixos.org/wiki/Using_Profiles_for_Development_Environments

@chaoflow has a more advanced setup which can be used to bootstrap and maintain a familiar python development environment: https://github.com/chaoflow/tpv.http/blob/master/Makefile

[domenkozar]

What about providing virtualenv a python with all modules available?

[cillianderoiste]

For example, patching virtualenv so that it copies the nix python wrapper instead of the python binary into the virtualenv? The problem with that is it would get garbage collected. If, instead we link to the current version of the wrapped python (e.g. /run/current-system/sw/bin/python) that will change with any update, also potentially breaking the virtualenv. IIUC this is always a problem with using virtualenv on nix without a dedicated profile (or similar). Any dependencies provided by nix may disappear after garbage collection, e.g. libxml for lxml which would break the virtualenv.

Maybe you have something else in mind?

I guess the motivation is to have virtualenv work on nix in the same way as it would on other distros. After upgrading a conventional distro don't you run into problems with virtualenv too? How does it work on gentoo? Maybe I'm imagining problems that don't exist :)

[peti]

The problem with [virtualenv so that it copies the nix python wrapper instead of the python binary] is it would get garbage collected.

My understanding is that if the virtualenv derivation refers to pythonFull -- which it uses to copy that script into generated environments --, then pythonFull would not be garbage collected as long as someone references virtualenv.

Or am I missing something?

[cillianderoiste]

Isn't pythonFull the wrapped version of python? So IIUC, you would install virtualenv, then use it to create a virtualenv, which would copy the wrapped python into venv/bin/python then you'd update your system and garbage collect. Nix wouldn't know about your copy of the wrapped python in venv/bin/python, right? So the wrapper would still have the old paths. I guess the best way to find out is to try it, but I don't see how it would work myself.

[peti]

Okay, I see your point. I don't understand, though, how use of plain
python instead of python-wrapper alleviates that particular issue in any
way? If python is re-built, then a virtual environment generated with
the old version has a dangling reference, too, doesn't it?

[cillianderoiste]

How about using myEnvFun to create your environment with pythonFull, you can also install virtualenv and use it in that environment https://nixos.org/wiki/Howto_develop_software_on_nixos?

[domenkozar]

ln -s /nix/store/ppag7jq94mbq3wy1m4ah7z3jdhshgsma-python-sqlite3-2.7.5/lib/python2.7/site-packages/_sqlite3.so lib/python2.7/site-packages/ works, I'm still figuring out why it's not done

[domenkozar]

@chaoflow @cillianderoiste what about putting following at the end of activate script:

export PYTHONPATH=/nix/store/c6463w4i2cg23sblknkqg3kr0k365yba-python2.7-virtualenv-1.10/lib/python2.7/site-packages:/nix/store/z7pci148ppkfkql275n00qvmf6brfszs-python-readline-2.7.5/lib/python2.7/site-packages:/nix/store/ppag7jq94mbq3wy1m4ah7z3jdhshgsma-python-sqlite3-2.7.5/lib/python2.7/site-packages:/nix/store/79pdp574ifsfpdw4dg5vhn1wwqqgqjqm-python-curses-2.7.5/lib/python2.7/site-packages:/nix/store/jphvspsmv66fx7m183ffzyblx3vqgdy7-python-recursive-pth-loader-1.0/lib/python2.7/site-packages:/nix/store/hhaf38aymksj1srybhr81xdd1l71sj60-python2.7-setuptools-0.9.8/lib/python2.7/site-packages```

[cillianderoiste]

I think I'd prefer to add the paths to the current profile site-packages directory ~/.nix-profile/lib/python2.7/site-packages/ (including the recursivePthLoader) rather than the /nix/store paths, since that should survive more updates. We should also take care to the right thing inside a myEnv environment.

Perhaps we just need to add a default virtualenv.ini or otherwise specify extra-search-dir parameters (if that does what I think it does). https://pypi.python.org/pypi/virtualenv#environment-variables-and-configuration-files

[domenkozar]

Yeah, let's put the symlinks from profile (no idea how to do that, but let's see).

extra-search-dir only tells virtualenv where to find setuptools/pip eggs (which we should use to achieve better purity), so that's not really an option.

From #pypa:

23:50 < iElectric> I'm still searching for a way to tell virtualenv about "extra packages"
23:51 < iElectric> the idea is to whitelist what system packages can it see
23:56 < dstufft> iElectric: don't think you can easily right now
23:56 < dstufft> without symlinking
23:57 < dstufft> I think Nick was working on something to enable that in the import lib Sig though

[brodul]

Tnx for the workaround. I run into this issue to.

[shlevy]

@iElectric What's the status on this? IMO having references to store paths outside the store (other than (possibly indirect) gcroots) is generally a bad idea.

[domenkozar]

Virtualenv is still broken when python updates. We need to fix this somehow.

[shlevy]

Is it urgent enough that we should mark this for the 14.04 milestone?

[domenkozar]

Since the problem exists from the very beginning, I don't think it's urgent.
On Apr 5, 2014 10:57 PM, "Shea Levy" [email protected] wrote:

Is it urgent enough that we should mark this for the 14.04 milestone?

—
Reply to this email directly or view it on GitHubhttps://github.com/NixOS/nixpkgs/issues/492#issuecomment-39651966
.

[cillianderoiste]

I noticed that the following works:

virtualenv py27; PYTHONHOME=/nix/store/v96gd7xfgr9llf3kj968wlppcmvfsqx8-python-2.7.6-wrapper py27/bin/python -c "import sqlite3"

I was thinking about patching the virtualenv generated site.py to add the relevant path(s) to sys.path. It's amazing, but the contents of site.py are actually inline, and compressed: https://github.com/pypa/virtualenv/blob/develop/virtualenv.py#L1847 so that could be tricky.

Perhaps modifying it after it gets written out https://github.com/pypa/virtualenv/blob/develop/virtualenv.py#L1193 (using python, patched into the virtualenv.py file via the nix expression) would work, but it's all a bit crazy. The code would need to get the path to the wrapper for the version of python specified e.g. virtualenv --python=python2.7 and then possibly add the corresponding lib/pythonX.X/site-packages to sys.path, and maybe also process the .pth files.

Maybe there's an easier way to tell it to do whatever python normally does with PYTHONHOME. Unless there are better ideas, I'll play around with that a bit.

[cillianderoiste]

There's an easier way. We can get virtualenv to create a sitecustomize.py with:

import sys
sys.path.append("/nix/store/7j2sss6x13p7mxm3wwf7jhdjjj9k8vrq-python-2.7.6-wrapper/lib/python2.7/site-packages")

[domenkozar]

It has to point to the profile, otherwise once the wrapper is gc, it will break again.

[cillianderoiste]

I don't think there's any way around that though. I think the virtualenv needs to be recreated after garbage collection e.g.

$ virtualenv venv
$ ls -l venv/lib/*
total 468
lrwxrwxrwx 1 goibhniu users    81 Apr 17 18:44 _abcoll.py -> /nix/store/jr5gs24iyhmcd9zj3fd3r3i2x293fn4d-python-2.7.6/lib/python2.7/_abcoll.py
-r--r--r-- 1 goibhniu users 28144 Apr 17 18:44 _abcoll.pyc
lrwxrwxrwx 1 goibhniu users    77 Apr 17 18:44 abc.py -> /nix/store/jr5gs24iyhmcd9zj3fd3r3i2x293fn4d-python-2.7.6/lib/python2.7/abc.py
-r--r--r-- 1 goibhniu users  6484 Apr 17 18:44 abc.pyc
lrwxrwxrwx 1 goibhniu users    80 Apr 17 18:44 codecs.py -> /nix/store/jr5gs24iyhmcd9zj3fd3r3i2x293fn4d-python-2.7.6/lib/python2.7/codecs.py
...

I think the best we can do here is to make virtualenv provide the same modules provided by the wrapper by default.

EDIT: brevity

[cillianderoiste]

This seems to work:

diff --git a/virtualenv.py b/virtualenv.py
index 0f5ae79..cfd19bd 100755
--- a/virtualenv.py
+++ b/virtualenv.py
@@ -1191,6 +1191,11 @@ def install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages, clear, sy
     site_filename_dst = change_prefix(site_filename, home_dir)
     site_dir = os.path.dirname(site_filename_dst)
     writefile(site_filename_dst, SITE_PY)
+    wrapper_path = join(prefix, "lib", py_version, "site-packages")
+    writefile(
+        join(site_dir, 'sitecustomize.py',),
+        "import sys; sys.path.append('%s')" % wrapper_path
+    )
     writefile(join(site_dir, 'orig-prefix.txt'), prefix)
     site_packages_filename = join(site_dir, 'no-global-site-packages.txt')
     if not site_packages:

[domenkozar]

it's an improvement, so +1

[cillianderoiste]

Cool! I'll do a bit more testing and if it looks OK (and there are no concerns) I'll push it tomorrow.

[cillianderoiste]

I'll close this since I believe the initial issue is solved.

We could create a new ticket to tackle the issue of virtualenvs surviving garbage collection, but I'm not sure there is a nice way. At any rate, we should document the situation and encourage people to use nix instead of virtualenv where possible and otherwise to recreate their virtualenvs after garbage collection.

[domenkozar]

Good job!

[copumpkin]

I'm getting this with just nix-shell and straight nix python (with no virtualenv). Should I open a separate ticket?

let
  pkgs = import <nixpkgs> {};
in
{ stdenv ? pkgs.stdenv }:

with pkgs; stdenv.mkDerivation {
  name = "python-nix";
  version = "0.1.0.0";
  src = ./.;
  buildInputs = [ python nixops sqlite ];
}

results in an environment that can't import sqlite3 properly.

[copumpkin]

Oh never mind, I needed pythonPackages.sqlite3

[jgeerds]

This also affects the tox test runner (which uses an unpatched version of virtualenv internally). This means that it is not possible to execute python2.7 tests for a django project/app because of the missing _sqlite3 module.

[domenkozar]

I think we should include ncurses and sqlite by default. They're both very light and convenient.

[FRidh]

@jgeerds could you show an example of how you use the tox test runner with Nix?

[FRidh]

Since https://github.com/NixOS/nixpkgs/pull/19309 we don't have separate modules anymore except tkinter, which is like a 'normal' Python package: pythonPackages.tkinter.