I'm currently working on packaging for NixOS: https://github.com/NixOS/nixpkgs/issues/49438
I may ask for help regarding the integration.
CMCDragonkai
All 34 comments
Hi @CMCDragonkai !
Great stuff! Thank you for your work! Please don't hesitate to ask any questions, we'll be happy to help 🙂
efiop
on 17 Jan 2019
@CMCDragonkai Also note that we have binary packages as well. For example, our deb and rpm packages contain dvc built with pyinstaller with all of the possible dependencies, so it doesn't rely on anything on the system and carries all the libs(including python itself) with it. You should be able to utilize that as well, if you are not keen on packaging every pip dependency separately. See scripts/build_posix.sh for more info.
efiop
on 17 Jan 2019
Nix prefers to get everything built from source barring proprietary applications.
CMCDragonkai
on 17 Jan 2019
I've packaged all the dependencies for dvc into NixOS. Now I'm onto packaging dvc itself. However your pip installation doesn't have a source distribution. NixOS prefers to build things from the source. Can you push up your source distributions to PyPi as well? https://pypi.org/project/dvc/#files
Note that source distribution should contain any tests for dvc as well.
CMCDragonkai
on 24 Jan 2019
@CMCDragonkai , what if instead of fetching from pypi you set something like:
src = fetchurl {
url = "https://github.com/iterative/dvc/archive/${pkgver}.tar.gz";
sha256 = "sha of the tar.gz";
};
I don't know if it's possible, but that's what I'm currently doing for the Arch build: PKGBUILD.
However, I think it is a good idea to release the source code to pypi
ghost
on 24 Jan 2019
By the way, great work so far, @CMCDragonkai ! I noticed that you also packaged th missing dependencies on nixpkgs :+1: !
Regarding your question about the optional dependencies, I've seen both enableX and supportX (e.g. supportSSH)
_NOTE: remember to include HDFS as a remote_
ghost
on 24 Jan 2019
I can use fetchFromGithub instead that's usually the standard as long as Github releases are considered as official as PyPI releases then it's fine.
CMCDragonkai
on 24 Jan 2019
Yes, you can rely on GitHub releases, as far as I know, we are supporting them :smiley:
ghost
on 24 Jan 2019
Ok building from source results in:
»» ~/Projects/nixpkgs
♜ nix-build ./default.nix --attr 'dvc' (dvc) pts/8 17:30:56
these derivations will be built:
/nix/store/rs84jf5z5bmra71ps6ikaw1jx9jp2wvh-dvc-0.24.0.drv
building '/nix/store/rs84jf5z5bmra71ps6ikaw1jx9jp2wvh-dvc-0.24.0.drv'...
unpacking sources
unpacking source archive /nix/store/wj4qqj4qvinvrgm1njq65klfdh6ngzx6-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/tests/utils/httpd.py
patching sources
configuring
building
Traceback (most recent call last):
File "/nix/store/y2ck276xvrpz7l6yghzyx1qy2kyw19aq-python3.7-GitPython-2.1.11/lib/python3.7/site-packages/git/__init__.py", line 83, in <module>
refresh()
File "/nix/store/y2ck276xvrpz7l6yghzyx1qy2kyw19aq-python3.7-GitPython-2.1.11/lib/python3.7/site-packages/git/__init__.py", line 73, in refresh
if not Git.refresh(path=path):
File "/nix/store/y2ck276xvrpz7l6yghzyx1qy2kyw19aq-python3.7-GitPython-2.1.11/lib/python3.7/site-packages/git/cmd.py", line 290, in refresh
raise ImportError(err)
ImportError: Bad git executable.
The git executable must be specified in one of the following ways:
- be included in your $PATH
- be set via $GIT_PYTHON_GIT_EXECUTABLE
- explicitly set via git.refresh()
All git commands will error until this is rectified.
This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
- quiet|q|silence|s|none|n|0: for no warning or exception
- warn|w|warning|1: for a printed warning
- error|e|raise|r|2: for a raised exception
Example:
export GIT_PYTHON_REFRESH=quiet
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "nix_run_setup", line 8, in <module>
exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
File "setup.py", line 5, in <module>
from dvc import VERSION
File "/build/source/dvc/__init__.py", line 53, in <module>
__version__ = _update_version_file()
File "/build/source/dvc/__init__.py", line 20, in _update_version_file
from git import Repo
File "/nix/store/y2ck276xvrpz7l6yghzyx1qy2kyw19aq-python3.7-GitPython-2.1.11/lib/python3.7/site-packages/git/__init__.py", line 85, in <module>
raise ImportError('Failed to initialize: {0}'.format(exc))
ImportError: Failed to initialize: Bad git executable.
The git executable must be specified in one of the following ways:
- be included in your $PATH
- be set via $GIT_PYTHON_GIT_EXECUTABLE
- explicitly set via git.refresh()
All git commands will error until this is rectified.
This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
- quiet|q|silence|s|none|n|0: for no warning or exception
- warn|w|warning|1: for a printed warning
- error|e|raise|r|2: for a raised exception
Example:
export GIT_PYTHON_REFRESH=quiet
builder for '/nix/store/rs84jf5z5bmra71ps6ikaw1jx9jp2wvh-dvc-0.24.0.drv' failed with exit code 1
error: build of '/nix/store/rs84jf5z5bmra71ps6ikaw1jx9jp2wvh-dvc-0.24.0.drv' failed
The tests require the existence of git. So I add that into buildInputs.
Furthermore the tests also require nosetests. But not available.
CMCDragonkai
on 24 Jan 2019
It appears that test dependencies need quite a lot more stuff: https://github.com/iterative/dvc/blob/master/test-requirements.txt
I'd like to know more about your test enviornment. Even after adding all the dependencies, the first error I get is:
Error: Failed to load dir cache '../../b/uild/dvc-test.4363.yk9hglkp.0d76514b-9d20-44bd-a35e-0abe6b4b2720/.dvc/cache/12/3.dir' - [Errno 2] No such file or directory: '/b/uild/dvc-test.4363.yk9hglkp.0d76514b-9d20-44bd-a35e-0abe6b4b2720/.dvc/cache/12/3.dir'
I just realised that some of the tests call awscli which means it relies on test AWS services. Does this mean the tests are meant to be executed with DVC's private testing servers? In that case is there a way to only test dvc with unattended tests? If not, I'll have to disable tests.
CMCDragonkai
on 24 Jan 2019
Another question, is that when I have finally built the dvc script. Running the script results in this message saying:
/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/bin/python3.7: No module named dvc
For simplicity, @CMCDragonkai, I would recommend you to disable the tests (we are already testing the code on our CI, so we are covered), we can enable them after fixing those dependency issues.
About the No module named dvc error, is hard to tell :thinking:
Do you have more details about it? What about running a shell and import dvc
ghost
on 24 Jan 2019
The PR for Nixpkgs is here: https://github.com/NixOS/nixpkgs/pull/54530
I've disabled all tests. So the last thing to figure out is that No module named dvc.
CMCDragonkai
on 24 Jan 2019
@mroutis That error is quite strange. It's an essentially an import error. However since dvc is imported everywhere obviously it can be imported otherwise the the whole thing should fail. Since it only reports it at the end. Maybe there's something that gets executed at the end that tries to import dvc in some special way and fails to do so?
And yes it's perfectly possible to import dvc.
CMCDragonkai
on 24 Jan 2019
@CMCDragonkai, if you start a Python interpreter and enter import dvc, does it works? I don't know if paths should be updated or if the shell has enough information to look at the package;
I would give it a try tomorrow :)
ghost
on 24 Jan 2019
Yes it does inside the build environment. But dvc is installed as an application, not a library. So my normal python interpreter won't be able to find it. NixOS considers apps and libraries separately. Hmm I'm wondering what is happening is that something inside dvc script calls python interpreter from the outside and imports dvc, if thats true then it would fail.
CMCDragonkai
on 24 Jan 2019
@CMCDragonkai , possibly bin/dvc?
#!/bin/bash
PYTHONPATH=$DVC_HOME python -m dvc $@
That's ignored by the Nix expression because you are using setup.py which doesn't reference that script. Instead it only uses this:
https://github.com/iterative/dvc/blob/master/setup.py#L73-L75
Or the scripts setting in setup.py which you don't have.
So it shouldn't be that bin/dvc script. In fact that script does not even exist at the build outputs.
CMCDragonkai
on 25 Jan 2019
@CMCDragonkai That is correct. It is hard to point out to the specific thing that had broke, since I'm not familiar with NixOS packages. I would take a look at other similar tools written in python and nixos packages for them. For example https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/admin/awscli/default.nix . Did you use a similar project as a template or did you create it from scratch?
efiop
on 25 Jan 2019
@efiop Yes, here's the PR: https://github.com/NixOS/nixpkgs/pull/54530/files
It's a very standard python application. I'm suspecting what I'm saying is true, somewhere during the execution of dvc, it tries to import dvc somewhere and that fails, but it doesn't cause a critical failure since the commands still work.
CMCDragonkai
on 25 Jan 2019
@CMCDragonkai Hm, thinking about it, it is probably our analytics daemon launching https://github.com/iterative/dvc/blob/master/dvc/daemon.py#L71 . Here is where it tries to run sys.executable -m dvc.
efiop
on 25 Jan 2019
We could probably change it up, so that it figures out __main__.py path from __file__ and runs it directly with something like sys.executable /path/to/dvc/__main__.py. Which not only should solve the issue you are having, but also be a more correct way of launching analytics daemon.
efiop
on 25 Jan 2019
That might be true. Here's an example:
»» ~/Projects/dvctest
♖ dvc add data/Posts.xml.zip pts/11 11:54:08
/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/bin/python3.7: No module named dvc
Adding 'data/Posts.xml.zip' to 'data/.gitignore'.
Saving 'data/Posts.xml.zip' to cache '.dvc/cache'.
Saving information to 'data/Posts.xml.zip.dvc'.
To track the changes with git run:
git add data/.gitignore data/Posts.xml.zip.dvc
/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/bin/python3.7: No module named dvc
See there's one report at the beginning and one report at the end.
CMCDragonkai
on 25 Jan 2019
Is dvc daemon something that gets launched in the background and continues to run between dvc invocations?
CMCDragonkai
on 25 Jan 2019
@CMCDragonkai Not really :slightly_smiling_face: It is kinda of a misleading name for what is actually a single-run worker, that sends analytics and then just dies.
efiop
on 25 Jan 2019
@CMCDragonkai I've submitted a PR https://github.com/iterative/dvc/pull/1531 that should help with making dvc work properly in your package. We'll release new dvc version with it ASAP, I'll try to notify you here when it is ready, so you could try it out.
efiop
on 26 Jan 2019
@CMCDragonkai 0.24.3 is out. Please give it a try.
efiop
on 26 Jan 2019
@efiop Unfortunately that's not sufficient to fix the problem. It is a step in the right direction.
Traceback (most recent call last): pts/10 11:48:04
File "/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/lib/python3.7/site-packages/dvc/__main__.py", line 5, in <module>
from dvc.main import main
File "/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/lib/python3.7/site-packages/dvc/main.py", line 3, in <module>
import dvc.logger as logger
File "/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/lib/python3.7/site-packages/dvc/logger.py", line 10, in <module>
import colorama
ModuleNotFoundError: No module named 'colorama'
Basically after successfully calling dvc daemon, which then tries to call dvc/__main__.py and then dvc/main.py which then calls dvc/logger.py, the first import of a foreign package is import colorama. That fails, because that script is not wrapped with the environmental dependencies that Nix provides. This is because the python environment that Nix produces is a wrapper around the bin script.
See this the package that is built by Nix:
.
├── bin
│ ├── dvc
│ └── .dvc-wrapped
├── lib
│ └── python3.7
│ └── site-packages
│ ├── dvc
│ │ ├── analytics.py
│ │ ├── cache.py
│ │ ├── cli.py
│ │ ├── command
│ │ │ ├── add.py
│ │ │ ├── base.py
│ │ │ ├── cache.py
│ │ │ ├── checkout.py
│ │ │ ├── config.py
│ │ │ ├── daemon.py
│ │ │ ├── data_sync.py
│ │ │ ├── destroy.py
│ │ │ ├── gc.py
│ │ │ ├── imp.py
│ │ │ ├── __init__.py
│ │ │ ├── init.py
│ │ │ ├── install.py
│ │ │ ├── lock.py
│ │ │ ├── metrics.py
│ │ │ ├── move.py
│ │ │ ├── pipeline.py
│ │ │ ├── __pycache__
│ │ │ │ ├── add.cpython-37.pyc
│ │ │ │ ├── base.cpython-37.pyc
│ │ │ │ ├── cache.cpython-37.pyc
│ │ │ │ ├── checkout.cpython-37.pyc
│ │ │ │ ├── config.cpython-37.pyc
│ │ │ │ ├── daemon.cpython-37.pyc
│ │ │ │ ├── data_sync.cpython-37.pyc
│ │ │ │ ├── destroy.cpython-37.pyc
│ │ │ │ ├── gc.cpython-37.pyc
│ │ │ │ ├── imp.cpython-37.pyc
│ │ │ │ ├── __init__.cpython-37.pyc
│ │ │ │ ├── init.cpython-37.pyc
│ │ │ │ ├── install.cpython-37.pyc
│ │ │ │ ├── lock.cpython-37.pyc
│ │ │ │ ├── metrics.cpython-37.pyc
│ │ │ │ ├── move.cpython-37.pyc
│ │ │ │ ├── pipeline.cpython-37.pyc
│ │ │ │ ├── remote.cpython-37.pyc
│ │ │ │ ├── remove.cpython-37.pyc
│ │ │ │ ├── repro.cpython-37.pyc
│ │ │ │ ├── root.cpython-37.pyc
│ │ │ │ ├── run.cpython-37.pyc
│ │ │ │ ├── status.cpython-37.pyc
│ │ │ │ └── unprotect.cpython-37.pyc
│ │ │ ├── remote.py
│ │ │ ├── remove.py
│ │ │ ├── repro.py
│ │ │ ├── root.py
│ │ │ ├── run.py
│ │ │ ├── status.py
│ │ │ └── unprotect.py
│ │ ├── config.py
│ │ ├── daemon.py
│ │ ├── dagascii.py
│ │ ├── data_cloud.py
│ │ ├── dependency
│ │ │ ├── base.py
│ │ │ ├── gs.py
│ │ │ ├── hdfs.py
│ │ │ ├── http.py
│ │ │ ├── __init__.py
│ │ │ ├── local.py
│ │ │ ├── __pycache__
│ │ │ │ ├── base.cpython-37.pyc
│ │ │ │ ├── gs.cpython-37.pyc
│ │ │ │ ├── hdfs.cpython-37.pyc
│ │ │ │ ├── http.cpython-37.pyc
│ │ │ │ ├── __init__.cpython-37.pyc
│ │ │ │ ├── local.cpython-37.pyc
│ │ │ │ ├── s3.cpython-37.pyc
│ │ │ │ └── ssh.cpython-37.pyc
│ │ │ ├── s3.py
│ │ │ └── ssh.py
│ │ ├── exceptions.py
│ │ ├── __init__.py
│ │ ├── istextfile.py
│ │ ├── lock.py
│ │ ├── logger.py
│ │ ├── __main__.py
│ │ ├── main.py
│ │ ├── output
│ │ │ ├── base.py
│ │ │ ├── gs.py
│ │ │ ├── hdfs.py
│ │ │ ├── __init__.py
│ │ │ ├── local.py
│ │ │ ├── __pycache__
│ │ │ │ ├── base.cpython-37.pyc
│ │ │ │ ├── gs.cpython-37.pyc
│ │ │ │ ├── hdfs.cpython-37.pyc
│ │ │ │ ├── __init__.cpython-37.pyc
│ │ │ │ ├── local.cpython-37.pyc
│ │ │ │ ├── s3.cpython-37.pyc
│ │ │ │ └── ssh.cpython-37.pyc
│ │ │ ├── s3.py
│ │ │ └── ssh.py
│ │ ├── progress.py
│ │ ├── project.py
│ │ ├── prompt.py
│ │ ├── __pycache__
│ │ │ ├── analytics.cpython-37.pyc
│ │ │ ├── cache.cpython-37.pyc
│ │ │ ├── cli.cpython-37.pyc
│ │ │ ├── config.cpython-37.pyc
│ │ │ ├── daemon.cpython-37.pyc
│ │ │ ├── dagascii.cpython-37.pyc
│ │ │ ├── data_cloud.cpython-37.pyc
│ │ │ ├── exceptions.cpython-37.pyc
│ │ │ ├── __init__.cpython-37.pyc
│ │ │ ├── istextfile.cpython-37.pyc
│ │ │ ├── lock.cpython-37.pyc
│ │ │ ├── logger.cpython-37.pyc
│ │ │ ├── __main__.cpython-37.pyc
│ │ │ ├── main.cpython-37.pyc
│ │ │ ├── progress.cpython-37.pyc
│ │ │ ├── project.cpython-37.pyc
│ │ │ ├── prompt.cpython-37.pyc
│ │ │ ├── stage.cpython-37.pyc
│ │ │ ├── state.cpython-37.pyc
│ │ │ ├── system.cpython-37.pyc
│ │ │ └── updater.cpython-37.pyc
│ │ ├── remote
│ │ │ ├── azure.py
│ │ │ ├── base.py
│ │ │ ├── gs.py
│ │ │ ├── hdfs.py
│ │ │ ├── http.py
│ │ │ ├── __init__.py
│ │ │ ├── local.py
│ │ │ ├── __pycache__
│ │ │ │ ├── azure.cpython-37.pyc
│ │ │ │ ├── base.cpython-37.pyc
│ │ │ │ ├── gs.cpython-37.pyc
│ │ │ │ ├── hdfs.cpython-37.pyc
│ │ │ │ ├── http.cpython-37.pyc
│ │ │ │ ├── __init__.cpython-37.pyc
│ │ │ │ ├── local.cpython-37.pyc
│ │ │ │ ├── s3.cpython-37.pyc
│ │ │ │ └── ssh.cpython-37.pyc
│ │ │ ├── s3.py
│ │ │ └── ssh.py
│ │ ├── scm
│ │ │ ├── base.py
│ │ │ ├── git.py
│ │ │ ├── __init__.py
│ │ │ └── __pycache__
│ │ │ ├── base.cpython-37.pyc
│ │ │ ├── git.cpython-37.pyc
│ │ │ └── __init__.cpython-37.pyc
│ │ ├── stage.py
│ │ ├── state.py
│ │ ├── system.py
│ │ ├── updater.py
│ │ └── utils
│ │ ├── compat.py
│ │ ├── __init__.py
│ │ └── __pycache__
│ │ ├── compat.cpython-37.pyc
│ │ └── __init__.cpython-37.pyc
│ ├── dvc-0.24.3.dist-info
│ │ ├── entry_points.txt
│ │ ├── INSTALLER
│ │ ├── LICENSE
│ │ ├── METADATA
│ │ ├── RECORD
│ │ ├── top_level.txt
│ │ └── WHEEL
│ └── tests
│ ├── unit
│ │ ├── daemon.py
│ │ ├── __init__.py
│ │ ├── __pycache__
│ │ │ ├── daemon.cpython-37.pyc
│ │ │ ├── __init__.cpython-37.pyc
│ │ │ ├── stage.cpython-37.pyc
│ │ │ └── state.cpython-37.pyc
│ │ ├── remote
│ │ │ ├── azure.py
│ │ │ ├── base.py
│ │ │ ├── gs.py
│ │ │ ├── __init__.py
│ │ │ ├── __pycache__
│ │ │ │ ├── azure.cpython-37.pyc
│ │ │ │ ├── base.cpython-37.pyc
│ │ │ │ ├── gs.cpython-37.pyc
│ │ │ │ ├── __init__.cpython-37.pyc
│ │ │ │ └── ssh.cpython-37.pyc
│ │ │ └── ssh.py
│ │ ├── stage.py
│ │ └── state.py
│ └── utils
│ ├── httpd.py
│ ├── __init__.py
│ └── __pycache__
│ ├── httpd.cpython-37.pyc
│ └── __init__.cpython-37.pyc
└── nix-support
└── propagated-build-inputs
27 directories, 180 files
The environment produced by Nix is all specified by
bin/dvc. Which is:
#! /nix/store/vs6d2fjkl4kb3jb7rwibsd76k9v2n4xy-bash-4.4-p23/bin/bash -e
export PATH='/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/bin:/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/bin:/nix/store/x54cj2603lvj6y5fmpkg7x0hvad8kp85-python3.7-setuptools-40.6.2/bin:/nix/store/64pfzf4mcgk8350rk9cmn4rfg0v148ah-python3.7-future-0.17.1/bin:/nix/store/qnzpg1iqbwrjhpjc6h99w328gz31lfv9-python3.7-jsonpath-rw-1.4.0/bin:/nix/store/b8hrpwr27x82zqyvch81zrdv8i2igdxm-python3.7-chardet-3.0.4/bin:/nix/store/4wrg4g7blmb4dd8f9vbg9c5qp4r464bp-python3.7-pyfiglet-0.8.post0/bin:/nix/store/2hzvyknrhys2yl6bw4791w3pfml659fl-python3.7-distro-1.3.0/bin'${PATH:+':'}$PATH
export PYTHONNOUSERSITE='true'
exec -a "$0" "/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/bin/.dvc-wrapped" "${extraFlagsArray[@]}" "$@"
And the called wrapped script is:
#!/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/bin/python3.7
# -*- coding: utf-8 -*-
import sys;import site;import functools;sys.argv[0] = '/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/bin/dvc';functools.reduce(lambda k, p: site.addsitedir(p, k), ['/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/lib/python3.7/site-packages','/nix/store/31yh08kpq9hdjyag0429iis36avma2ah-python3.7-ply-3.11/lib/python3.7/site-packages','/nix/store/x54cj2603lvj6y5fmpkg7x0hvad8kp85-python3.7-setuptools-40.6.2/lib/python3.7/site-packages','/nix/store/28n5hxpbqwfpf1mxlk5xlrd8isn0wkyd-python3.7-configparser-3.5.0/lib/python3.7/site-packages','/nix/store/yfa20508fkp32363cdyka87fbw26nffs-python3.7-zc.lockfile-1.4/lib/python3.7/site-packages','/nix/store/vpppyclhskfc79qhnb6qzipglxq7hkcc-python3.7-zope.testing-4.7/lib/python3.7/site-packages','/nix/store/f7lxf28icqxz7kx7j7fp43qcpja20l24-python3.7-zope.interface-4.6.0/lib/python3.7/site-packages','/nix/store/01f9mqlgdqj2plq1flwdgkn08r6bi14w-python3.7-zope.event-4.4/lib/python3.7/site-packages','/nix/store/i8rqa42ci9fma700d9dvcigzzmh6hjya-python3.7-zope.exceptions-4.3/lib/python3.7/site-packages','/nix/store/r6kg1ih7k97z4cdp5r2ab2z2jkd95ii8-python3.7-zope.location-4.2/lib/python3.7/site-packages','/nix/store/bj47fxx4ix1agbqs4q4iqzjxaqr3ywn2-python3.7-zope.proxy-4.3.1/lib/python3.7/site-packages','/nix/store/64pfzf4mcgk8350rk9cmn4rfg0v148ah-python3.7-future-0.17.1/lib/python3.7/site-packages','/nix/store/dfhzqvvr2l7b3cs3l1fs0n197hfarbr1-python3.7-colorama-0.3.9/lib/python3.7/site-packages','/nix/store/gp4vh1czqzm378q9c1fmr7gqnacw2gdr-python3.7-configobj-5.0.6/lib/python3.7/site-packages','/nix/store/ckw5mkgwwhh1j3fn33ysvgicdw8wr4f8-python3.7-six-1.11.0/lib/python3.7/site-packages','/nix/store/pjwlgy35f021mw6bbmvpdi6hjcx6r9v5-python3.7-networkx-2.2/lib/python3.7/site-packages','/nix/store/izdjyd0jkp4mb2pzl67rs1gmy2g68f43-python3.7-decorator-4.3.0/lib/python3.7/site-packages','/nix/store/x5xlhq0gkfjwn8r0w8c1ls43qhwhwxs5-python3.7-PyYAML-3.13/lib/python3.7/site-packages','/nix/store/8zydrhqxlcz39ljx9fplkbs652wws9km-python3.7-GitPython-2.1.11/lib/python3.7/site-packages','/nix/store/iqh4mm2d3yjfc0da78sgbrpjkx847r41-python3.7-gitdb2-2.0.5/lib/python3.7/site-packages','/nix/store/y6hzhdqlba3szsf2k9smp70ypi2fas8h-python3.7-smmap2-2.0.5/lib/python3.7/site-packages','/nix/store/p1kvm1h9d0kc74q3i1nf4c9y6z6nmw4y-python3.7-ddt-1.2.0/lib/python3.7/site-packages','/nix/store/8prggywiwn97glw6svr4spkav9m260vr-python3.7-nanotime-0.5.2/lib/python3.7/site-packages','/nix/store/8l79hrfgs3sqc0x4p544zkxz502wj4y4-python3.7-pyasn1-0.4.4/lib/python3.7/site-packages','/nix/store/j88dn1df0scz96flq1g4wgpphnfbz5a3-python3.7-schema-0.6.8/lib/python3.7/site-packages','/nix/store/qnzpg1iqbwrjhpjc6h99w328gz31lfv9-python3.7-jsonpath-rw-1.4.0/lib/python3.7/site-packages','/nix/store/mk0xch65ygwqq42g5162mncpmnjbgi8r-python3.7-requests-2.20.1/lib/python3.7/site-packages','/nix/store/zils6n7ql04i5njw01y5xlm5b8jvm0hv-python3.7-urllib3-1.24.1/lib/python3.7/site-packages','/nix/store/rv73j22qykp80w1h1b1v0nshwj1fbfsy-python3.7-idna-2.7/lib/python3.7/site-packages','/nix/store/6k3a8bq435wwp4sn6xk1qz205qzxa19l-python3.7-asn1crypto-0.24.0/lib/python3.7/site-packages','/nix/store/fla3svslcmnir7yr8w3is7img0kxdq72-python3.7-packaging-18.0/lib/python3.7/site-packages','/nix/store/2fy8lhi7hryajhh2p527yqh8r51q40ik-python3.7-pyparsing-2.3.0/lib/python3.7/site-packages','/nix/store/wfl54yjr4gwa6y07r5g4sp5l0acz6ds6-python3.7-pycparser-2.19/lib/python3.7/site-packages','/nix/store/6gfsg5k2lw4v513gynbnfaz12yx15ams-python3.7-cffi-1.11.5/lib/python3.7/site-packages','/nix/store/zp061rvpg2faa81k1rnflxmm1d1li4zi-python3.7-cryptography-2.3.1/lib/python3.7/site-packages','/nix/store/sxwiv6yf0ni6wa9y2j5hwsr8dg3yd9y6-python3.7-pyOpenSSL-18.0.0/lib/python3.7/site-packages','/nix/store/83qzrsx5zwym562bscwdgdry47qgpb60-python3.7-certifi-2018.11.29/lib/python3.7/site-packages','/nix/store/grqc8i2sqzmzkzi3dskik91sp4xi3pwy-python3.7-pysocks-1.6.8/lib/python3.7/site-packages','/nix/store/b8hrpwr27x82zqyvch81zrdv8i2igdxm-python3.7-chardet-3.0.4/lib/python3.7/site-packages','/nix/store/mv181w2mrlwy0y68nb0mbv2c8lckw7w1-python3.7-grandalf-0.6/lib/python3.7/site-packages','/nix/store/fqqk8a885y2rb6x386m8rdw2kc0vbapy-python3.7-asciimatics-1.10.0/lib/python3.7/site-packages','/nix/store/4wrg4g7blmb4dd8f9vbg9c5qp4r464bp-python3.7-pyfiglet-0.8.post0/lib/python3.7/site-packages','/nix/store/0s05jsiinkcjf7chyjmqzpnndpnaf4sz-python3.7-Pillow-5.3.0/lib/python3.7/site-packages','/nix/store/1iyg4kndi8yp62is4xh5zmrvcgcp61sl-python3.7-olefile-0.46/lib/python3.7/site-packages','/nix/store/klqxj9aqzwm0yc5wjyqh0hmw94k30av2-python3.7-wcwidth-0.1.7/lib/python3.7/site-packages','/nix/store/2hzvyknrhys2yl6bw4791w3pfml659fl-python3.7-distro-1.3.0/lib/python3.7/site-packages','/nix/store/jkbn8sak87m67p20z58l19qy2r5mxmg3-python3.7-appdirs-1.4.3/lib/python3.7/site-packages'], site._init_pathinfo());
import re
import sys
from dvc.main import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(main())
As you can see the PYTHONPATH required is actually filled by the wrapped script.
I think the only proper way to solve is to not call dvc daemon via the library module method basically using python -m dvc daemon -q. But instead just calling dvc daemon normally using the absolute path to the dvc script. However the dvc script is generated by setup.py, and not a fixed part of the source code. I have usually used the scripts=['bin/something'] in my setup.py instead of the entry_points configuration which allows this to easily work (https://github.com/iterative/dvc/blob/master/setup.py#L73-L75).
So right now I need to ask around about this problem. https://github.com/NixOS/nixpkgs/pull/54530#issuecomment-458368361
CMCDragonkai
on 29 Jan 2019
Hi @CMCDragonkai !
How is it going? :slightly_smiling_face: I see you've introduced a patch that is simply calling dvc daemon directly, which seems to fix the issue for you. It is hard for me to wrap my head around what NixOS packages are doing with that substitution that was suggested to you in https://github.com/NixOS/nixpkgs/pull/54530#discussion_r252581188 . I see that it has been quite a while since your last comment in that discussion, did you decide to use pip instead? Or do you plan to get back to packaging it at some point?
Thanks,
Ruslan
efiop
on 5 Apr 2019
@CMCDragonkai ?
Closing for now as inactive.
efiop
on 18 Jun 2019
The MR is ready to be merged https://github.com/NixOS/nixpkgs/pull/54530 with some help, the patch is ready to be used.
Although in the future, I would be more careful with self-recursive programs with it shelling it out to itself. The path should be ensured to work in all situations.
CMCDragonkai
on 28 Oct 2019
@CMCDragonkai Great news! But I have to point out that 0.24.1 is extremely old now, we've moved a lot since then, and you might have less or more issues now. 0.24.1 is something we would not recommend to be used by anyone these days.
efiop
on 28 Oct 2019
This is a good starting point for the next guy to upgrade it!
On 29 October 2019 01:35:06 GMT+11:00, Ruslan Kuprieiev notifications@github.com wrote:
@CMCDragonkai Great news! But I have to point out that 0.24.1 is
extremely old now, we've moved a lot since then, and you might have
less or more issues now. 0.24.1 is something we would not recommend to
be used by anyone these days.--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/iterative/dvc/issues/1509#issuecomment-546974686
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
CMCDragonkai
on 28 Oct 2019
Related issues
robguinness
·
3Comments
tc-ying
·
3Comments
gregfriedland
·
3Comments
mdscruggs
·
3Comments
ghost
·
3Comments