Ray: Cannot be installed directly from repository

[mitar]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• Ray installed from (source or binary): source (at 1a39fee9c6fa0f81e1184182b47ef106269613fe)
• Python version: 3.6.7
• Exact command to reproduce:

$ pip3 install -e 'git+https://github.com/ray-project/ray.git#subdirectory=python&egg=ray'
Obtaining ray from git+https://github.com/ray-project/ray.git#subdirectory=python&egg=ray
  Cloning https://github.com/ray-project/ray.git to /src/ray
Requirement already satisfied: numpy>=1.14 in /usr/local/lib/python3.6/dist-packages (from ray) (1.16.3)
Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from ray) (3.0.12)
Requirement already satisfied: funcsigs in /usr/local/lib/python3.6/dist-packages (from ray) (1.0.2)
Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from ray) (7.0)
Requirement already satisfied: colorama in /usr/local/lib/python3.6/dist-packages (from ray) (0.4.1)
Requirement already satisfied: pytest in /usr/local/lib/python3.6/dist-packages (from ray) (4.5.0)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from ray) (5.1)
Requirement already satisfied: redis in /usr/local/lib/python3.6/dist-packages (from ray) (3.2.1)
Requirement already satisfied: six>=1.0.0 in /usr/lib/python3/dist-packages (from ray) (1.11.0)
Requirement already satisfied: typing in /usr/local/lib/python3.6/dist-packages (from ray) (3.6.6)
Requirement already satisfied: flatbuffers in /usr/local/lib/python3.6/dist-packages (from ray) (1.11)
Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.6/dist-packages (from pytest->ray) (1.3.0)
Requirement already satisfied: more-itertools>=4.0.0; python_version > "2.7" in /usr/local/lib/python3.6/dist-packages (from pytest->ray) (7.0.0)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from pytest->ray) (0.1.7)
Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from pytest->ray) (1.8.0)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.6/dist-packages (from pytest->ray) (19.1.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from pytest->ray) (41.0.1)
Requirement already satisfied: pluggy!=0.10,<1.0,>=0.9 in /usr/local/lib/python3.6/dist-packages (from pytest->ray) (0.11.0)
Installing collected packages: ray
  Running setup.py develop for ray
    Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/src/ray/python/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps:
    running develop
    running egg_info
    writing ray.egg-info/PKG-INFO
    writing dependency_links to ray.egg-info/dependency_links.txt
    writing entry points to ray.egg-info/entry_points.txt
    writing requirements to ray.egg-info/requires.txt
    writing top-level names to ray.egg-info/top_level.txt
    reading manifest file 'ray.egg-info/SOURCES.txt'
    writing manifest file 'ray.egg-info/SOURCES.txt'
    running build_ext
    + set -e
    +++ dirname ../build.sh
    ++ cd ..
    ++ pwd
    + ROOT_DIR=/src/ray
    ++ uname
    + unamestr=Linux
    + [[ Linux == \L\i\n\u\x ]]
    + PARALLEL=1
    + RAY_BUILD_PYTHON=YES
    + RAY_BUILD_JAVA=NO
    + PYTHON_EXECUTABLE=
    + BUILD_DIR=
    + [[ 2 -gt 0 ]]
    + key=-p
    + case $key in
    + PYTHON_EXECUTABLE=/usr/bin/python3
    + shift
    + shift
    + [[ 0 -gt 0 ]]
    + [[ -z /usr/bin/python3 ]]
    + echo 'Using Python executable /usr/bin/python3.'
    Using Python executable /usr/bin/python3.
    ++ PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/.bazel/bin
    ++ which bazel
    + BAZEL_EXECUTABLE=
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/src/ray/python/setup.py", line 183, in <module>
        license="Apache 2.0")
      File "/usr/local/lib/python3.6/dist-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.6/dist-packages/setuptools/command/develop.py", line 38, in run
        self.install_for_development()
      File "/usr/local/lib/python3.6/dist-packages/setuptools/command/develop.py", line 140, in install_for_development
        self.run_command('build_ext')
      File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/src/ray/python/setup.py", line 74, in run
        subprocess.check_call(command)
      File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['../build.sh', '-p', '/usr/bin/python3']' returned non-zero exit status 1.

    ----------------------------------------
Command "/usr/bin/python3 -c "import setuptools, tokenize;__file__='/src/ray/python/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps" failed with error code 1 in /src/ray/python
You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Describe the problem

In documentation it is written that I can install Ray directly from the repository. This has also worked well before migration to Bazel. But it seems it is not working anymore.

This is important for us so that we can list all our dependencies in requirements.txt and that if we make a fork that we can continue to use Ray without having to publish it to PyPi. So installing directly from the repository using pip is important to support.

[robertnishihara]

Certain things need to be present on your laptop when you try to build Ray from source (e.g., a C++ compiler, git, and bazel) come to mind. Are you saying that you want Ray to install these things itself if they are not present?

[mitar]

Documentation does not mention that Bazel has to be installed, only:

sudo apt-get update
sudo apt-get install -y build-essential curl unzip psmisc

# If you are not using Anaconda, you need the following.
sudo apt-get install python-dev  # For Python 2.
sudo apt-get install python3-dev  # For Python 3.

pip install cython==0.29.0

[mitar]

I am OK if I can do apt-get and pip. But not a custom ray/ci/travis/install-bazel.sh which I would have first to clone the repo to get.

[robertnishihara]

Good point, we should augment the installation instructions to include the relevant commands from https://github.com/ray-project/ray/blob/master/ci/travis/install-bazel.sh. Would you be interested in adding this?

[mitar]

So I do not think this is nice. You are requiring one to install your build system without a package. This makes the whole system dirty.

Can you make it so that build system is installed local to the Python build dir, builds the Python package, installs it, and then build dir gets deleted?

[mitar]

Maybe using custom apt repository would be better: https://docs.bazel.build/versions/master/install-ubuntu.html

[robertnishihara]

@mitar That would be fine, although I don't think it's a big deal because only Ray developers should have to compile Ray. Users should just use pip install.

[mitar]

Users should just use pip install.

Yes, pip install from the branch in the GitHub repo. :-)

So imagine that user reports an issue, you make MR, you ask them to test it, so they want to test that branch. They will do pip install git+https://github.com/ray-project/ray.git@branch#subdirectory=python and things will not work.

[jovany-wang]

I‘d like to add the building requirment: bazel >= 0.2x.x to README.md file or document if some body want to build Ray from source.
And I do agree that we shouldn't recommend the install-bazel.sh script to users since it may make the system dirty. So I'd like to users can use their bazel built by himself to build Ray from source.

[robertnishihara]

@jovany-wang I don't think the main README is the right place for it, but it should be in the installation instructions.

[mitar]

Any idea why any version of bazel is not bundled with Ubuntu? Because if Ray does not require the latest bazel it could be nice if some Ubuntu-provided bazel package could be used. But I have not found any.

[kiddyboots216]

This line on the installation instructions:

pip install git+https://github.com/ray-project/ray.git#subdirectory=python

Is quite misleading. At least, it should be noted that the user must git clone and run the bazel installation.

[richardliaw]

@kiddyboots216 feel free to push a patch!

[kiddyboots216]

@richardliaw I was working on a patch for this for a few hours on Thursday, but I'm not too sure how to fix the underlying issue, unfortunately, and @simon-mo is aware of the underlying issue so I will leave it to him.

[richardliaw]

I think my comment was mainly just to push a docs patch.

On Sat, Jun 22, 2019 at 1:33 PM Ashwinee Panda [e-mail]
wrote:

@richardliaw https://github.com/richardliaw I was working on a patch
for this for a few hours on Thursday, but I'm not too sure how to fix the
underlying issue, unfortunately, and @simon-mo
https://github.com/simon-mo is aware of the underlying issue so I will
leave it to him.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ray-project/ray/issues/4836?email_source=notifications&email_token=ABCRZZNOU6NALSBMQKMY2M3P32EC3A5CNFSM4HOW7GYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYKQ4OI#issuecomment-504696377,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABCRZZOMVBJQ5QJS4JFVF6TP32EC3ANCNFSM4HOW7GYA
.