Drake: pip install drake

[RussTedrake]

Was titled pip install drake; apt-get install drake; brew install drake. See #12782 for brew install drake and #12783 for apt-get install drake.

[peteflorence]

this would be awesome!

On Wed, Jul 15, 2015 at 2:35 PM, Russ Tedrake [email protected]
wrote:

make debian packages, etc.

—
Reply to this email directly or view it on GitHub
https://github.com/RobotLocomotion/drake/issues/1183.

[jhoare]

A good way to support apt-get for Ubuntu is to run a drake PPA. You could have a stable/nightly/etc versions where users could automatically get updates via apt-get update.

https://help.launchpad.net/Packaging/PPA

[RussTedrake]

while I’m sure there are many examples, I found this page to be very compatible to what I think we should strive for here:
http://dreal.github.io/download/

[david-german-tri]

I think this is blocked on stable Drake releases, which in turn is blocked on #1769.

[jamiesnape]

Note drake is already taken as a name in homebrew/core. I was just trying out building a Homebrew formula. Looks like a few complications relating to pkg-config and the Homebrew environment, nothing that cannot be fixed, though. I can already build an apt package.

[jamiesnape]

I have a Homebrew package that builds, but we have conflicts with the eigen, bullet, fmt, spdlog so it will not link.

[jwnimmer-tri]

Possibly pip install pydrake is also a useful offering.

[jamiesnape]

Indeed. We have a lot of experience with that. (You can pip install cmake itk vtk).

[EricCousineau-TRI]

Regarding pip, per Jamie's comment in #9456 and when talking with him, it'd be nice to keep this on low-priority, esp. with regards to distribution (e.g. making it such that pip install drake does not have any dependencies on system packages).

Ideally, it'd be nice to support one pip environment, and thus wait until we transition to just using Python3.

[RussTedrake]

That would make #9456 all the more important. My goal is simply to make it easy to use/install. apt-get/brew is another good solution, as far as I'm concerned. But even downloading the tarball might be ok if the workflow is not so onerous.

[EricCousineau-TRI]

Merging #7982 here, "Installed locations of vendored dependencies conflict with Homebrew installed versions":

Namely eigen, bullet, fmt, and spdlog. Following Homebrew conventions, we could install to libexec/vendor/{include|lib} or we could even just use system versions.

[RussTedrake]

Can we revisit this now (we're python3 only)? "Why don't we support pip install drake?" (or pydrake) is the single most common question I've been getting lately. (fwiw -- it's pip install torch to get pytorch)

[jamiesnape]

The reason is not to do with Python versions, but rather the heavy dependence on dynamically linking system dependencies. We can do it, but it changes/disrupts the C++ core significantly.

[jamiesnape]

(I can give you brew cask install drake straight away, however, and apt-get install drake is do-able if you install to /opt rather than /usr)

[jamiesnape]

Note, there is https://github.com/pypa/auditwheel, but only handles a subset of the issues (https://www.python.org/dev/peps/pep-0599/ is also instructive).

[EricCousineau-TRI]

I think we could potentially revisit, but I would still like to defer this until after we have the bindings automation (#7889) underway, given that this would require a hefty bit of dependency rewiring.
Would that be OK?

FTR, I'd be down for slotting apt-get and brew for the near-term, given Jamie's eval on it being less overhead.

[RussTedrake]

i think apt-get and brew would be a huge improvement!

[jamiesnape]

You are welcome to try out:

brew tap jamiesnape/drake
brew cask install drake

Needs more testing and some work to make it maintainable.

[jamiesnape]

I took a quick look at the apt packaging and pending on how important this is, it is looking like 3-5 days work for manual packaging of the latest numbered release (subsequent releases would be a few hours). You would get apt-get install python3-drake libdrake1 libdrake-dev libdrake-java at a minimum. Full automation would take longer. Most of the task is actually administrative, rather than technical.

Really, we nee a test suite for the packages that I can run during the builds (something like a script that extracts a few files from drake/examples and writes both a CMakeLists.txt and {BUILD,WORKSPACE}.bazel. drake-external-examples is not sufficient for that purpose (and nor was it supposed to be).

[jamiesnape]

Can we split this into either two issues (apt and brew cask one issue) or three? pip is so much more complicated that this issue is going to be two thirds complete for a very long time.

[EricCousineau-TRI]

Can we split this into either two issues (apt and brew cask one issue) or three? [...]

Yes, please!

[jamiesnape]

Done. Let's keep this as pip install drake since I think there is some useful information here.

[RussTedrake]

@jamiesnape -- when we talked about this last week, you further explained your concern about the linking issues (and how it would disrupt the c++ core). Basically, you're worried that pip means you have to provide the same experience for all users on all platforms, and the "disrupt the c++ core" was because you thought it implies that we have to support redhat, etc in the main repo.

i don't think we should do that. but my ask for you was whether we can support only a subset of platforms with pip. E.g. pip users either get pydrake, or they get an unsupported platform if they are not on ubuntu or mac. where you able to look into that?

And btw -- i'm ok if we support ubuntu only in the first pass.

[EricCousineau-TRI]

Huh - I thought the "disrupt c++ core" would've implied moreso that you have to do what TensorFlow / PyTorch do - bake all non-Python dependencies into this binary, and have next to zero reliance on the system's package manager; not so much the platform support side of things.

Had y'all already discussed that?

[jwnimmer-tri]

Any technical exposition, details, or work planning debates from realtime meetings should be transcribed into the issues database (or incorporated via reference to a separate document). It sounds like that has not been happening for the Russ <=> Jamie meetings.

[jamiesnape]

Huh - I thought the "disrupt c++ core" would've implied moreso that you have to do what TensorFlow / PyTorch do - bake all > non-Python dependencies into this binary, and have next to zero reliance on the system's package manager; not so much the platform support side of things.

Had y'all already discussed that?

It is both, to be honest, though there are a few more ways to solve the issue if you know what platform you are on.

Any technical exposition, details, or work planning debates from realtime meetings should be transcribed into the issues database (or incorporated via reference to a separate document). It sounds like that has not been happening for the Russ <=> Jamie meetings.

I do not think I have said anything that has not been written elsewhere (and the meetings were minuted in the usual location in any case).

[jamiesnape]

FWIW This is how platform compatibility tags work: https://packaging.python.org/specifications/platform-compatibility-tags/. As I have said, you are limited to flavors of manylinux. manylinux2014 was not approved until 2019, so any approval of a more up-to-date manylinux is looking to be years away, if ever (the 2014 process caused enough issues that things will probably change). Since drake requires C++17, it would need to fall back to C++17 polyfills at the very least.

[ToffeeAlbina-TRI]

Jamie, Jeremy, and Eric still believe pip is much more complicated, risky and expensive than apt. It’s not worth pursuing for the September timeframe - too risky.

[jwnimmer-tri]

I'm moving this into "On Deck" so that we can start planning out the work.

[ludwigschmidt]

Yay! Looking forward to pip install drake!