Nixpkgs: ROS packaging

[chaoflow]

I'm looking into packaging ros, but before doing so would like some suggestions what you consider the best way to do it.

http://wiki.ros.org/ROS/Installation

Any ideas?

[sephalon]

I need ROS packaged too and would be willing to help. However as I'm quite new to NixOS I'm unable to give any advice on the best procedure :-(

[therealpxc]

I don't know about all of ROS, but I'm currently packaging some other OSRF items for NixOS. I've got working packages for Gazebo, SDFormat, Ignition-Math2, Ignition-Transport 0.9.0 and Ignition-Transport 1.0.1. I've also got an updated Bullet package which is compatible with Gazebo. Right now I'm not sure if I want to say the packages are complete, so they're just sitting around in my nixos-unstable fork.

Shall we start with those? I can make a new branch based on nixpkgs master or nixos master for the PR.

[therealpxc]

I started work today on using Nix to set up the build environment for PX4, as well. Do you guys want to get together online and discuss making a little Nix robotics+UAS maintainers' team?

[therealpxc]

The guy who maintains the Arch packages for ROS in the Arch Users' Repository uses this script to generate the PKGBUILDs from the catkin packages. Maybe we could reuse some of his work to do a similar thing for Nix?

It also seems like getting just catkin packaged would be a good first step, as would just trying to reproduce in NixOS what they do on Debian with their 'Debian Installation from Source' instructions. This would make sure all of the peripheral dependencies were packaged, and get their build system into nixpkgs.

[expipiplus1]

I started adding some python packages in #14798

I've bumped into some issues packaging catkin.

I'm probably going to just install everything in a VM for now, if someone else wants to bring that PR forward I'd be happy to help.

[acowley]

I have ROS installed via Nix with one derivation for a distro variant, and I'm now working on atomizing it so we get derivations for each package. I'll open a PR for nixpkgs when it's working, but I'm happy to hear if anyone has thoughts on organization. My current aim is to have, eg, ros.indigo.perception as a package set.

[jyp]

@acowley : was there any progress? Would you consider sharing your work in its present state so I could build upon it?

[acowley]

@jyp: I'll push things today. I updated to kinetic, and added a mechanism to extend a ROS variant package set (e.g. perception) with custom packages. I don't yet know how best to organize things, so I'll have to write up a quick tour guide, too.

[acowley]

@jyp Here it is

I'd like to roll-up the use of the actual ros2nix executable in the ros.nix file, but haven't done so yet so it is a two-step process. Also, do bear in mind that building everything takes a long time.

Edit: You can actually skip the first step (stack exec) shown in the README as I included the generated file in the repo.

[jyp]

Awesome, thanks. We'll be deploying ROS in the next few weeks, I'll let you
know how it goes.

On Thu, Nov 17, 2016 at 4:49 AM, Anthony Cowley [email protected]
wrote:

@jyp https://github.com/jyp Here it is
https://github.com/acowley/ros2nix

I'd like to roll-up the use of the actual ros2nix executable in the
ros.nix file, but haven't done so yet so it is a two-step process. Also,
do bear in mind that building everything takes a long time.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/11713#issuecomment-261148075,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABsYxxI2PFSmdpum-hxh3eq2cJg0Cboks5q-87OgaJpZM4G089k
.

[akru]

Based on @acowley great job I make ROS Lunar ros_comm package set channel:
https://hydra.aira.life/project/aira/channel/latest
Also x86_64 binary cache available at https://hydra.aira.life

[awesomebytes]

Hey guys, I'm interested in having ROS running (and specially, automatically compiled) by using nix. May I know what's your current status? (I'm new to nix)

[therealpxc]

I'm no longer working at a robotics company, so I haven't touched related packages in a long time. But I would like to at least update Gazebo one last time, and to record my lapsed maintainership by removing myself from that list.

I'm also happy to work with anyone (@acowley ?) who is still working with ROS stuff to get a solution into Nixpkgs.

@akru it looks like you're still building those packages on your Hydra instance. Are you still using them, as well?

[akru]

@therealpxc Yes, currently I have minimalistic port of ROS packages - ros_comm package stack. It builded on my hydra and used in my company for supporting ROS in NixOS. I’ll really happy if anyone helps me to integrate more complex ROS packages into nixpkgs tree.

[nuclearsandwich]

Howdy, I work on release tooling for the ROS project. We've been working on some tooling that might make it easier to support packaging ROS for Nix. superflore is a tool for ingesting ROS distribution data and generating packaging infrastructure for different systems. It's very much a new project and currently supports gentoo and openembedded but if it looks useful we'd gladly accept contributions to add support for Nix/NixOS.

[acowley]

That effort is great to see, @nuclearsandwich! Is there anything yet written about how the pieces fit together for supporting a different packaging system?

Something I'll note about my efforts is that I ran into some pretty unusual corner cases. The approach I was taking was to break everything up into distinct nix derivations, then build a development environment that pulls them all in. As it happens, I ran into problems with my shell environment getting too big due to separate, long paths being piled into PYTHONPATH and ROS_PACKAGE_PATH. I ended up needing to write custom wrappers for clang on macOS to make better use of response files as the command lines were too long, and I had to script some pruning of the environment before even calling exec as it too was groaning under the thousands of paths. Even so, building with catkin is still very slow (much slower than on a normal Linux setup), I think due in large part to these things. Other minor issues arose, too, but I don't think my circumventions were interesting as they were a consequence of the epic search path design problem.

What I'd like to try is more carefully producing a nix store path that collects symlinks to everything in the particular ROS distro to make the paths a bit more humane.

[szobov]

@akru would it be possible to share your ports of ROS packages?

[akru]

@szobov yes, sources here, and channel/cache here.

[nuclearsandwich]

Is there anything yet written about how the pieces fit together for supporting a different packaging system?

Not yet. Currently there's some duplicated effort between bloom and superflore with respect to generating templates (although they target different platforms). I think integrating them better so that template creation and population for all platforms is done via bloom and invoked via superflore when building whole rosdistros at once is probably higher in my priority queue than documenting the current state. But I'm not focusing on either at present as they're in a bit of a steady state for our current needs.

Regarding the path size challenges: I'm not too familiar with how Nix manages things under the hood. ROS tools build their own scripts for managing overlays of workspaces and if my weak understanding of Nix isn't totally incorrect it uses symlinks to unify isolated packages under a virtual prefix? Does your current strategy use one or the other or both of these?

[acowley]

@nuclearsandwich Some languages have tooling in nixpkgs that builds a directory full of symlinks, but a typical Nix thing is to say you want to try out a couple programs without installing them. In that case, you could say, nix-shell -p ruby imagemagick and you'd have a shell where the binaries associated with those packages are on PATH. That's the approach that doesn't scale to rosdistro variants too far beyond ros_comm. ROS somewhat exacerbates the potential problems with the environment variable strategy because you have PATH, ROS_PACKAGE_PATH, and PYTHONPATH off the top of my head.

I don't think this is a fatal issue in general, just an observation I wanted to share because it's not often you have things like exec fail due to environment size! 😛

[nuclearsandwich]

Thanks for clarifying. If I may summarize: nix-shell -p .... 30 or 40 ROS packages is fragile. Does nix have any notion of metapackages or package groups that would enable the use of say nix-shell -p ros-kinetic backed by a single or minimal number of path extensions?

I don't think this is a fatal issue in general, just an observation I wanted to share because it's not often you have things like exec fail due to environment size!

No indeed. It's a neat trick!

[acowley]

The helper one should use is symlinkJoin. It does the symlink work. In addition to that, one may need additional wrapping to inform the gathered packages of their common home. The python tooling does that by creating a wrapper script for each package in any ad hoc package set that sets PYTHONHOME before calling into the wrapped script.

[breakds]

Hi everyone! Not sure if anyone is still working on this.

I am working on a self-driving project and would like to get ROS packaged with nix. I am fairly new to nix packaging, so this is quite ambitious. I see many good progress being made in this thread, and am wondering whether I can join the effort and learn from everyone.

I am targeting ros kinetic. So I have been able to build the python dependencies from http://wiki.ros.org/kinetic/Installation/Source

With that, I started to package each single module from ros. I pick catkin as the first one to attack. @akru 's catkin package is a very good start for me (Thanks!), but I was not able to get it working (tried to patch the cmake files, but still cannot fix it).

In short, I am wondering:

1. Is there anyone who is still working on this? If so, can I join the effort to learn instead of fighting alone?
2. Is my strategy correct (to package each ros-module one by one, fighting with cmake)?
3. I am currently working on it in one of my own repos - should it be eventually moved to nixpkgs? I think as a newbie I might be doing many things wrong as there might be some coding/structure style guide for nix.

Thanks a lot!

[akru]

Hi @breakds! Thank you for your interest and welcome to NixOS ROS community!

I’ve try to describe my approach of ROS development under NixOS in this post: https://github.com/airalab/aira/issues/75#issuecomment-440318394

[breakds]

Thank you for the prompt reply, @akru ! Thanks for sharing your ROS development approach - looks reasonable to me.

Have some more newbie questions if you don't mind:

1. It seems that pkgs in airalab is a fork from the official nixpkgs. What is the best way for me to try the ROS packages there without breaking my nixos setup which is currently based on the official nixpkgs? So far I have been developing .nix files in my own project directory, and run nix-build on them manually - does that sound wrong?

2. I tried copy your catkin.nix as shown in this repo with slight modification. Does this seem sane to you (I think I may be making newbie's mistake here). It seems to me that in this version of catkin the generated scripts from cmake (Catkin's CMakeLists.txt is generating installation scripts to install, by inspecting it), where some of the scripts has various assumptions of existed python packages or shebangs. I followed your example to patch them, but seemingly not enough. Is there any other typical patches that I could use to work this around?

Really appreciate your help!

[akru]

@breakds Currently for building ROS packages I use approach with custom nixpkgs fetch script:https://github.com/airalab/robonomics_comm/blob/master/fetchNixpkgs.nix and use custom nixpkgs for building package https://github.com/airalab/robonomics_comm/blob/master/release.nix.

Probably overlay approach could be better but for me custom nixpkgs enough. Unfortunately overlays doesn’t support nixos services that we also implement in Airalab. I periodically sync it with official nixpkgs.

[breakds]

Thanks for the pointers, @akru ! Also haven't heard of overlays, but I will look it up.

[stale[bot]]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[lopsided98]

I have an overlay that provides most ROS packages: https://github.com/lopsided98/nix-ros-overlay. I have been using it for classes and research for over a year now. It is semi-automatically updated and a binary cache is available on Cachix.

[Renha]

Cool! I'm still interested in more or less complete packaging

[dawagner]

So far, I'm happy with @lopsided98 's overlay :)