Yarp: Is sudo really necessary in Travis?

[diegoferigo]

Travis provides two kinds of infrastructure where the linux tests can run: VM-based and Container-based.

The pros of the container based is that CPU are not capped under intense load and the booting time is way faster. The cons is a smaller RAM amount, but for our purpose 4GB in a headless system are more than enough.

In order to enable the Container-based environment, sudo: false is needed, which means that we cannot operate outside the user's home directory.

Having a quick look at our .travis.yml, I spot only two main locations where sudo is used:

https://github.com/robotology/yarp/blob/3a4778976fa6c36292aadf4066bc2fe99f15cd2e/.travis.yml#L252-L256

and

https://github.com/robotology/yarp/blob/3a4778976fa6c36292aadf4066bc2fe99f15cd2e/.travis.yml#L308-L309

I'm not aware of the latter and neither if it can be solved with other workarounds, but for sure the ccache path can be set easily without the need of administration rights.

Since the robotology builds are a bit clogged, would it be a nice idea to give it a try?

[drdanz]

Ok, this is very weird, I swear I noticed this issue right now, but I was investigating exactly this a (see https://github.com/drdanz/yarp/commit/4298c7614c196a803eb4cff292892060c60bf9c6 committed on Mar 26 21:42) in the same exact moment while you were opening this issue (Mar 26 21:24) :rofl::rofl::rofl:

I did not commit it, because I didn't see any improvement, actually the container based one is slightly slower... See for example
a normal yarp sudo enabled build vs. a container based build on my fork (both builds are using the cache from ccache)

I had very good results using gitlab and docker images with everything pre-installed, though see this picture (NOT using ccache here)

Here I'm testing all CMake versions (from 3.5 to 3.11-rc), and then building on different debian/ubuntu releases.

So there is definitely a lot to improve here, I believe that using containers with all the dependencies pre-installed is the way to go, so perhaps use docker in travis but I don't think this can be done without sudo enabled.

[diegoferigo]

Very interesting comparison, this is exactly what I had in mind when I opened this issue, well done @drdanz! The new Gitlab CI/CD looks promising, are you using a yarp mirror or the Github integration?

For what concern your experiments, despite build time was a bit slower (I didn't expect this), the time that takes for a build to start was considerably shorter. This might compensate the difference of build times.

In my experience, I employed docker images for shipping dependencies for two of our projects: human-dynamics-estimation, and gazebo-yarp-plugins. Together with ccache integration (it is doable also within docker containers), the build time had a threefold improvement. I agree that this is the way to go. For curiosity, what images did you use in your tests?

So there is definitely a lot to improve here, I believe that using containers with all the dependencies pre-installed is the way to go, so perhaps use docker in travis but I don't think this can be done without sudo enabled.

Unfortunately no, docker support requires sudo: enabled.

[drdanz]

The new Gitlab CI/CD looks promising, are you using a yarp mirror or the Github integration?

yes exactly...

For curiosity, what images did you use in your tests?

These: https://hub.docker.com/u/drdanz/ :rofl:
The Dockerfiles are not committed anywhere yet, I will commit them somewhere next week...

[diegoferigo]

What about https://hub.docker.com/u/robotology/ on the long run? :smile:

[alecive]

Just my two cents: I have experienced similar results to what you showed here. The docker-ization of travis builds does not necessarily speed up the builds themselves but their start-up time is significantly shorter.

[drdanz]

What about https://hub.docker.com/u/robotology/ on the long run? :smile:

Sure, that's the plan, I'm just using my personal space until it's ready :wink:

[drdanz]

Closing this discussion for now...