Repo2docker: Request for sudo in the postBuild

[markwilkinson]

Hi guys!
There are things I need to do post-Build, such as installing new apt repos, R packages and Ruby gems. My post-build contains, for example:

add-apt-repository ppa:chris-lea/zeromq
apt-get update
apt-get install -y libzmq3-dev libzmq3
echo 'install.packages(c("devtools", "dplyr","ggplot","gapminder"),repos="http://cran.r-project.org" )' | R --no-save
echo 'devtools::install_github('IRkernel/IRkernel')' | R --no-save
echo 'IRkernel::installspec()' | R --no-save
gem update --no-document --system && gem install --no-document iruby rbczmq pry bio xml-simple gene_ontology
gem install --no-document rdf -v 2.0.2
gem install --no-document rdf-raptor -v 2.0.0

The errors are permissions errors, such as :

Error: must run as root

Installing packages into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
Warning in install.packages(c("devtools", "dplyr", "ggplot", "gapminder"), :
'lib = "/usr/local/lib/R/site-library"' is not writable
Error in install.packages(c("devtools", "dplyr", "ggplot", "gapminder"), :
unable to install packages
Execution halted

ERROR: While executing gem ... (Gem::FilePermissionError)
You don't have write permissions for the /var/lib/gems/2.3.0 directory.

If I could sudo these commands, I think I'd be racing!

[betatim]

Installing other apt repos could be interesting for #148.

For the R packages, is there something like python virtual environments in R that allow people to install packages in their home directory instead of globally? Would be good to know so we can build on that if it exists.

[brooksambrose]

+1 for this, as it seems natural to run postBuild as root. A --debug Dockerfile for r2d when using postBuild shows the last lines as:

...
# We always want containers to run as non-root
USER ${NB_USER}

RUN ./binder/postBuild

@yuvipanda Is there a reason these lines couldn't be swapped so postBuild runs before the user change? Would it open a security hole?

[yuvipanda]

I'm mostly resistant to doing it for the following reason:

It means users would have to use sudo to a non-root user for 99% of tasks (like downloading files, enabling extensions, moving things around, etc). Otherwise you're going to end up with things accidentally owned by root & not usable by the non-root user we run as. It'll also make all the current postBuild files 'wrong'.

IMO, for the use case of installing software that isn't made available with the current apt.txt, users should use a Dockerfile directly. We should support the 80% use case, since the other 20% is going to be very different for everyone.

For R, there's install.R which triggers setting up a specific, non-root writeable library path users can write to. This is new, and is an example of how we should tackle figuring out which is the 80% case!

For installing language packages, finding a non-root location to set them up is almost always the right thing to do, rather than use root. For PPAs, I think we should recommend users use a Dockerfile for now.

I hope this makes sense?

[choldgraf]

@yuvipanda maybe worth adding this to the list of "when to use a Dockerfile" points somewhere?

[brooksambrose]

@yuvipanda If a Dockerfile is the answer then it may be worth reconsidering the "no you can't" guidance on using r2d to bootstrap a Docker file. The 80/20 rule may happen within a single project, in which case it may make sense to use r2d locally to build a base image within the r2d scope, then use a Dockerfile to build resources that require root on top of that layer. That avoids the risk of rebuilding the 80 incorrectly. Images built locally with r2d don't always work on mybinder.org though, IME.

[betatim]

How about adding/collecting the uses where we have said "for now use a Dockerfile" in the binder-examples repo? Or for higher efficiency: encourage people who have been using their own Dockerfile to add an example.

The problem with Dockerfiles is that they let you do too many things in too many ways for us to efficiently support them. You could have a base image you want to add to to make it work on Binder, you could want to start from something that works on Binder and layer on top, or you could want to do your own thing all the way. These are already three legit use-cases that are all completely different :-/

One way to tame this potential chaos is to have some good examples that are so nice to use that people use those instead of doing other things.

Not sure I understand the 80/20 scope thing. There is nothing stopping you from using your own FROM in a Dockerfile on Binder. So you could create one with repo2docker and then inherit from it. I'd be -1 for supporting multiple different FROMs in repo2docker, too much maintenance effort.

[brooksambrose]

I meant the 80 is everything that can be done as a non sudo user from a postBuild script, versus the 20 that must be done with root. The decision so far is that custom installation or configuration requiring root must be done with a Dockerfile.

My use case for needing root was having to run the normal install instruction inside an x11 virtual frame buffer to install the R package cairoDevice, which otherwise fails when it tries to open a graphic device to test the package install. Fortunately I got around the issue by installing r-cran-cairodevice using apt.txt. So don't have a use case for it anymore.

[yuvipanda]

@brooksambrose great you could find a solution that works! IMO what you did is the 'right' way to do things, and providing root (with sudo or otherwise) in postBuild would lead to a similar sprawl problem as Dockerfiles - except we'll actually have to support it! I'm keenly aware that the team maintaining this is small, so we should keep the amount of advanced use cases we support small too. I'll posit that doing anything with X in a container counts as advanced...

@choldgraf That's a good idea!

[brooksambrose]

@yuvipanda So the story so far seems to be that postBuild should give you all the customization options you need without root. Dockerfiles are all the customization you would need, period. So using Dockerfiles is a strong signal that this user is in the 20% that is beyond the support agreement. So this issue could be closed if that's the decision.

Here's a thought: imagine a composable solution from a single repo, appendDocker, that would marry the convenience of r2d with the flexibility of Dockerfiles. It would be a composable file that simply lets you keep writing in the Dockerfile after the final line of the r2d build packs are finished. Yeah YMMV, but you'd actually probably be in a stronger position than if you tried to write the whole Dockerfile from scratch.

FWIW my experience has been that 99% of my project is achievable through the available build packs, but there's just one last little piece that needs root. Then there's that realization that I might need to start from scratch with a Dockerfile after having made an investment in the build packs which are still super useful even if you do need to go rogue.

The current guidance on Dockerfiles with binder or r2d basically asks people to recreate some of the patterns already established in r2d. Building on top of r2d is the right approach to Dockerfiles imho, rather than writing them from scratch. My current complicated strategy, which isn't very obvious, is to use r2d locally to build and push a base image composed from one repo, then have an additional repo or branch with a Dockerfile using the first image in FROM.

Dockerfiles are already supported but sort of in a noncommittal way. An "appendix to the composed Dockerfile" option might actually make the process of using them less error prone. It would work best if people knew to inspect the base Dockerfile using the r2d debug flag. I've come to regard r2d as a stand alone product anyway, with binder an added benefit, rather than r2d as simply binder under the hood.

Just a thought! Can't thank you enough for all the work the team is doing; it's already changed the way I do both research and teaching.

[betatim]

You are suggesting that appendDocker contains lines to append to the end of the Dockerfile that r2d generates?

I think we would like to maintain the freedom to change the contents and structure of the docker instructions r2d uses to build images "at any moment". I am not sure we can maintain that and have happy users who are appending instructions to them. It just feels like we'd either have to limit the amount of changing we can do or would end up breaking people's appended lines.

In the cases where I need to use r2d's escape hatch most of the work is in understanding the generated instructions from r2d. Given I have to do all that thinking it doesn't feel like a big burden to take the output of --debug and transfer it to a Dockerfile. This also means that if you don't understand the instructions r2d generates you will have a bad time appending lines to them.

For me the spirit of the "no you can't" advice is that if you look at the output of --debug and copy it into a file. If you do that though you are breaking the warranty seal on a radio or CD player. It might be a good idea, but you can't complain if all your favourite tunes sound funny afterwards.

Overall I think the right way forward is to collect uses cases that currently aren't supported "natively" in r2d and once there are a few people who want one particular thing discuss adding it.

---

In that spirit, the OP wanted:

• ruby gems
• R packages
• custom apt repos

Ruby gems aren't something we have in r2d yet, but you can install them as user.

R packages are supported via install.R

Custom apt repos are the only thing we can't do right now. So far there have not been that many people who needed it (less than five).

[jzf2101]

I think we have resolved this issue since we don't need executable anymore right?

[betatim]

Closing this. If we want to discuss a appendToDockerfile build pack lets create a new issue.