Nixpkgs: Running NixOS inside Docker

[domenkozar]

Follow up from #1088. I'd like to run NixOS inside a Docker container. The motivation is to provision Docker images with NixOS as devops are used to Docker API.

Solutions:

• https://github.com/hercules-ci/arion

[offlinehacker]

Create nixos rootfs using https://nixos.org/wiki/NixOS_and_libvirt and then
run container in "--privileged" mode, you have development environment you
can distribute. You could of course just use lxc or libvirt-lxc, and have
better experience, because docker was not made for system virtualization,
just sayin.

On Mon, Jun 9, 2014 at 8:53 PM, Domen Kožar [email protected]
wrote:

Follow up from #1088 https://github.com/NixOS/nixpkgs/issues/1088. I'd
like to run NixOS inside a Docker container. The motivation is to provision
Docker images with NixOS as devops are used to Docker API.

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/issues/2878.

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.11 (GNU/Linux)

mQENBFEY1PEBCADPOfERF2wo4qeoq9L1m2z4pKfWqNd4B6BsoFUWPNd7BXmY+9JG
jJddSkmYobWec7XjAFTBL0Xbttt+rK9SIED2dCOmU1FYMQElhGlM3PNA3kaiQFeV
ijgH318GCfZzDd0dWa5TN/IshVeWXwgngsIEmZTVf1VSeb3eO3B8Fxe3zsSLUq0b
71MmU5eLVP9pMkm5V5BTYp+lV70FIekKygkKq+uTDo1csWUatbs4Qvgv37Bymy2t
oTwOBXGoinQk5N/6asR1jWs3vKv0L0SruoZy/kEG/jXb4l2OZUP85EVMganYKouE
OchVmcmhBdWV+t3HK4r2ATfyEcMRzvzSflA1ABEBAAG0Jkpha2EgSHVkb2tsaW4g
PGpha2FodWRva2xpbkBnbWFpbC5jb20+iQE+BBMBAgAoBQJRGNTxAhsDBQkB4TOA
BgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRD6Zxi5hZclKnXNCACLOKa8abQp
eTWv9SXUwC7LVM5pP2mXcgn+Ipqr6YWBdLx4Iij0YlvUok9VeKvwTpUlT+cx++o3
wCM3AYrUyJE+zrtw49lInUmutz9seqJLU895oq+D+UuGoORrLBpEZrYR5f83uUmQ
E3Z1ZmWrNGXYtITWDtVZD/KauMF2nkPcmy/XaYXhd4WHD81DGNlKtGAHig6A3Phc
8Mr0A4yLDeRQJm8lCFEsxMJUNTupgY+ybbsMfVGx1gQvvGOTioV8CLCoRchUCCcm
YPArFg40KzIDSNjwdo9EVZDnlPx1hbOppfQydxP+JVnZsqoYmVY4UhIWi/NfOl3V
UMjl338INW1zuQENBFEY1PEBCADRSIfelOMjaTH7IfpMFFUc5Gys//njFnW9QAUg
wyfs2AFxUp6vKQ7nxXQiJXVhKTwe9iqo+oGxaHp4AeTjC7vXsfMuF5g5lfttbAo3
YEobEe6OG5so41nbwan6SyeIIQ2AmQqJBw8TKKMSec2qUN0Pw7iZRs0o9uJM/obG
DPsAsMOQgNLxJyMCP7X2jBtDXxkMFVHMmk50Tl3h3Fi9qWuNxgTXjs0tUvKkXiu2
Pco952jnm7HpCIKBek2pqR/UJXXb5qxy5G6Lc0qaMWZ5GKnSMTJmTY6Xl44EnaLK
zh0rqgF9qpoWck470ZbiGASMtB008hy2l0cyxUfvDaS3tY4hABEBAAGJASUEGAEC
AA8FAlEY1PECGwwFCQHhM4AACgkQ+mcYuYWXJSoT6AgAkvzvC0EGmeCR3cn9O3Gf
yG00Kqk9/1gJvlphis7AAce8iUgU+4xd94Vp0u8rghpdy88xKN5lF1W2YZQmmBaf
AVe6b7TOg6kxc3GKkVsWDxNyQKkpB46BwefIGaSljH7502X9aEWosrqWyJJNYCtt
QDit4BysX0Ww3Ka5Rx6ZFhC9ybPKoW2i8JwpyBaXDt7R2k+PC/ClBf9qzL+sb2es
zh/zCMVKNdm8KUITHU/5lgn2qZpUFZwiASPCMGGFP9u8g6UKeUTYTPD+GWaHIW63
RAgNIAffxx0M1r3P/2ipkAdI3NX/1iBKDQNG8Odsf+BswFKrNCnyUDdLPvJAhODS
gw==
=tmrm
-----END PGP PUBLIC KEY BLOCK-----

[chexxor]

Thanks for making this, @iElectric. I think Docker is very different from other container projects, such as libvirt, because Docker is super-easy to get started with. It has images in a central repo for easy download, like VagrantCloud, and it's got projects, both big and small, which provide nice GUIs.

[offlinehacker]

So patch for running systemd has been applied to docker master(libcontainer), and you can test it by updating and patching docker https://github.com/NixOS/nixpkgs/pull/3015. Now when it looks more promising, i will try to get nixos running inside docker.

[wmertens]

This is awesome! I suppose it still needs a privileged container? Where is this patch discussed?

[offlinehacker]

Okay to make this happen these 3 commits should be merged: #3106 #3105 #3104 With these 3 commits systemd gets started in container with required services, except shell does not work(and will be probably impossible to fix), but you can ssh. Then i will write build script for docker base image.

[wmertens]

Awesome! They look innocent enough. Why doesn't shell work? Not that it's
needed I think...

On Fri, Jun 27, 2014 at 11:28 AM, Jaka Hudoklin [email protected]
wrote:

Okay to make this happen these 3 commits should be merged: #3106
https://github.com/NixOS/nixpkgs/pull/3106 #3105
https://github.com/NixOS/nixpkgs/pull/3105 #3104
https://github.com/NixOS/nixpkgs/pull/3104 With these 3 commits systemd
gets started in container with required services, except shell does not
work(and will be probably impossible to fix), but you can ssh. Then i will
write build script for docker base image.

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/issues/2878#issuecomment-47324497.

[offlinehacker]

Well getty does not seem to work, you can spawn init script in the
backgrount( with & at the end of init script) and shell after init, but
that's not real login shell, but it might be enough for docker. It looks
like other distros have similar problems. If you figure out how to make
getty run in libcontainer, please let me know :)

On Fri, Jun 27, 2014 at 12:17 PM, wmertens [email protected] wrote:

Awesome! They look innocent enough. Why doesn't shell work? Not that it's
needed I think...

On Fri, Jun 27, 2014 at 11:28 AM, Jaka Hudoklin [email protected]

wrote:

Okay to make this happen these 3 commits should be merged: #3106
https://github.com/NixOS/nixpkgs/pull/3106 #3105
https://github.com/NixOS/nixpkgs/pull/3105 #3104
https://github.com/NixOS/nixpkgs/pull/3104 With these 3 commits
systemd
gets started in container with required services, except shell does not
work(and will be probably impossible to fix), but you can ssh. Then i
will
write build script for docker base image.

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/issues/2878#issuecomment-47324497.

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/issues/2878#issuecomment-47328475.

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.11 (GNU/Linux)

mQENBFEY1PEBCADPOfERF2wo4qeoq9L1m2z4pKfWqNd4B6BsoFUWPNd7BXmY+9JG
jJddSkmYobWec7XjAFTBL0Xbttt+rK9SIED2dCOmU1FYMQElhGlM3PNA3kaiQFeV
ijgH318GCfZzDd0dWa5TN/IshVeWXwgngsIEmZTVf1VSeb3eO3B8Fxe3zsSLUq0b
71MmU5eLVP9pMkm5V5BTYp+lV70FIekKygkKq+uTDo1csWUatbs4Qvgv37Bymy2t
oTwOBXGoinQk5N/6asR1jWs3vKv0L0SruoZy/kEG/jXb4l2OZUP85EVMganYKouE
OchVmcmhBdWV+t3HK4r2ATfyEcMRzvzSflA1ABEBAAG0Jkpha2EgSHVkb2tsaW4g
PGpha2FodWRva2xpbkBnbWFpbC5jb20+iQE+BBMBAgAoBQJRGNTxAhsDBQkB4TOA
BgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRD6Zxi5hZclKnXNCACLOKa8abQp
eTWv9SXUwC7LVM5pP2mXcgn+Ipqr6YWBdLx4Iij0YlvUok9VeKvwTpUlT+cx++o3
wCM3AYrUyJE+zrtw49lInUmutz9seqJLU895oq+D+UuGoORrLBpEZrYR5f83uUmQ
E3Z1ZmWrNGXYtITWDtVZD/KauMF2nkPcmy/XaYXhd4WHD81DGNlKtGAHig6A3Phc
8Mr0A4yLDeRQJm8lCFEsxMJUNTupgY+ybbsMfVGx1gQvvGOTioV8CLCoRchUCCcm
YPArFg40KzIDSNjwdo9EVZDnlPx1hbOppfQydxP+JVnZsqoYmVY4UhIWi/NfOl3V
UMjl338INW1zuQENBFEY1PEBCADRSIfelOMjaTH7IfpMFFUc5Gys//njFnW9QAUg
wyfs2AFxUp6vKQ7nxXQiJXVhKTwe9iqo+oGxaHp4AeTjC7vXsfMuF5g5lfttbAo3
YEobEe6OG5so41nbwan6SyeIIQ2AmQqJBw8TKKMSec2qUN0Pw7iZRs0o9uJM/obG
DPsAsMOQgNLxJyMCP7X2jBtDXxkMFVHMmk50Tl3h3Fi9qWuNxgTXjs0tUvKkXiu2
Pco952jnm7HpCIKBek2pqR/UJXXb5qxy5G6Lc0qaMWZ5GKnSMTJmTY6Xl44EnaLK
zh0rqgF9qpoWck470ZbiGASMtB008hy2l0cyxUfvDaS3tY4hABEBAAGJASUEGAEC
AA8FAlEY1PECGwwFCQHhM4AACgkQ+mcYuYWXJSoT6AgAkvzvC0EGmeCR3cn9O3Gf
yG00Kqk9/1gJvlphis7AAce8iUgU+4xd94Vp0u8rghpdy88xKN5lF1W2YZQmmBaf
AVe6b7TOg6kxc3GKkVsWDxNyQKkpB46BwefIGaSljH7502X9aEWosrqWyJJNYCtt
QDit4BysX0Ww3Ka5Rx6ZFhC9ybPKoW2i8JwpyBaXDt7R2k+PC/ClBf9qzL+sb2es
zh/zCMVKNdm8KUITHU/5lgn2qZpUFZwiASPCMGGFP9u8g6UKeUTYTPD+GWaHIW63
RAgNIAffxx0M1r3P/2ipkAdI3NX/1iBKDQNG8Odsf+BswFKrNCnyUDdLPvJAhODS
gw==
=tmrm
-----END PGP PUBLIC KEY BLOCK-----

[offlinehacker]

Currently this https://github.com/NixOS/nixpkgs/pull/3105 or https://github.com/dotcloud/docker/issues/6730 issues are blocking nixos running on docker. If one of this is resolved, we are ready :)

[offlinehacker]

This is test case:

{ system ? builtins.currentSystem
, pkgs ? import <nixpkgs> { inherit system; }
}:
  with pkgs;
  with pkgs.lib;

let

  container = (import <nixpkgs/nixos/lib/eval-config.nix> {
    modules = [{
      nixpkgs.system = system;
      boot.isContainer = true;
      networking.firewall.enable = false;
    }];
  }).config.system.build.toplevel;

in pkgs.writeTextFile {
  name = ''docker'';
  text = ''
    FROM busybox
    CMD ["${container}/init"]
  '';
}

To run:

docker build --rm - < result
docker run -v /nix/store:/nix/store --rm d6e5b7dc2858

[offlinehacker]

You can beta test here: https://registry.hub.docker.com/u/offlinehacker/nixos/

[domenkozar]

@offlinehacker this works now right? Closing.

[offlinehacker]

Yeah, but this is not intended usage of docker. Docker is basically process
manager that runs processes in containers, so its purpose is running single
process in containers and not whole os(app deployment vs. Os deployment).
I'm waiting for nixup services, which will (hopefully) solve nix(without
os!) declerative docker processes/containers.
On Oct 23, 2014 12:30 AM, "Domen Kožar" [email protected] wrote:

Closed #2878 https://github.com/NixOS/nixpkgs/issues/2878.

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/issues/2878#event-182433319.

[MasseGuillaume]

@offlinehacker how did you build https://registry.hub.docker.com/u/offlinehacker/nixos/ ? Can you publish an updated version ?

you can also use the following Dockerfile:

FROM offlinehacker/nixos
COPY *.nix /etc/nixos/
CMD "nixos-rebuild switch --upgrade"

and config

{ config, pkgs, ... }:

{
  imports = [ <nixos/modules/virtualisation/docker-image.nix> ];
  environment.systemPackages = with pkgs; [
    tree
  ];
}

to customize the container

[saulshanabrook]

@offlinehacker Is it possible to add an updated version? Or give instructions on how to build our own image?

[573]

@saulshanabrook see #4642 for instructions and followup.

[nh2]

@573 I don't understand what the final outcome is, is there now a docker image that contains a recent version of NixOS?

[jacknlliu]

Is it possible that NixOS running inside docker container? And what is the "closing" meaning? @domenkozar

[bbarker]

I made an attempt at this with all steps used to reproduce, see https://hub.docker.com/r/bbarker/nixos/ and associated urls (including git repo). Some work is needed to get systemd fully working, but it is partly working. Need the help of some NixOS/systemd experts! Maybe open a new issue?

[domenkozar]

There is no official image to run NixOS inside docker, but some people have reported to have success doing so. I'll reopen to track any effort.

[573]

@nh2 as far as I know there is none.

[573]

Note: Not exactly for people interested in running NixOS (systemd etc. etc.) in a docker container but as a reference for using nix-related technology from inside a docker container, sorry for cross-posting.

I slightly reenacted my efforts and found @LnL7
The nix-docker explains enough to get you started. Just sayin' you can do lots of nice things already (tested on a Windows 7 machine as docker host - Docker Toolbox):

• I got an instant nix-shell with dependencies for a certain ghc version of the haskell time package - i. e. for interactive development docker@default:~$ docker run --rm -it lnl7/nix nix-shell -p "haskell.packages.gh c821.ghcWithPackages (pkgs: [pkgs.time])" (see i. e. this article and especially the nixpkgs manual as well
• I got a nix-repl for testing out nix expressions as the readme promised (yes it is <nixpkgs> literally)
• the possibilities are probably endless depending on what you wanted to achieve there are lots of articles and different approaches i. e. just building some piece of software in a docker container using nix and run it etc.

[srghma]

@offlinehacker would like to use custom /etc/nixos/configuration.nix in docker, but offlinehacker/nixos is not working for me

 ✘  ~/project   nix ●✚  docker pull offlinehacker/nixos                                                              Using default tag: latest
latest: Pulling from offlinehacker/nixos
118f3436959c: Already exists
Digest: sha256:0a2326bcc26551cd0f1380217a3715011998a8d7559172ee8e210feba43bc4e8
Status: Image is up to date for offlinehacker/nixos:latest
 ✘  ~/project   nix ●✚  docker run -t -i --cap-add=SYS_ADMIN --cap-add=SYS_RESOURCE -e "container=docker" offlinehacker/nixos /bin/init
/nix/store/1vahrabkmxxhh1pq8l6m1ilvs33zp29b-docker-17.09.1-ce/libexec/docker/docker: Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused "exec: \"/bin/init\": stat /bin/init: no such file or directory".

P.S. where is the source code of this image, would like to know how this was created

[FRidh]

Just to show the current status:

{ system ? builtins.currentSystem
, pkgs ? import <nixpkgs> { inherit system; }
}:
  with pkgs;
  with pkgs.lib;

let

  container = (import <nixpkgs/nixos/lib/eval-config.nix> {
    modules = [{
      nixpkgs.system = system;
      boot.isContainer = true;
      networking.firewall.enable = false;
    }];
  }).config.system.build.toplevel;

in dockerTools.buildImage {
  name = "nixos";
  config = {
    Cmd = [ "${container}/init" ];
  };
}

Build it, load it, then run:

$ docker run nixos:xj0lfkd8hqkwgxlmgp9nifjja13chf7a

<<< NixOS Stage 2 >>>

mount: /: permission denied.
mount: /nix/store: permission denied.
mount: /nix/store: permission denied.
running activation script...
setting up /etc...
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 120.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 120.
hostname: you don't have permission to set the host name
Activation script snippet 'hostname' failed (1)
mount: /dev: permission denied.
mount: /dev/pts: permission denied.
mount: /dev/shm: permission denied.
mount: /proc: permission denied.
mount: /run: permission denied.
mount: /run/keys: permission denied.
mount: /run/wrappers: permission denied.
Activation script snippet 'specialfs' failed (32)
starting systemd...

[srghma]

@Fridh maybe this example from arion is helpful

https://github.com/nix-community/todomvc-nix/blob/254dbc72895d1d45f91504e7e78be1e4f08391d6/deploy/arion/service-nginx.nix#L32

Though not sure how to make all systemd units to run

[domenkozar]

We are using https://github.com/hercules-ci/arion for local development and reusing NixOS services. It's not trivial to wire them up, but it has gotten better in last few months :)

[domenkozar]

Robert has built Arion, docker-compose integration with Nix that can run in containers:

• plain commands using Nix/Nixpkgs
• single systemd units from NixOS
• full NixOS
• docker hub image

I've refreshed README today, would love some feedback

[eyJhb]

Doesn't this fix, what the issue https://github.com/NixOS/docker ?

[offlinehacker]

Podman natively supports systemd in a secure manner and does not require a running daemon, and whole lot of unecessary docker stuff, justsaying

[eyJhb]

Podman natively supports systems in a secure manner and does not require a running daemon, and whole lot of unecessary docker stuff, justsaying

Looks nice! But was more of, does this issue still make sense, or is it solved :D

[offlinehacker]

Idk it seems that people do it in their own ways and it seems solved to me, at least in a sense what you can force docker and nixos to do. I decided to try to find better alternatives to docker, from podman, kata containers, firecracker, cri-o,... and better alternatives to abstract services.

[eyJhb]

Considering that the issue mentioned (#1088) mentions, that NixOS should support Docker, the link I have given provides a answer that we do, plus we have this in unstable ( https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/docker-containers.nix ), that allows us to declare containers inside our configuration file. :)

If DevOps need to support anything, then the issue should maybe be opened in DevOps repo :)

[wmertens]

@eyJhb the link you gave makes a docker image with Nix, not NixOS. There's no systemd running, no way to declare which services run etc.

[flokli]

Running NixOS inside docker requires running systemd, which doesn't work in there. There's multiple solutions available these days to

• build Docker images containing single applications
• build declarative/imperative docker containers and run them from NixOS
• build NixOS to run inside an nspawn container

Other proposed things above (firecracker and whatnot) probably should go into a separate issue, but I don't see anything actionable here.

I propose closing this issue.

[offlinehacker]

systemd runs inside docker, you just need to mount cgroups. It even runs with user namespaces, using cgroup hierarchy v2, unfortunately currently only using podman.

[flokli]

Well, in that case, there should be an issue about building an OCI image containing a whole NixOS system, and some documentation/tooling (?) on how to mount the cgroups.

[nh2]

in that case, there should be an issue about building an OCI image containing a whole NixOS system

I think that issue ... is this issue :grin:

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[nh2]

@volth Why was this closed? From what i understood from the last few posts, it _should_ be possible to make this work.

[domenkozar]

I think we could have a function that generates a docker image with NixOS configured, taking inspirations from arion.

[malte-v]

@offlinehacker Could you tell us how you got NixOS running inside podman? I tried https://github.com/NixOS/nixpkgs/issues/2878#issuecomment-469679468 with podman instead of Docker but it just gives the exact same error messages.

[offlinehacker]

I only managed it on fedora 32, as I already knew you can run systemd podman containers out of the box on it. I need to figure out what nixos has different configured but I remembered something with cgroups and user slices. I tried both cgroups hierarchy v1 and v2, it will not work with v1 hierarchy at all.