I am wondering if it is possible to run LXC containers in my chroot. Has anyone tried this? I tried to create one in ubuntu precise but got an error "wrong fs type on /dev/shm"
I think running a container inside a chroot may not be a supported use case for lxc yet. But for chromeos it's sort of our only option, aside from doing a dev_install and some portage overlay magic?
I'm just kind of rambling here, but I couldn't find any information about LXC on chromeOS and this seemed like a reasonable enough place to find some.
kzahel
All 138 comments
I gave a quick try at LXC, but I didn't manage to get it to work on my Arch laptop, so I got bored of it and gave up (maybe I should try again)... Instead I went for a lower level approach (dirty branch here: https://github.com/drinkcat/chroagh/commits/container.tmp), and I managed to "boot" Arch in a container, directly from Chrome OS. My idea is that it would be neat to start the crouton chroots in their own containers, so we can use systemd services, shut down chroots in a cleaner way (processes belonging to the chroot are clearly separated), control hardware access, etc.
So, in short, kernel support is available, and it's something I'd really like to get working. The only issue I have now is that the kernel for Samsung ARM is still stuck on 3.4, so it lacks some features, like being able to rejoin an existing container (important for crouton), or create a new instance of /dev/pts (not so critical for crouton, more so if you care about container isolation...).
Now, back to LXC, I see 2 options:
- Compile it statically in a chroot (no idea how difficult that may be), and move it to
/usr/local/binin Chrome OS. Chances are it may work... - Find out what prevents it to work from inside the chroot. It looks like it's complaining about
/dev/shmbeing a bind mount, you could try to remount/dev/shmwith something likemount -t tmpfs tmpfs /dev/shm, and see where it fails next...
drinkcat
on 12 Sep 2013
@Kyle - Hadn't heard of 'LXC Containers' but after Googling it I think this
concept sounds pretty exciting and promising - 'a chroot on steroids' is
one description. I'll play around with it on a desktop Ubuntu install and
see what happens...
@drinkcat - Are 'croagh' 'c_h_roagh' the same thing? I had a '
croagh:031df18f' quantal chroot installed with kde, unity & xfce a while
back but haven't played with it in a while, I'll tinker with it again too.
If they _are_ different, I'll get the 'chroagh' tarball and try it out.
On Thu, Sep 12, 2013 at 12:04 AM, drinkcat [email protected] wrote:
I did a quick try at LXC, but I didn't manage to get it to work on my Arch
laptop, so I got bored of it and gave up (maybe I should try again)...
Instead I went for a lower level approach (dirty branch here:
https://github.com/drinkcat/chroagh/commits/container.tmp), and I managed
to "boot" Arch in a container, directly from Chrome OS. My idea is that it
would be neat to start the crouton chroots in their own containers, so we
can use systemd services, shut down chroots in a cleaner way (processes
belonging to the chroot are clearly separated), control hardware access,
etc.So, in short, kernel support is available, and it's something I'd really
like to get working. The only issue I have now is that the kernel for
Samsung ARM is still stuck on 3.4, so it lacks some features, like being
able to rejoin an existing container (important for crouton), or create a
new instance of /dev/pts (not so critical for crouton, more so if you
care about container isolation...).Now, back to LXC, I see 2 options:
- Compile it statically in a chroot (no idea how difficult that may
be), and move it to /usr/local/bin in Chrome OS. Chances are it may
work...- Find out what prevents it to work from inside the chroot. It looks
like it's complaining about /dev/shm being a bind mount, you could try
to remount /dev/shm with something like mount -t tmpfs tmpfs /dev/shm,
and see where it fails next...—
Reply to this email directly or view it on GitHubhttps://github.com/dnschneid/crouton/issues/364#issuecomment-24293903
.
DennisLfromGA
on 12 Sep 2013
@DennisLfromGA :
croaghwas the name of a branch of crouton, written by @dnschneid, that allows multiple distributions to be supported (currently Debian and Ubuntu). It was merged a few months back now.chroaghis my own repository (https://github.com/drinkcat/chroagh), where I stage branches before pushing them here. Themasterbranch is a fork of crouton, with Arch support, and a few other branches that are not pushed to crouton yet. It should be useable, but is a bit outdated currently as I haven't backported the latest modifications in crouton (I'm waiting for my audio branch to get in before I rebase again).
drinkcat
on 13 Sep 2013
@drinkcat, Thanx for clearing that up, I guess I didn't understand exactly
what it was and what David had done with it.
I had downloaded your 'chroagh' repo (7a16b4b) but hadn't installed it and
played with it yet. From reading your comment above '"boot" Arch in a
container, directly from Chrome OS', it sounds a little different from a
'crouton' chroot, can you explain it a little more for me please? I'll wait
until you've rebased again and then give it a whirl.
P.S. I'm due to get my 128GB SSD for my Acer C7 today so I'm excited!
On Fri, Sep 13, 2013 at 10:44 AM, drinkcat [email protected] wrote:
@DennisLfromGA https://github.com/DennisLfromGA :
- croagh was the name of crouton's branch, written by @dnschneidhttps://github.com/dnschneid,
that allows multiple distributions to be supported (currently Debian and
Ubuntu). It was merged a few months back now.- chroagh is my own repository (https://github.com/drinkcat/chroagh),
where I stage branches before pushing them here. The master branch is
a fork of crouton, with Arch support, and a few other branchs that are not
pushed to crouton yet. It should be useable, but is a bit outdated
currently as I haven't backported the latest modifications in crouton (I'm
waiting for my audio branch to get in before I rebase again).—
Reply to this email directly or view it on GitHubhttps://github.com/dnschneid/crouton/issues/364#issuecomment-24399405
.
DennisLfromGA
on 13 Sep 2013
@DennisLfromGA. First off, the container thing will not be in my next rebase, there is still quite a bit of work to be done, and I need to wait for kernel 3.8 to be pushed on the ARM Chromebook to get some important features. On the other hand, you are still welcome to test ArchLinux ,-)
About containers vs chroot:
A normal chroot just changes the root filesystem of the current process. This is by no means secure (there are easy ways to escape), and does not really isolate processes you run in the chroot from the outside.
Namespaces are far more powerful. It's basically a set of kernel features that allows you to separate a set of process from another, very similar to a virtual machine, but without the overhead of that. You can choose at what level you want to isolate the processes: mount, PID, network, user, etc. (see this article for details: https://lwn.net/Articles/531114/). LXC, systemd-nspawn are tools using those features.
My main reason for trying this out is issue https://github.com/drinkcat/chroagh/issues/8 by @kdb424. systemd, the init system shipped with Archlinux, is not able to start services, like ssh, in a chroot (unlike upstart used by Ubuntu).
By using a PID namespace, you can start a "proper" init process, with PID 1, and you "boot" the system: you see init starting services, and then you get a login prompt: just like when you boot a regular Linux machine. When you are done with the container, you type "halt", and it shuts down, terminating all the processes in the container. So neat and satisfying ,-)
Using a mount namespace is also useful: no more polluting of the output of mount on Chrome OS with lots of bind mounts. And all the mounts are automatically removed when the container shuts down.
Finally, user namespace allows a normal user (chronos on Chrome OS) to be root inside the chroot. I haven't tested that yet (it needs kernel 3.8), but I think that would mean we would not need to add sudo in front of most crouton commands, and would make sure the the chroot could not change kernel settings (stuff in /proc/sys or /sys). This might not be wanted in all cases, though.
UTS namespace would allow a different hostname in chroot. And network namespace could isolate the chroot from the network. Not quite sure why we would want either of that, but we could do it ,-)
In short, quite a bit of potential I think. Still a lot of work to be done to see exactly what it can bring to crouton, and where are the limitations when using some of the namespaces.
drinkcat
on 14 Sep 2013
Thanx @drinkcat for the explanation, I'll read through the article you
mentioned and learned a little more about it. Hopefully, the ARM is not too
far away from 3.8, it looks like we non-ARM users are on 3.8.11. The
namespaces and containers do look promising for crouton though, I tried to
change the hostname in a chroot on crouton and it was a disaster.
On Sat, Sep 14, 2013 at 12:12 AM, drinkcat [email protected] wrote:
@DennisLfromGA https://github.com/DennisLfromGA. First off, the
container thing will not be in my next rebase, there is still quite a bit
of work to be done, and I need to wait for kernel 3.8 to be pushed on the
ARM Chromebook to get some important features. On the other hand, you are
still welcome to test ArchLinux ,-)About containers vs chroot:
A normal chroot just changes the root filesystem of the current process.
This is by no means secure (there are easy ways to escape), and does not
really isolate processes you run in the chroot from the outside.Namespaces are far more powerful. It's basically a set of kernel features
that allows you to separate a set of process from another, very similar to
a virtual machine, but without the overhead of that. You can choose at what
level you want to isolate the processes: mount, PID, network, user, etc.
(see this article for details: https://lwn.net/Articles/531114/). LXC,
systemd-nspawn are tools using those features.My main reason for trying this out is issue drinkcat#8https://github.com/drinkcat/chroagh/issues/8by
@kdb424 https://github.com/kdb424. systemd, the init system shipped
with Archlinux, is not able to start services, like ssh, in a chroot
(unlike upstart used by Ubuntu).By using a PID namespace, you can start a "proper" init process, with PID
1, and you "boot" the system: you see init starting services, and then you
get a login prompt: just like when you boot a regular Linux machine. When
you are done with the container, you type "halt", and it shuts down,
terminating all the processes in the container. So neat and satisfying ,-)Using a mount namespace is also useful: no more polluting of the output of
mount on Chrome OS with lots of bind mounts. And all the mounts are
automatically removed when the container shuts down.Finally, user namespace allows a normal user (chronos on Chrome OS) to be
root inside the chroot. I haven't tested that yet (it needs kernel 3.8),
but I think that would mean we would not need to add sudo in front of
most crouton commands, and would make sure the the chroot could not change
kernel settings (stuff in /proc/sys or /sys). This might not be wanted in
all cases, though.UTS namespace would allow a different hostname in chroot. And network
namespace could isolate the chroot from the network. Not quite sure why we
would want either of that, but we could do it ,-)In short, quite a bit of potential I think. Still a lot of work to be done
to see exactly what it can bring to crouton, and where are the limitations
when using some of the namespaces.—
Reply to this email directly or view it on GitHubhttps://github.com/dnschneid/crouton/issues/364#issuecomment-24436430
.
DennisLfromGA
on 14 Sep 2013
Have you considered something like docker? It wraps the lxc subsystem in a pretty nice way.
alexanderkyte
on 16 Sep 2013
I'm trying to get docker going on my Google I/O Stumpy's.
Docker wants a 3.8 kernel. Looks like Pixel's and Samsung 550's run a 3.8 kernel as of 7/22/2013, but not Stumpy yet.
- Anyone on a Pixel or Sammy 550 know if docker with crouton/Ubuntu "just works"? I'm very new to playing with Docker and LXC so I could be naive here...
- Anyone know if Chrome OS Beta or Dev is running a 3.8 kernel on Stumpy? Or does anyone know when 3.8 kernel will come to hardware (if it will) other than Pixel's and 550s?
Thanks...
Later: Answered #2 myself, dev channel is using 3.8.11. Will report back when I've answered #1...
TreverN
on 22 Sep 2013
The Samsung arm chromebook still appears to be on 3.4.
On Sun, Sep 22, 2013 at 12:50 AM, TreverN [email protected] wrote:
I'm trying to get docker going on my Google I/O Stumpy's.
Docker wants a 3.8 kernel. Looks like Pixel's and Samsung 550's run a 3.8
kernel as of 7/22/2013, but not Stumpy yet.1.
Anyone on a Pixel or Sammy 550 know if docker with crouton/Ubuntu
"just works"? I'm very new to playing with Docker and LXC so I could be
naive here...
2.Anyone know if Chrome OS Beta or Dev is running a 3.8 kernel on
Stumpy? Or does anyone know when 3.8 kernel will come to hardware (if it
will) other than Pixel's and 550s?Thanks...
—
Reply to this email directly or view it on GitHubhttps://github.com/dnschneid/crouton/issues/364#issuecomment-24876085
.
alexanderkyte
on 23 Sep 2013
Eventually all devices will be on 3.8, but it will be a while.
dnschneid
on 24 Sep 2013
Is there an easy way to test out 3.8 on one of the older devices?
On Sep 23, 2013 8:46 PM, "David Schneider" [email protected] wrote:
Eventually all devices will be on 3.8, but it will be a while.
—
Reply to this email directly or view it on GitHubhttps://github.com/dnschneid/crouton/issues/364#issuecomment-24967388
.
alexanderkyte
on 24 Sep 2013
Switching to dev channel is the easiest, assuming it has 3.8 enabled for dev channel. Otherwise you'd have to compile chromium yourself.
dnschneid
on 25 Sep 2013
Docker no longer requires a kernel to have aufs compiled into it.
http://blog.docker.io/2013/11/docker-0-7-docker-now-runs-on-any-linux-distribution/
alexanderkyte
on 26 Nov 2013
It seems that most chromebooks are using kernel >= 3.8. Is there any plan to utilize PID/user/mount namespaces?
BTW, I think the stock wrapper, /sbin/minijail0 is very capable. It seems to support mount(including bind)/network/pid namespaces. But I'm not shure if it can be used by non-root user.
ccaapton
on 30 Dec 2013
Hmm. I have been checking daily for weeks for any updates pushed down the pipe but my Samsung Snowy (ARM v7) is still at 3.4.0. I'm hesitant to go upstream with associated risk.
jpw1116
on 30 Dec 2013
Yes, stable on the ARM is still 3.4. I've been battling the dev channel and crouton for awhile, so yesterday I power washed, and reverted to stable, so far crouton, with chromium, and xfce targets on default precise release are working very well.
tedm
on 31 Dec 2013
It looks like ARM may get 3.8 soon, I see kernel 3.8.11 being built in the Chromium OS builders:
http://build.chromium.org/p/chromiumos/builders/daisy%20full/builds/6104/steps/BuildPackages/logs/stdio
So it looks like it's just a matter of time until 3.8 appears on the dev channel (if it's not already there).
drinkcat
on 1 Jan 2014
Yes, I think you're right @drinkcat - I had to powerwash and revert to stable, as even the most basic crouton functions weren't working on the dev channel. After reverting to stable, crouton (xfce, chromium) on precise works well. I expect that the dev channel will be upgraded soon, as there were a multitude of other issues, such as extension flags not being preserved, and random crashes, (more than usual) ... I wonder what the beta channel is like ? I'm surprised at the lack of testing on ARM systems, as it's by far the most popular platform for the Chromebooks, from the data I"ve seen.
tedm
on 1 Jan 2014
Dev channel _is_ testing.
dnschneid
on 1 Jan 2014
right. But basic functionality, IMHO, such as extensions and flags still working, ctrl-keys in crosh behaving, not crashing frequently, should generally be supported in a dev channel. Again, IMHO.
tedm
on 1 Jan 2014
@dnschneid and all: I apologize for complaining about my issues with the Dev channel. I should be grateful that for almost a year, I was able to function ok with Crouton while running in it, and shouldn't have taken that for granted.
Thank you for retaining Crouton support for those of us on the Samsung ARM, whose stable channel is kernel 3.4.0 and may be so for a while.
Crouton has probably been the most useful software ever for Chromebook users. Thanks David, Drinkcat, and all Crouton developers.
tedm
on 3 Jan 2014
@tedm it sounds like beta channel would be ideal for you; it's a pretty happy medium between stability and previewing new features.
dnschneid
on 3 Jan 2014
Yeah, beta would probably be fine to move to for me, and also since USB sticks, and catching up, aren't needed to change channels anymore on the arm, just powerwashes, it's now simpler to change channels.
If the desktop environments, or the clipboard extensions are ARM ready, I'm ready to test those.
tedm
on 3 Jan 2014
On my Acer C720 Chromebook (reports kernel 3.8.11 with uname -r), I have the docker 0.7.5 daemon running in a crouton chroot with Ubuntu raring. I'm using
sudo docker -d -b none
However I can't yet start a container. Current error is
$ sudo docker run -i -t ubuntu /bin/bash
WARNING: IPv4 forwarding is disabled.
lxc-start: No such file or directory - failed to create '/sys/fs/cgroup/cpu/lxc' directory
lxc-start: failed to spawn '6e7d451def231c2eb1f6ec84fbdac605f69d9a799d196135fb82f49ee187997c'
[error] commands.go:2458 Error getting size: bad file descriptor
air
on 14 Jan 2014
The issue appears to be that ChromeOS can see the mounts under /sys/fs/cgroup, but the chroot OS cannot. In ChromeOS:
chronos@localhost / $ mount|grep cgroup
none on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
chronos@localhost / $ ls -l /sys/fs/cgroup
total 0
drwxr-xr-x 4 root root 0 Jan 14 14:29 cpu
drwxr-xr-x 2 root root 0 Jan 14 14:29 freeze
Whereas in Ubuntu raring chroot:
(raring)air@localhost:~/dev$ mount|grep cgroup
(raring)air@localhost:~/dev$ ls -l /sys/fs/cgroup
total 0
I see that /usr/local/chroots/raring/proc is empty, so I assume the chroot inherits the mounts from ChromeOS.
@dnschneid, sorry for my ignorance - could take a moment to explain why the cpu and freezer mounts under cgroup don't make it into the chroot? Hopefully this isn't a blocker for the whole Docker approach.
air
on 14 Jan 2014
@air: /sys isn't mounted recursively, nor is it mounted shared such that new mounts will appear inside the chroot.
Try doing something similar to the /media mount in enter-chroot. You'd probably want chroot /sys to be a slave mount of the system /sys, so you'd --make-rshared /sys, then --rbind it in, then --make-rslave the chroot's /sys.
dnschneid
on 16 Jan 2014
Thanks @dnschneid, I patched my enter-chroot to set up a slaved /sys space:
# Bind-mount /sys recursively in order to get working cgroups
echo "Mounting /sys recursively"
mount --make-rshared /sys
mount --rbind /sys "$CHROOT/sys"
mount --make-rslave /sys "$CHROOT/sys"
This works - inside the chroot we can now see /sys/fs/cgroup/cpu and all the contents. We no longer fail on the filesystem error.
This brings us to the next error, a 'device busy' from lxc-start:
(raring)air@localhost:~/dev$ sudo docker run -i -t ubuntu /bin/bash
WARNING: IPv4 forwarding is disabled.
lxc-start: Device or resource busy - failed to mount a new instance of '/dev/pts'
lxc-start: failed to setup the new pts instance
lxc-start: failed to setup the container
lxc-start: invalid sequence number 1. expected 2
lxc-start: failed to spawn '6e282af6df4d902fcf7bf26aed756d354f99a0ddcde718bb3a72593bf80da32b'
[error] commands.go:2458 Error getting size: bad file descriptor
I'd like to run lxc-checkconfig but this currently returns
Kernel configuration not found at /proc/config.gz; searching...
lxc-checkconfig: unable to retrieve kernel configuration
Try recompiling with IKCONFIG_PROC, installing the kernel headers,
or specifying the kernel configuration path with:
CONFIG=<path> lxc-checkconfig
So next step is perhaps to sanity check that LXC is capable of running anything. Any advice appreciated : )
air
on 17 Jan 2014
So lxc-checkconfig isn't going to work because it's looking for kernel headers. For example it attempts this file read (found with strace), /lib/modules/3.8.11/build/.config. If this was ordinary Ubuntu (not in a chroot) we would just apt-get install linux-headers-generic and the 3.8.11 package would get placed in that location.
However with ChromeOS as the host, we are running a _non-Ubuntu_ kernel 3.8.11, from Google. Ubuntu does not have headers for non-Ubuntu kernels, so apt-get is not going to help.
It does appear possible to get ChromeOS kernel headers by building them, but it looks a bit hairy: http://superuser.com/questions/657845/cannot-install-3-4-linux-headers-on-acer-c7-chromebook-running-chrubuntu-12-04
I will give up on lxc-checkconfig for now.
air
on 17 Jan 2014
OK, so I didn't give up quite yet : ) Here's the output of CONFIG=/usr/src/linux-headers-3.8.11/.config lxc-checkconfigon a fresh chroot (with no cgroup binding yet - observe 'required' below):
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: missing
--- Control groups ---
Cgroup: enabled
Cgroup namespace: required
Cgroup device: missing
Cgroup sched: enabled
Cgroup cpu account: missing
Cgroup memory controller: missing
Cgroup cpuset: missing
--- Misc ---
Veth pair device: enabled
Macvlan: missing
Vlan: missing
File capabilities: enabled
To get the magic .config file I
built the kernel headers as described in the crouton wiki. Note that I didn't bother making the linux-image, just the headers. This gets you the file linux-headers-3.8.11_3.8.11-10.00.Custom_amd64.deb which you install with dpkg.
air
on 20 Jan 2014
You don't have to fully build the Linux headers to get the config file to check against. The only step you need is make oldconfig as described in the wiki (only takes a few seconds versus many minutes for the headers). This generates a .config at which you can point lxc-checkconfig.
It looks like the LXC tool is just checking flags in the config. So e.g. Multiple /dev/pts instances: missing is just reporting that in the config, # CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set.
Could someone more expert comment on what this means, maybe @ccaapton or @dnschneid? My interpretation here is that features required for LXC are simply not compiled into the ChromeOS kernel (3.8 branch); so LXC cannot function on Chromebooks. Is that correct?
air
on 20 Jan 2014
Here's the explanation of what CONFIG_DEVPTS_MULTIPLE_INSTANCES is for. It was added specifically for container support.
air
on 21 Jan 2014
@air: Is there a way to get LXC to run without another instance of /dev/pts?
I understand this is desired for additional separation between the host and the container, but it is possible to start containers without it.
drinkcat
on 21 Jan 2014
@air I have a basic chroot script on my blog located here: http://cuttingedgelinux.com/2013/03/10/command-line-interface-chroot-access-another-linux-from-your-current-linux/
In that script it requires a bindmount to /proc /sys /dev and /dev/pts
I would think that if you bindmounted as you did with /sys above for these four directories. Doing /dev before /dev/pts it should work in the most basic manner we could expect.
edge226
on 15 Feb 2014
Hi, guys. I played with chromeos's own minijail, and I'm able to enter my crouton folder using minijail instead of enter-chroot. It also supports new PID namespace, so ps/top will only show stuff inside the chroot. The global mount points are not polluted in chromeos.
Here is the command:
"sudo minijail0 -b /proc,/proc,1 -b /dev,/dev,1 -b /sys,/sys,1 -C /usr/local/chroots/precise/ -u 1000 -g 1000 -p -vr /bin/bash"
So it is very close to LXC, right?
UPDATE: On my current C720 stable channel. PID 1 is still occupied by minijail0, but this update https://chromium.googlesource.com/chromiumos/platform/minijail/+/88f76a7746083aaeff0d714571945306f6f06c2d will allow us to have the child process running as PID 1.
ccaapton
on 16 Feb 2014
@ccaapton: Nice! Yes, that's, at least, a basic Linux container (LXC is just wrapper scripts).
-b /proc,/proc,1 should not be there: /proc must be mounted again within the container, not bind-mounted (since PIDs are different, /proc is different). I suspect minijail is overriding your command line parameter (probably because of -r), ps would not be working fine otherwise...
I'm curious if we can get systemd to "boot" within minijail...
drinkcat
on 17 Feb 2014
@drinkcat I have successfully run bash as PID 1. Here are the steps:
Step 1: In the chromeos shell, run
sudo minijail0 -b /dev,/dev,1 -C /usr/local/chroots/precise -r /bin/bash
Step 2: Now we are in new mount namespace. Compile http://lwn.net/Articles/533492/ source code as "ns_child_exec", and run it:
ns_child_exec -p /bin/bash
Step 3: Now we are in new pid namespace with bash as the PID 1.
echo $$
1 :)
Possible Stage 4: At this stage, we can mount procfs to make it visible for the whole session:
mount -t proc proc /proc
Now you can "ps" to verify it
I can not test if systemd works, as currently I don't have any chroot containing systemd. Could anyone try systemd with this?
ccaapton
on 17 Feb 2014
I will certainly be playing around with this the next few days during my weekend. I really want to get systemd working properly and if this works correctly and is easy to implement I am definitely down for some testing!
edge226
on 17 Feb 2014
But will there be any barriers to sharing X this way?
On Feb 17, 2014 5:24 AM, "ccaapton" [email protected] wrote:
@drinkcat https://github.com/drinkcat I have successfully run bash as
PID 1. Here are the steps:Step 1: In the chromeos shell, run
sudo minijail0 -b /dev,/dev,1 -C /usr/local/chroots/precise -r /bin/bashStep 2: Now we are in new mount namespace. Compile
http://lwn.net/Articles/533492/ source code as "ns_child_exec", and run
it:ns_child_exec -p /bin/bash
Step 3: Now we are in new pid namespace with bash as the PID 1.
echo $$
1 :)
Possible Stage 4: At this stage, we can mount procfs to make it visible
for the whole session:mount -t proc proc /proc
Now you can "ps" to verify it
I can not test if systemd works, as currently I don't have any chroot
containing systemd. Could anyone try systemd with this?
Reply to this email directly or view it on GitHubhttps://github.com/dnschneid/crouton/issues/364#issuecomment-35244514
.
alexanderkyte
on 17 Feb 2014
@alexanderkyte As long as we don't create a separate IPC namespace, and bind mount /tmp, we should be fine.
drinkcat
on 17 Feb 2014
Hi, folks! I've already got LXC running upstart as PID 1, and booting the crouton directory. Here are my steps:
- Before entering crouton, bind mount /usr/local
sudo minijail0 -b /dev,/dev,1 -b /usr/local,/usr/local,1 -C /usr/local/chroots/precise -r /bin/bash - Download newest LXC source file from http://linuxcontainers.org/downloads
./autogen.sh && ./configure && make && sudo make install
Now lxc is installed in chromeos, which is outside crouton chroot.- Exit crouton, create a conf in home directory. Say delta.conf:
lxc.utsname = delta lxc.autodev = 1 lxc.rootfs = /usr/local/chroots/precise lxc.arch = amd64 lxc.tty=8 sudo lxc-start -n delta -f ~/delta.conf -dsudo lxc-attach -n delta /bin/bash
Now we are inside the LXC :)
@drinkcat, @dnschneid Can't wait to see this been integrated into crouton!
ccaapton
on 18 Feb 2014
@ccaapton I am a little confused at which layer you are running step 3 on. Is this inside the crouton? If so how does this install lxc onto chrome os?
edge226
on 18 Feb 2014
@edge226 - Step 1. puts you in a precise chroot
It's handy to have previously downloaded the LXC source file and place it in /usr/local/src or similar before you do Step 3.
@ccaapton - Step 3. gives me an error -
./autogen.sh: 6: ./autogen.sh: aclocal: not found
I tried linking aclocal.m4 to aclocal but it still bombs out for me.
DennisLfromGA
on 18 Feb 2014
@ccaapton dont you need to modify something to tell it to compile to /usr/local instead of the standard directory or is /usr/local the standard directory?
edge226
on 18 Feb 2014
@ccaapton - Sorry to have to hold my hand through this but after installing m4, Automake, & Autoconfig, when I run 'Step 3.' the 'make' seems to fail -
(saucy-x64-unity-desktop)root@localhost:/usr/local/src/lxc-0.9.0# ./autogen.sh && make && make install
+ test -d autom4te.cache
+ rm -rf autom4te.cache
+ aclocal -I config
+ autoheader
+ autoconf
+ automake --add-missing --copy
make: *** No targets specified and no makefile found. Stop.
When I ran it again after the 'autogen.sh' step, it looks like it can't find a 'make' file -
(saucy-x64-unity-desktop)root@localhost:/usr/local/src/lxc-0.9.0# make
make: *** No targets specified and no makefile found. Stop.
Also, 'make install' fails -
(saucy-x64-unity-desktop)root@localhost:/usr/local/src/lxc-0.9.0# make install
make: *** No rule to make target `install'. Stop.
@DennisLfromGA Sorry, this is my fault. You need "./configure" after "./autogen.sh". Already updated my previous post.
ccaapton
on 19 Feb 2014
Okay, got it figured out.
Had to run './configure --disable-capabilities' first, then 'make' & 'make install' worked fine...
DennisLfromGA
on 19 Feb 2014
Close but no cigar. After creating the delta.conf file it complains about GLIBC_2.17 -
sudo lxc-start -n delta-saucy-x64 -f ~/delta-saucy-x64.conf -d
lxc-start: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by /usr/local/lib/liblxc.so.0)
sudo lxc-attach -n delta-saucy-x64 /bin/bash
lxc-attach: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by /usr/local/lib/liblxc.so.0)
I checked and I do have the files mentioned -
ll /usr/local/lib/libxc*
lrwxrwxrwx 1 root root 11 Feb 18 18:21 liblxc.so -> liblxc.so.0*
lrwxrwxrwx 1 root root 15 Feb 18 18:21 liblxc.so.0 -> liblxc.so.0.9.0*
But I guess GLIBC_2.17 isn't a part of this 3.4.0 OS. Here's what I'm using -
Installed : Sat Feb 15 19:10:11 EST 2014
Node name : localhost
Machine : x86_64
Rel. name : Chrome OS
OS Track : beta-channel
OS Release : Linux 3.4.0
CROS Root : ROOT-B (/dev/sda5)
OS Board : parrot-signed-mp-v3keys
OS Coder : Googler
OS Vers. : 5116.80.0
Browser Ver: Google Chrome 33.0.1750.93
The 'croutonversion' output of my saucy chroot is -
(saucy-x64-unity-desktop)user@localhost:~$ croutonversion
crouton: version 1-20131220111353~desktops:63b0426f
release: saucy
architecture: amd64
targets: unity-desktop,unity
host: version 5116.80.0 (Official Build) beta-channel parrot
This is no biggy for me, I'll place this on hold for now and wait 'til it's a little more ready-for-prime-time or at least testing.
Thanx all,
-DennisL
DennisLfromGA
on 19 Feb 2014
I had locale issues in building with the manjaro mix so I switched it over to the precise mix and I get different errors.
It fails while doing ./automake
Looks like I had some missing dependencies. running ./configure complained about python3-dev. after sudo apt-get install python3-dev it still complains there is no python3-dev...
edge226
on 19 Feb 2014
@DennisLfromGA This kind of library issues are not unexpected, as Chromium OS ships different library versions. If there is a need to compile an binary in the chroot, for use in Chromium OS, it needs to be compiled statically.
Now, to get containers working, we have a few options:
- Use
minijail0. That's the nicest as the tool is already provided by Chromium OS. But, AFAIK, the tool is not able to join an existing container. Not nice, as we probably only want to have 1 instance of each chroot running at a time. - Use
unshare/nsenter(we'll need to fetch the latest util-linux, build the tools statically). We also need to check that the tools can do everything we need to do (I suspect they do). - Use a custom tool (that's the approach I've taken in my old branch: https://github.com/drinkcat/chroagh/tree/container.tmp).
Finally, Chromebooks on kernel 3.4 wil probably need to stay with the current approach, as kernel 3.4 lacks an important feature: being able to join an existing namespace.
drinkcat
on 19 Feb 2014
@drinkcat - Thanx for the info., I think I'll just cool my heels on this for a while until it's a little more settled. I'll spend a my time playing with the new 'multi-profiles' feature in the canary channel, it's really slick. I made a screencast that shows off some of it's features, I've just scratched the surface I'm sure.
DennisLfromGA
on 19 Feb 2014
@drinkcat The info is very helpful, Thank you. I think the rest could potentially be done by abusing a target. I think if it is done this way it will work across multiple distributions easily. I definitely see the possibility of this easily being done with lxc, docker or minijail0 however I have not had success with minijail0 in getting anything other than bash to work. systemd-nspawn was not happy. I like the idea of a docker implementation because it seems like it should be easy to migrate a backup of the chroot into a docker image file. The benefit of the docker image file being cross compatible onto a traditional desktop as well as the chromebook seems very cool and beneficial.
edge226
on 19 Feb 2014
I don't think we should try so hard to get systemd working in crouton. Systemd tries to be include everything, and use any possible dependencies one can think of. This make it a bit bloat, inadaptive and hard to be deployed in the virtualized world. For instance, systemd can not even work well with docker: https://github.com/dotcloud/docker/issues/3629
I think the major obstacle is the use of cgroups, which is a strange kernel subsystem, not well-studied process management model. In terms of implementation, it is orthogonal with PID namespace, but in terms of functionality, there is a clash in process containment. Not to mention that not all cgroups functionalities are included in chromeos kernel.
The other thing is the inclusion of udev inside systemd, which is kind of crazy for a guest system having no access to physical devices.
If systemd as a guest is so unfriendly even to docker, why don't we stick with upstart/sysvinit, or even openrc for crouton? We don't need that much functionality within a crouton environment, we don't need super fast booting time for crouton, so I guess those choices will be acceptable.
ccaapton
on 19 Feb 2014
Currently crouton uses none of the above (it doesn't really "boot" per se), and upstart straight out doesn't work. The primary issue is that DEs like unity and gnome are increasingly depending on init daemon functionality. The secondary issue is that it takes some effort to launch daemons, since you have to manually adapt the init script.
For the primary issue, if we can find a way to launch the DEs we care about without systemd/upstart, then I agree, we don't need it. The problem is we're not compiling the DEs ourselves, so often the choice between requiring systemd/not requiring it is a compile-time choice that has already been made by the distribution, usually in the direction we don't want.
The secondary issue is actually easier; it's just a bit of scripting to parse an upstart config and extract the commands to run to launch/keep track of/kill the daemon, at least for basic configs.
dnschneid
on 19 Feb 2014
I think it would be cool to have a target, or targets, with systemd in crouton, not for any added functionality, as the kernel is running and the chroot is "booted" as mentioned, but for educational considerations. All major distros use systemd, or will soon. To use systemd on the ARM hardware today, you can do it by booting from USB or SD, but then you can't toggle to chromeos with hot-keys.
tedm
on 19 Feb 2014
@ccaapton I personally disagree on the need of systemd. I think it is very needed at this time with debian and ubuntu moving over to systemd. There are things that just dont work without it. Daemons as mentioned are very difficult with a standard chroot and often do not work at all in the distributions that run systemd. It would also simplify a vast majority of things having this functionality working. The people in your post started systemd with init and not with the suggested systemd-nspawn as is the suggested manner for using containers.
I have not gotten it working yet, I am pretty sure it is entirely possible though.
edge226
on 20 Feb 2014
some interesting information on lxc and systemd: https://wiki.archlinux.org/index.php/Lxc-systemd
edge226
on 21 Feb 2014
Issue #457 just supplements my reasoning. Without proper init things do not quite work as they usually would.
edge226
on 22 Feb 2014
@edge226 The issue #457 mentioned upstart may solve this as a PID 1. Since upstart is the default init for ubuntu as for now, and I already started LXC with upstart, why not use it at the moment?
The crouton project is kind of special target. It is not a traditional desktop distribution, but similar to xen/debian kfreebsd in the sense that software need to be ported/patched. If you want to get systemd working, you can not do anything with crouton, by patch systemd!!! Given that systemd is never friendly to portability concerns, and crouton is exactly in the minority group that get abandoned, the only word I can say is good luck.
Put it in another way: this lxc/systemd issue, together with docker issue 3629, is a upstream bug, it is not crouton's business.
ccaapton
on 22 Feb 2014
@ccaapton I see. That sort of sucks. I use a systemd based distribution, including with crouton. It really would be nice if everything worked correctly.
edge226
on 22 Feb 2014
@dnschneid To get crouton play well with PID 1, I suggest to refactor enter-chroot into separate parts:
- Arguments and environment variables parsing, including $USER/$BACKGROUND/$NOLOGIN...
- Changes need for the host, including disablehungtask/Allow X server running as normal user, and so on.
- bindmount/tmpfsmount. Minijail can do all the bindmount atomically together with chroot. We could collect all the need task in bindmount{}, and delay the mount until chroot is to be called, so it behaves more similar to minijail.
- job after chroot. This include launching system dbus. I expect that we will do less and less work in this part, but delegate the work to PID 1(sysvinit/upstart/openrc at the moment, and hopefully systemd eventually).
I hope the last part will be more declarative in style, maybe similar to inittab.
This refactoring will be very helpful for all kinds of experiments. Currently my minijail/unshare tricks only deal with 3, so I can not integrate part 1,2,4, thus not able to start dbus/X11 for further experiments.
Should I open a separate issue for this?
Update: I'm not familiar with bash scripts, so the [-n $var}/[ -z $var]/${2:-"$1"} thing really drive me crazy. Chromeos already include python, so why not use python for enter-chroot?
Update 2: tmpfsmount and "ln -sfT" should be done inside chroot
ccaapton
on 22 Feb 2014
What about just enabling aliases so one can learn the systemd syntax, for starting and stopping processes, even if enable / disable and boot related process manipulation is not possible because of the chroot status.
All enterprise distros, and many hobby distros are utilizing systemd, not just from the init process, but real-time system management, from d-bus to udev, from socket listeners to i/o status, they leverage these subsystems, and if crouton chroots don't support it, folks won't be able to learn the syntax for running and maintaining a real linux OS.
Upstart from Ubuntu was tried by red hat, suse, fedora, and others, and now all have reverted, and Ubuntu will be reverting starting in some support for dependencies in 14.04, and fully in 16.04.
It was painful for me to learn systemd when Arch forced it in, in 2012, but it is harder to teach folks systemd, when they have some systems that use it, and some that don't.
So, please, at least consider putting the dormant syntax or aliases, or enable them to work where applicable in their native form when enabled as a crouton target.
tedm
on 23 Feb 2014
@ccaapton: sure, open a separate issue to keep the discussion clean. ChromeOS doesn't include python on release images.
dnschneid
on 23 Feb 2014
Hi, folks. I've successfully started systemd as PID 1 on my C720, using minijail and the most recent version of unshare. Hope this will help us move forward.
My chroot is Debian/sid, and I installed systemd inside it. I also copied binary executable "unshare" from archlinux in to $CHROOT/bin/unshare
Below are the scripts I used:
start-chroot.sh
#sh -e
OVERLAY='/home/chronos/user/Downloads/Crouton'
CHROOT="/usr/local/chroots/sid"
# NOTICE that I binded mount rc.local and fstab from OVERLAY into $CHROOT/etc/
# in the later part
PREINIT="mount -a; \
sh -e /etc/rc.local ;\
exec /lib/systemd/systemd;"
echo 0 > /proc/sys/kernel/hung_task_panic
drm_relax_file="/sys/kernel/debug/dri/drm_master_relax"
if [ -f "$drm_relax_file" ]; then
echo 'Y' > "$drm_relax_file"
fi
LC_ALL='POSIX' nohup minijail0 \
-b /home/chronos/user/Downloads,/home/trx/Downloads,1 \
-b /dev,/dev,1 \
-b /dev/pts,/dev/pts,1 \
-b /dev/shm,/dev/shm,1 \
-b $OVERLAY/etc/fstab,/etc/fstab \
-b $OVERLAY/etc/rc.local,/etc/rc.local,1 \
-b /sys,/sys,1 \
-b /sys/fs/fuse/connections,/sys/fs/fuse/connections,1 \
-b /var/run/dbus,/var/host/dbus,1 \
-b /var/run/shill,/var/host/shill,1 \
-b /var/run/cras,/var/host/cras,1 \
-b /var/lib/timezone,/var/host/timezone,1 \
-C $CHROOT \
-v \
/bin/bash -c "/bin/unshare -pf /bin/bash -- -c \"$PREINIT\"" >/dev/null &
$OVERLAY/etc/fstab:
# Begin /etc/fstab
# file system mount-point type options dump fsck
# order
proc /proc proc nosuid,noexec,nodev,relatime 0 0
tmpfs /run tmpfs noexec,nosuid,mode=0755,size=10% 0 0
#tmpfs /run/lock tmpfs noexec,nosuid,nodev,size=5120k 0 0
$OVERLAY/etc/rc.local:
touch /tmp/.X0-lock
fixabslinks() {
echo "`readlink -m $1`"
}
# If /var/run isn't mounted, we know the chroot hasn't been started yet.
if mountpoint -q "`fixabslinks '/var/run'`"; then
firstrun=''
else
firstrun='y'
fi
# Add a shm symlink to our new /var/run
ln -sfT /dev/shm "`fixabslinks '/var/run'`/shm"
# Add a /run/udev symlink for later versions of udev
ln -sfT /dev/.udev "`fixabslinks '/var/run'`/udev"
# Add a /var/host/cras symlink for CRAS clients
ln -sfT /var/host/cras "`fixabslinks '/var/run'`/cras"
# Be backwards-compatible, just in case
if [ -f "/etc/X11/host-Xauthority" ]; then
ln -sfT '/var/host/Xauthority' "/etc/X11/host-Xauthority"
fi
# Prepare chroot filesystem
# Soft-link resolv.conf so that updates are automatically propagated
ln -sf '/var/host/shill/resolv.conf' "$CHROOT/etc/resolv.conf"
exit 0
To start chroot with systemd as PID 1:
- Run "sudo sh -e start-chroot.sh", then the whole chromeos session is somehow killed, and restarted.
- Now switch to tt2 using ctrl+alt+F2(forward).
- Switch to tt3 using Ctrl+Alt+F3(Refresh). In fact tty3 is started and controlled by debian/sid.
Now you can see all the services are running using "systemctl" or "ps -ejH". - In tty3, you can run "xinit". I successfully launched awesome in this way.
Need to solve:
- Stop systemd from killing chrome session
- Better integration.
Update: 1 can be solved by disable console-getty.service & [email protected]
Update2: Added nohup so the systemd/chroot now can run headlessly.
Update3: After addingtouch /tmp/.X0-lock in etc/rc.local as in above code, X11 works! systemd starts tty3, and no longer kill chrome.
ccaapton
on 25 Feb 2014
3) Multiple entry into the same chroot, or will the namespace be automagically shared?
dnschneid
on 25 Feb 2014
@dnschneid I don't know. In this script I changed namespace in two step. minijail0 for the mount namespace, and unshare for the pid namespace. But I suspect it is doable as I didn't not use any seccomp filter/drop capability stuff in the minijail, so it should behave similar to a normal unshare -m.
At this point, I lost access to chroot inside chromeos(chrome session is killed, including crosh), and can only access it in tty3/tty4, etc, so the "start-chroot.sh" script should be non-reentrant. It's more like two independent systems running on the same box. But I think we can still find better way to:
- disable these tty, as I guess very few will people will need that
- stop killing chromeos
- find a way to re-attach.
Regarding 3, I can already re-attach using "ssh myusername@localhost", but there must be a better way.
ccaapton
on 25 Feb 2014
@ccaapton its a decent start :+1:
edge226
on 25 Feb 2014
Hi folks! Looks like docker 0.9 is released, with very exciting features that may benefit we crouton users!
http://blog.docker.io/2014/03/docker-0-9-introducing-execution-drivers-and-libcontainer/
I think libcontainer is particularly interesting, as it offers much finer-grained controls compare to vanilla lxc in terms of namespaces and cgroups. It is also self-contained, and does not depend on lxc or docker itself. So including it into crouton should be easy.
Check this out at https://github.com/dotcloud/docker/tree/master/pkg/libcontainer !
ccaapton
on 10 Mar 2014
I'm glad to report that the Snow Chromebook I referenced before is now up to 3.8.11.
I really look forward to finding the cleanest route to a Dockered localdev environment for Drupal.
jpw1116
on 12 Mar 2014
Exciting! Sounds like namespaces is in the crouton radar now.
dnschneid
on 27 Mar 2014
@dnschneid I actually believe docker+golang should be the main future direction for crouton. Docker, and its most recent subproject nsinit take care of all the namespaces for us, so we can clean up some monstrous scripts like enter-chroot easily.
Golang is also a very attractive alternative to bash script, for these reasons:
- It has official support for both x86 and arm
- Golang binary are almost "statically linked". So unlike python which need a huge runtime, the golang-generated binary is self-contained, this is a huge sell-point for easy-deployment.
- It is also easier for developers and testers. If someone want to compile the golang version of crouton, he can download and install golang into "/usr/local" (see http://golang.org/doc/install#tarball), without a crouton environment needed, i.e golang does not need a bootstrap.
- It has excellent support for cross-compiling. You don't need an arm chromebook to generate binary for arm.
After adopting docker, I think we can get rid of many target files, which are very ubuntu-specific. Crouton could focus more on the host side, and delegate the target-files' jobs to prebuild-image distributors(or docker image-build scripts, aka dockerfile, if someone don't like the idea of prebuild image).
ccaapton
on 30 May 2014
I guess python fell out of favor?
dnschneid
on 31 May 2014
@dnschneid Well, I still believe that python as a scripting language could encourage more people to play around the code on the fly. But python is not included in chromeos, and bootstrap it will be quite painful. So I guess golang can make the right balance between fun to hack and easy to deploy.
Either way, I do think the selection of language could impact the long run popularity and community involvement, so we should ditch bash.
ccaapton
on 31 May 2014
Update on docker 0.9.1 compatibility: no good news. The LXC dependency may be gone but docker still requires config options that are not built into ChromeOS kernels.
(trusty) $ sudo docker.io -d
[/var/lib/docker|4d96f761] +job serveapi(unix:///var/run/docker.sock)
2014/06/04 11:18:26 Listening for HTTP on unix (/var/run/docker.sock)
[/var/lib/docker|4d96f761] +job initserver()
[/var/lib/docker|4d96f761.initserver()] Creating server
[/var/lib/docker|4d96f761] +job init_networkdriver()
[/var/lib/docker|4d96f761.init_networkdriver()] creating new bridge for docker0
[/var/lib/docker|4d96f761.init_networkdriver()] getting iface addr
Failed to inject docker in PREROUTING chain: iptables failed: iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER: iptables: No chain/target/match by that name.
(exit status 1)
[/var/lib/docker|4d96f761] -job init_networkdriver() = ERR (1)
(exit status 1)
[/var/lib/docker|4d96f761] -job initserver() = ERR (1)
2014/06/04 11:18:26 (exit status 1)
Rather than just a iptables error, as explained in this issue, this is likely due to a config option like CONFIG_NETFILTER_XT_MATCH_ADDRTYPE not being set.
Here's an example of the kind of config shenanigans needed on a different Linux platform.
air
on 4 Jun 2014
@air If it can be build as modules then it shouldn't be a problem to include.
You can load additional kernel modules in chromeos. An issue then would be how to maintain the different modules for the different architectures/kernels.
divx118
on 4 Jun 2014
@air You can always play around with the command arguments to change docker's default behavior. For instance "--iptables=false" will get rid of docker's iptables settings, "-g" will change the default directory for settings and images, etc.
I will say that the current chromeos kernel meet all the hardcore requirements of docker.
ccaapton
on 4 Jun 2014
Thanks @ccaapton. Just to document the current state with docker v1.0.1 on Ubuntu Trusty, installed as per Docker install guide:
$ dpkg -l|grep docker
ii lxc-docker 1.0.1 amd64 Linux container runtime
The daemon does run (as per @ccaapton) if you twiddle the options:
$ sudo docker -d --storage-driver=vfs --iptables=false
2014/07/01 16:58:08 docker daemon: 1.0.1 990021a; execdriver: native; graphdriver: vfs
[9203cf46] +job serveapi(unix:///var/run/docker.sock)
[9203cf46] +job initserver()
[9203cf46.initserver()] Creating server
[9203cf46] +job init_networkdriver()
2014/07/01 16:58:08 Listening for HTTP on unix (/var/run/docker.sock)
[9203cf46] -job init_networkdriver() = OK (0)
2014/07/01 16:58:08 WARNING: mountpoint not found
Loading containers: ....: done.
[9203cf46.initserver()] Creating pidfile
[9203cf46.initserver()] Setting up signal traps
[9203cf46] -job initserver() = OK (0)
[9203cf46] +job acceptconnections()
[9203cf46] -job acceptconnections() = OK (0)
And we can extract basic info:
$ sudo docker info
Containers: 2
Images: 5
Storage Driver: vfs
Execution Driver: native-0.2
Kernel Version: 3.8.11
WARNING: No memory limit support
WARNING: No swap limit support
But containers don't run, with a mountpoint error. We seem to get less useful error output than before. Even with -D on the server it's not clear which mountpoint isn't found. On the client:
$ sudo docker run -i -t ubuntu /bin/bash -D
2014/07/01 17:00:41 Error response from daemon: Cannot start container 17f1133fd36df8cf0cb69b246685e2c418d335a4aae57e5cb8ca5a510d554d60: mountpoint not found
And the server:
...
[error] container.go:525 17f1133fd36df8cf0cb69b246685e2c418d335a4aae57e5cb8ca5a510d554d60: Error closing terminal: invalid argument
Cannot start container 17f1133fd36df8cf0cb69b246685e2c418d335a4aae57e5cb8ca5a510d554d60: mountpoint not found
[9203cf46] -job start(17f1133fd36df8cf0cb69b246685e2c418d335a4aae57e5cb8ca5a510d554d60) = ERR (1)
[error] server.go:1025 Error making handler: Cannot start container 17f1133fd36df8cf0cb69b246685e2c418d335a4aae57e5cb8ca5a510d554d60: mountpoint not found
@air Are you using docker inside crouton? I don't think docker can work inside a chroot environment.
There are pre-built docker binary here, you can copy it to /usr/local/bin and have fun!
ccaapton
on 2 Jul 2014
So @ccaapton, should this bug be closed?
alexanderkyte
on 6 Jul 2014
@ccaapton are you suggesting we just run the binary from the ChromeOS shell without entering chroot?
I attempted to run it after downloading the binary and placing it in /usr/local/bin and then running the following command
sudo docker -d --storage-driver=vfs --iptables=false -g ~/data/docker
It all seemed to work ok. I could pull the ubuntu image but when I tried to run
sudo docker run -t -i ubuntu /bin/bash
I get this error message
Error response from daemon: Cannot start container 817f980ff0e6a78510098017c0dc71f0e21f84d9a7a6fed91d5b513afb1bea5b: fork/exec /home/chronos/user/data/docker/init/dockerinit-1.1.0: permission denied
Any thoughts?
shannon
on 8 Jul 2014
@shannonmpoole I have no idea what happened base on the error message. Maybe you can add "-D" to get more debug info.
ccaapton
on 8 Jul 2014
@ccaapton -D gives me the same message. It does actually create the container it just won't let me start it. Anything else I could do to help debug?
For more info, I am running a freshly power-washed Pixel on beta channel (developer mode). Downloaded the docker binary into /usr/local/bin. Started the daemon, did docker pull ubuntu:latest, then tried to run it. Haven't done much else. Maybe I am missing a step?
shannon
on 9 Jul 2014
I got process isolation working on my Chromebook with jchroot, a really basic implementation just calling the relevant syscalls, so it's pretty low on dependencies. I compiled it within my chroot and the binary worked fine outside of it.
Was able to get myself a simple bash shell in my crouton chroot with process namespacing and a different hostname (my main reason for wanting this ;) ) but it's going to take more work (i.e. replacing a whole bunch of stuff in enter-chroot and other shell scripts) to actually get a working Xorg environment including all the dbus/powerd/cras/etc etc glue.
But it might be a more feasible approach than getting Docker or LXC to work on Chrome OS.
mheijkoop
on 16 Aug 2014
@mheijkoop : Yes that's one of approaches I considered in this old comment https://github.com/dnschneid/crouton/issues/364#issuecomment-35471379. There are still some Chromebooks on kernel 3.4 though, so some of the issues remain.
At this stage, I would like to go for the unshare/nsenter solution (we could even try to get unshare shipped by default with Chrome OS: this commit explicitely removed it). I hope to be able to work on that over the next few months.
drinkcat
on 16 Aug 2014
Right. I completely missed that comment, sorry!
mheijkoop
on 16 Aug 2014
Right. I completely missed that comment, sorry!
That's completely understandable, this thread is getting very long ,-)
drinkcat
on 16 Aug 2014
Just another finding: when I use 'pivot_root' instead of 'chroot', I can use 'nsenter' to enter into the crouton space directly from other terminals. There is no need for 'chroot' after 'nsenter', or any local sshd/telnetd forwards. It is very very sweet:)
Sample hacking code snippets:
/usr/local/bin/unshare -Upmfr /bin/bash -c "\
mount --make-rprivate /; \
mount --bind $CHROOT $CHROOT; \
mount --bind $CHROOT/opt $CHROOT/opt; \
/sbin/pivot_root $CHROOT $CHROOT/opt; \
cd /; mount -t proc /proc /proc; \
mount -M /opt/dev /dev; mount -M /opt/var/run /var/host; mount -M /opt/sys /sys; \
umount -l /opt; umount -l /opt;\
mount --bind /var/host/shill/resolv.conf /etc/resolv.conf; \
cd; exec bash --login"
So, reading the Rocket announcement last week, I wondered if it wouldn't be easier to get running than Docker.
Lo and behold, it was easy. Steps:
1) Get the rocket binary
2) Make it executable
3) Execute it, making sure data gets written on an executable mount point.
Explict steps, in the shell:
mkdir -p ~/Downloads/src
cd ~/Downloads/src
wget https://github.com/coreos/rocket/releases/download/v0.1.1/rocket-v0.1.1.tar.gz
tar xzf rocket-v0.1.1.tar.gz
sudo cp rocket-v0.1.1/rkt /usr/local/bin
sudo rkt --dir=/usr/local/rkt --debug=true run https://github.com/coreos/etcd/releases/download/v0.5.0-alpha.4/etcd-v0.5.0-alpha.4-linux-amd64.aci
As per the docs, press ^] three times to exit the container once it's up. (That's ctrl + right-bracket).
The reason I'm using --dir=/usr/local/rkt is because most of the filesystem is mounted noexec, and rocket executes the init script inside the file it downloads. This is probably not the ideal location to keep all these containers, but worked for quick testing.
I have really _no_ idea how crouton works, so I've I don't know how much work needs to be done to use Rocket for crouton, or even if rocket is ready to run any kind of real container.
But yeah, this was very easy.
fwip
on 12 Dec 2014
Heads up, I've been doing some research on rocket, it looks like it's possible to make it run a container that has the container filesystem mounted as root and then just boot /bin/init. While playing around with it though, it seems like something it is doing is causing crosh to start erroring out when trying to start.
I'll probably see if I can get arch booting by this weekend via rkt.
c00w
on 17 Dec 2014
When I did it there was an option to specify the directory to root things
in and I set that to my home directory. If you can't figure it out I'll do
it at home. My biggest problem was that there is too much pass through
required to use it, and it makes chromeos break. The option of less
resistance was to go fix the stuff broken in chromeos that stops me from
installing native Linux, which I still haven't done.
On Sun, Mar 1, 2015, 22:25 Watney [email protected] wrote:
I'm stumbling along, following after @c00w https://github.com/c00w and
@fwip https://github.com/fwip to try out Rocket
https://github.com/coreos/rocket.I successfully installed and ran rkt in /usr/local/rocket, the same
directory that crouton uses.Next, I followed the guide to install an ACI beginning by importing the
keys, but straightaway, terminal reports: Error adding keys: Error adding
key: mkdir /etc/rkt: read-only file systemHow do I enable rkt to write to the directories in Chrome OS, given that
(probably) all of those it's designed to use are read only?—
Reply to this email directly or view it on GitHub
https://github.com/dnschneid/crouton/issues/364#issuecomment-76653711.
c00w
on 4 Mar 2015
@c00w Here’s how I installed and ran Rocket
Create a folder in /usr/local/ for rocket, the same directory that crouton uses
sudo mkdir /usr/local/rocket
Download the current tar.gz into the new folder
sudo wget https://github.com/coreos/rocket/releases/download/v0.3.1/rocket-v0.3.1.tar.gz
Unpack
sudo tar xzvf rocket-v0.3.1.tar.gz
Check permissions and correct to root:root; verify that rkt is executable
cd into rocket-v0.3.1
sudo ./rkt help
This verifies that rkt is executable, and it gives you the switches and options
Create a folder in rock to save your ACIs
sudo mkdir /usr/local/rocket/aci
Now, for the blastoff!
In the same folder as rkt, run this
sudo ./rkt --dir=/usr/local/rocket/aci --debug=true --insecure-skip-verify=true run https://github.com/coreos/etcd/releases/download/v2.0.0/etcd-v2.0.0-linux-amd64.aci
To kill the Container press Ctrl ] three times
Watney
on 4 Mar 2015
I've read though this entire issue, and I really appreciate you're guys commitment to trying to get this to work. My interest is with docker specifically so I can test docker deployments locally on my pixel before paying AWS costs.
It looks like @ccaapton @drinkcat @air and @edge226 spent a considerable amount of time debugging this and trying to find the solution many months ago, but could someone please summarize where we are and if this is still ongoing? hopeful? The conclusion I saw was that the required kernel flags simply aren't compiled into ChromeOS itself?
comjf
on 29 Mar 2015
@comjf You might see Ron Minnich's guide to install Docker. Brace yourself, Ron's bandwidth is off the charts
Watney
on 29 Mar 2015
I have an Acer Chromebook 13 (nyan-big) running arch in a chroot and I have successfully rebuilt the kernel with additional features as indicated by Docker's check-config.sh script. I'm still not getting much success with Docker, though.
Here's the output of the script:
[37minfo: reading kernel config from /proc/config.gz ...[m
Generally Necessary:
- [37mcgroup hierarchy[m: [32mproperly mounted[m [/sys/fs/cgroup]
- [37mCONFIG_NAMESPACES[m: [32menabled[m
- [37mCONFIG_NET_NS[m: [32menabled[m
- [37mCONFIG_PID_NS[m: [32menabled[m
- [37mCONFIG_IPC_NS[m: [32menabled[m
- [37mCONFIG_UTS_NS[m: [32menabled[m
- [37mCONFIG_DEVPTS_MULTIPLE_INSTANCES[m: [32menabled[m
- [37mCONFIG_CGROUPS[m: [32menabled[m
- [37mCONFIG_CGROUP_CPUACCT[m: [32menabled[m
- [37mCONFIG_CGROUP_DEVICE[m: [32menabled[m
- [37mCONFIG_CGROUP_FREEZER[m: [32menabled[m
- [37mCONFIG_CGROUP_SCHED[m: [32menabled[m
- [37mCONFIG_CPUSETS[m: [32menabled[m
- [37mCONFIG_MACVLAN[m: [32menabled[m
- [37mCONFIG_VETH[m: [32menabled[m
- [37mCONFIG_BRIDGE[m: [32menabled[m
- [37mCONFIG_NF_NAT_IPV4[m: [32menabled[m
- [37mCONFIG_IP_NF_FILTER[m: [32menabled[m
- [37mCONFIG_IP_NF_TARGET_MASQUERADE[m: [32menabled[m
- [37mCONFIG_NETFILTER_XT_MATCH_ADDRTYPE[m: [32menabled[m
- [37mCONFIG_NETFILTER_XT_MATCH_CONNTRACK[m: [32menabled[m
- [37mCONFIG_NF_NAT[m: [32menabled[m
- [37mCONFIG_NF_NAT_NEEDED[m: [32menabled[m
- [37mCONFIG_POSIX_MQUEUE[m: [32menabled[m
Optional Features:
- [1mCONFIG_MEMCG_SWAP[m: [1;31mmissing[m
- [1mCONFIG_RESOURCE_COUNTERS[m: [1;31mmissing[m
- [37mCONFIG_CGROUP_PERF[m: [32menabled[m
- Storage Drivers:
- "[34maufs[m":
- [1mCONFIG_AUFS_FS[m: [1;31mmissing[m
- [37mCONFIG_EXT4_FS_POSIX_ACL[m: [32menabled[m
- [37mCONFIG_EXT4_FS_SECURITY[m: [32menabled[m
- "[34mbtrfs[m":
- [37mCONFIG_BTRFS_FS[m: [32menabled[m
- "[34mdevicemapper[m":
- [37mCONFIG_BLK_DEV_DM[m: [32menabled[m
- [37mCONFIG_DM_THIN_PROVISIONING[m: [32menabled[m
- [37mCONFIG_EXT4_FS[m: [32menabled[m
- [37mCONFIG_EXT4_FS_POSIX_ACL[m: [32menabled[m
- [37mCONFIG_EXT4_FS_SECURITY[m: [32menabled[m
- "[34moverlay[m":
- [1mCONFIG_OVERLAY_FS[m: [1;31mmissing[m
- [37mCONFIG_EXT4_FS_SECURITY[m: [32menabled[m
- [37mCONFIG_EXT4_FS_POSIX_ACL[m: [32menabled[m
As you can see, the "generally necessary" bits are all enabled as are the btrfs and devicemapper bits. In theory, this would mean I could use either option as a storage driver. I also have the lxc package installed, so I could use either lxc or native exec drivers.
When I run the docker daemon with all the defaults (docker -d), I am able to pull containers like busybox. However, I can't run them. Here is the output when executing the run command:
[jmt@localhost ~]$ sudo docker run --rm -it busybox /bin/bash
FATA[0000] Error response from daemon: Cannot start container e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d: mountpoint for cpuset not found
[jmt@localhost ~]$
And here is the output from the daemon:
[jmt@localhost ~]$ sudo docker -d
INFO[0000] +job serveapi(unix:///var/run/docker.sock)
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
INFO[0000] +job init_networkdriver()
INFO[0000] -job init_networkdriver() = OK (0)
INFO[0000] WARNING: mountpoint for memory not found
INFO[0000] Loading containers: start.
..........
INFO[0000] Loading containers: done.
INFO[0000] docker daemon: 1.5.0 a8a31ef; execdriver: native-0.2; graphdriver: devicemapper
INFO[0000] +job acceptconnections()
INFO[0000] -job acceptconnections() = OK (0)
INFO[0007] POST /v1.17/containers/create
INFO[0007] +job create()
INFO[0008] +job log(create, e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d, busybox:latest)
INFO[0008] -job log(create, e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d, busybox:latest) = OK (0)
INFO[0008] -job create() = OK (0)
INFO[0008] POST /v1.17/containers/e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d/attach?stderr=1&stdin=1&stdout=1&stream=1
INFO[0008] +job container_inspect(e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d)
INFO[0008] -job container_inspect(e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d) = OK (0)
INFO[0008] +job attach(e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d)
INFO[0008] POST /v1.17/containers/e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d/start
INFO[0008] +job start(e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d)
INFO[0008] +job allocate_interface(e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d)
INFO[0008] -job allocate_interface(e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d) = OK (0)
INFO[0008] +job log(start, e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d, busybox:latest)
INFO[0008] -job log(start, e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d, busybox:latest) = OK (0)
INFO[0008] +job release_interface(e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d)
INFO[0008] -job release_interface(e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d) = OK (0)
INFO[0008] -job attach(e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d) = OK (0)
INFO[0008] +job release_interface(e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d)
INFO[0008] -job release_interface(e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d) = OK (0)
ERRO[0008] Warning: error unmounting device e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d: UnmountDevice: device not-mounted id e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d
INFO[0008] +job log(die, e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d, busybox:latest)
INFO[0008] -job log(die, e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d, busybox:latest) = OK (0)
Cannot start container e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d: mountpoint for cpuset not found
INFO[0008] -job start(e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d) = ERR (1)
ERRO[0008] Handler for POST /containers/{name:.*}/start returned error: Cannot start container e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d: mountpoint for cpuset not found
ERRO[0008] HTTP Error: statusCode=500 Cannot start container e68722942d423dc0f26dbe848211554c2f2bcc0a7e093222905759d281e0b34d: mountpoint for cpuset not found
A similar issue is discussed in docker/docker#6681 and one of the collaborators posts a request for certain commands to be run, to indicate which cgroup objects are available (I think). I ran those same commands, and here is the resulting output:
[jmt@localhost ~]$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 0 1 1
cpu 1 8 1
cpuacct 0 1 1
devices 0 1 1
freezer 2 9 1
perf_event 0 1 1
[jmt@localhost ~]$ cat /proc/self/cgroup
2:freezer:/
1:cpu:/
[jmt@localhost ~]$ grep cgroup /proc/mounts
none /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,nosuid,nodev,noexec,relatime,cpu 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
none /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,nosuid,nodev,noexec,relatime,cpu 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
The cpuset cgroup subsystem is enabled but for some reason is not mounted nor is it listed in /proc/self/cgroup. I believe this is what Docker is looking for but cannot find. I do not yet know how to fix this problem, but I am working on it.
mathuin
on 29 Mar 2015
It works! So far. Under the most minimal of testing. But it works.
I wrote the following script to properly mount all the cgroups:
#!/bin/bash
for subsys in `cut -f1 < /proc/cgroups | grep -v subsys_name`; do
mnt=/sys/fs/cgroup/$subsys
if [ ! -e $mnt ]; then
mkdir $mnt
mount -t cgroup none $mnt -o $subsys
fi
done
I also modified the kernel to support the memory cgroup. This required adding new config variables:
CONFIG_MEMCG_SWAP=y
CONFIG_MEMCG_SWAP_ENABLED=y
I started up my arch chroot and ran docker -d in one window as root, then ran the following command in another window of the same chroot:
docker run --rm -it armv7/armhf-ubuntu:14.10
I got a bash prompt from inside a container. https://github.com/djmaze/armhf-ubuntu-docker/README.md suggests tagging releases so existing Dockerfiles referencing FROM ubuntu and the like work correctly.
I have _not_ done any of the following yet:
- constructed an image from a Dockerfile
- linked containers together
- tried fig^Wdocker compose
But this is very exciting and I wanted to share it with you. :-)
mathuin
on 30 Mar 2015
@Watney Thanks for that awesome guide, however, I am skeptical of following because of the warning section. That it will break things like netflix going forward. Is there no way of just updating the kernel without switching to ChromiumOS? I want my cake and to eat it too haha.
@mathuin This is great news, but when you say things like you _modified the kernel to support_ what exactly do you mean by that? Does this involve following the steps for installing ChromiumOS? Also have you thought about writing a blog post about this? I am guessing with the new chromebooks (especially pixel 2) many more people are interested in making a true developer laptop.
comjf
on 30 Mar 2015
@comjf I documented the process I used to build the kernel here: http://www.reddit.com/r/chromeos/comments/307lpx/need_help_building_and_installing_kernel_on_acer/cpuvsky This does not mean following the steps for installing ChromiumOS but instead something different. I may make a blog post about what I've learned through my work blog, and if I do I will ping you here and let you know. :-)
mathuin
on 30 Mar 2015
@mathuin Docker ... _totally amazing_! Really nice work with the kernel
Watney
on 1 Apr 2015
@Watney @fwip
It seems that rkt can run docker images: https://github.com/coreos/rkt/blob/master/Documentation/running-docker-images.md
I tried this myself to see if I could run an archlinux image, but alas it failed:
$ sudo rkt --dir=/usr/local/rkt/aci --debug --insecure-skip-verify fetch docker://logankoester/archlinux
rkt: fetching image from docker://logankoester/archlinux
rkt: warning: signature verification has been disabled
Downloading layer: 626e4432ce9d50a71be849f7ebc7b0df14539eb93d57e0598e6144925df2cefd
Downloading layer: f7705805f4fbadb202715bf3c75b15e6b34382b19363586c21d06c527642f9f6
Downloading layer: d8c5babd88e24549413cf2a862b8795de32b619a583e069c7d7c8bc849f814ae
Downloading layer: dce0559daa1b07151a9af703077f50cbe8300bee79707fe663118c098e73269e
Downloading layer: 9b0516337e5a10feed4b9e2d312aeec22e745da6bcc19401c9987cfaa0ab7f8e
Downloading layer: 511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158
sha512-4a1ecbd991bf7c82be13309382770140
$ sudo rkt --dir=/usr/local/rkt/aci --debug --insecure-skip-verify run sha512-4a1ecbd991bf7c82be13309382770140
2015/04/20 14:16:52 Preparing stage1
2015/04/20 14:16:53 Loading image sha512-4a1ecbd991bf7c82be133093827701402dc23ddab35bff5144c5889568bbd432
run: error setting up stage0: error: image sha512-4a1ecbd991bf7c82be133093827701402dc23ddab35bff5144c5889568bbd432 has no app section
I honestly don't know if this is how rocket was intended to be used, but it would be cool if I could get a full arch system going with this.
bendavis78
on 20 Apr 2015
@bendavis78 I used fetch docker://pritunl/archlinux and rocket similarly returned has no app section Given that I don't know what I'm doing and that I don't understand much about the Arch docker image, it would be pretty much luck if I got a command line (_shrug_)
Next, I moved on to Ubuntu by substituting fetch docker://ubuntu followed by
sudo ./rkt --dir=/usr/local/rkt/aci --debug --insecure-skip-verify run sha512-82f05b9f14e785e54fa7a674e4e0aaaa /bin/echo 'Hello world'
Rocket successfully launched, but abruptly shut down reporting:
Service exit-watcher.service is not needed anymore. Stopping.
Shutting down.
This is apparently a known issue with a confirmed workaround. However, I haven't been able to get it to work, which says more about me than discussion over at rkt.
Please let us know if you make any progress!!
Watney
on 22 Apr 2015
I installed a custom kernel on my Pixel 2015, with options similar to @mathuin 's above, ie. enough for docker to use the devicemapper storage backend.
I am able to start/run containers (well, only tested with 1 which runs a JIRA server), only the "docker exec -it
To help others interested, I put the scripts I used to compile/install the kernel from my archlinux chroot here: https://gist.github.com/perpen/b07606f91a7dcbafcaf8
I voluntarily do not document the scripts as I don't want anybody to run them blindly - that would pretty much brick your chromebook. I may clean write robust/portable scripts next weekend.
@mathuin did you play some more with docker?
perpen
on 18 May 2015
I confess that I stopped playing with docker on the Chromebook after realizing that the kernel changes I had made were causing some stability issues. I recommend running the docker daemon in debug mode to see why the exec command fails, that was helpful to me.
mathuin
on 18 May 2015
Thanks Jack. I have been running the custom kernel for 3 days and didn't notice stability issues - but then I didn't use docker much.
May be some other option you may have enabled - you could compare your kernel config with the one in my gist linked above.
I run the daemon with "-D" but it is not helping much. This is the error I get when doing docker exec -it jira6 bash:
INFO[0080] -job log(exec_start: bash , a9ca5bde6de932d81b3d59abc717e6fab9a0fc4686ab525aeffc4013809f4b4e, jira6:latest) = OK (0)
DEBU[0080] attach: stdin: begin
DEBU[0080] attach: stdout: begin
DEBU[0080] attach: stderr: begin
DEBU[0080] Calling POST /exec/{name:.*}/resize
INFO[0080] POST /v1.18/exec/10e615a3664c59170cef11078235897ab06665df11f62b1327454e8cd5e4b014/resize?h=28&w=193
INFO[0080] +job execResize(10e615a3664c59170cef11078235897ab06665df11f62b1327454e8cd5e4b014, 28, 193)
WARN[0080] exit status 1
ERRO[0080] Error running command in existing container a9ca5bde6de932d81b3d59abc717e6fab9a0fc4686ab525aeffc4013809f4b4e: [8] System error: open /proc/158/status: no such file or directory
DEBU[0080] Exec task in container a9ca5bde6de932d81b3d59abc717e6fab9a0fc4686ab525aeffc4013809f4b4e exited with code -1
DEBU[0080] attach: stdout: end
DEBU[0080] attach: stderr: end
bad file descriptor
I'll try some stracing before posting to some docker group.
perpen
on 19 May 2015
I see the discussion has been hijacked on Docker :)
I have some experience with Docker, and although I am a big fan of it for bigger machines I don't think it should be used at all for crouton. The reason is, Docker is a container engine for managing applications, which is a good fit for servers. Here we have a chromebook, and we want to still use it as a minidesktop-tablet-you-name-it, so the added complexity of Docker is not worth it.
I think we should stick with LXC which is native, or maybe some container technology which is more desktop-oriented? I know Ubuntu has been working on something like that, although I didn't try it.
Docker is totally not a replacement for full systems chroots, and although somebody here mentioned systemd I would skip that problem completely for now (and use hacks as @dnschneid said), until cgroups support (both in-kernel and userland) is stabilized.
Once again, the right tool for the right job.
gdm85
on 22 May 2015
Maybe to you the chromebook is just for "minidesktop"/tablet stuff, but I (and many others) use it differently.
My personal use case: I have a collection of dockerfiles I use to bring up apps under different configurations (build servers, Atlassian tools, Nexus, etc), to test custom code or various integrations. Great use case for docker.
Without docker on my Pixel I had to remote into other boxes (e.g. Digital Ocean nodes) to use my dockerfiles. Not anymore.
Plus... I _like_ docker and I enjoy using it :)
Beyond this, crouton allows us to make our chromebooks into proper development environments for a variety of technologies, and I don't see why we should limit ourselves to what works right now out of the box.
perpen
on 22 May 2015
Same here. I am still opting for an easy to spin up, destroyable local dev environment. The question still seems to be if a native Chrome OS solution will ever arise. Otherwise, what's the simplest chroot method?
jpw1116
on 22 May 2015
@jpw1116 the simplest chroot method is...crouton?!
@perpen by reading this comment I thought that there was even an idea at some point to use containers instead of crouton's chroot, which I think it's not practical.
If you want to use it from within the chroot then it's your business :) inception anyone?
gdm85
on 22 May 2015
@gdm85, a major advantage of using LXC (or similar container system) would be the ability to use systemd, which requires a PID namespace (right now chromeos/crouton doesn't support this out of the box). For better or worse, Linux distros are moving in this direction (including Ubuntu). More and more applications will come to rely on systemd, so it's inevitable that we'll need a chroot solution that supports PID namespacing.
Another advantage from a developer's perspective would be to only spin up the services that you need for a particular development task, saving CPU, memory, and battery life.
bendavis78
on 22 May 2015
@gdm85, by simplest I meant a _functioning_ Docker instance via chroot. Of course, I've been running an LTS in crouton for ages.
jpw1116
on 22 May 2015
Ok, so it was not my impression, the discussion has really gone OT... :)
1) the original topic is about how to run containers (LXC) within the crouton chroot
2) @bendavis78 if I understood correctly you would like to replace the chroot with a container, may I suggest that goes in a separate issue?
3) @jpw1116 nowadays Docker doesn't even run LXC anymore by default, may I suggest another issue too? Because there might be more Docker-specific issues arising also over time, as both crouton and Docker are in high flux development
gdm85
on 23 May 2015
I apologise @gdm85, but I am going to confuse things even more wrt containers being inside or outside the chroot :)
I am now running the docker daemon outside my chroot, but I use it (as normal) from inside the chroot.
I run the following script on Chrome OS boot: https://gist.github.com/perpen/980743bb39410e3ac9f6
All seems fine so far, at least for the scenarios I tried. And it fixed my problems with docker exec!
Of course this still requires a custom kernel :(
Ofc having the daemon in Chrome OS made me think of using docker as a replacement for the current chroots. Taking inspiration from https://github.com/maci0/docker-systemd-unpriv I tried to create a non-privileged archlinux container running systemd and I failed, probably b/c of my lack of understanding of systemd.
I started this same container just once with --privileged, and it seemed to work at least a bit, e.g. journalctl worked. But this had undesired side-effects (changed the audio volume?!, remounted a USB key) and I gave up at this point as I don't like the privileged idea. I'll explore again some other weekend.
perpen
on 23 May 2015
Well, I just couldn't resist. I have a seemingly functioning non-privileged docker container with systemd.
Yesterday I was unable to view any systemd output, but tonight I started the container with process strace -f systemd --system and at least I could see what the problems were.
I had to do manually some of the things systemd couldn't do in a non-privileged container: I created in Chrome OS a dedicated /run-systemd tmpfs mount point, mapped to volume /run. And I mounted in Chrome OS /sys/fs/cgroup/systemd using options I got from a working archlinux vm, and mapped /sys/fs/cgroup into the container.
Then systemd ran normally.
Interestingly it seems some systemd units now check for required linux capabilities before starting. I suppose this is intended for use in containers, nice to see them thinking of the use case.
Was fun but I'm probably not going to be using this at all! But that made me curious about the possibility of using minijail or other compact solutions and have the option of using systemd in our chroots.
I searched the issues but I'm still unsure about how far we are with this.
perpen
on 25 May 2015
@perpen I feel like systemd and cgroups have still a long way to go before being mature..and you can't expect non-breaking changes in Docker either ;)
gdm85
on 25 May 2015
This pm I got an archlinux/systemd chroot running using nsinit, with mount/ipc/uts/pid namespaces. nsinit is a utility which comes with libcontainer, the low-level lib docker uses to create containers.
I am running a custom kernel but looking at the config for the standard Chrome OS kernel it seems to have all the basic namespacing stuff enabled - so I am not expecting problems with a vanilla crouton system.
I'm going on hols soon but after that I'll try making a PoC replacement for the current chroot mechanism.
perpen
on 31 May 2015
@perpen Did you try nsinit in unprivileged mode, aka uid_map/gid_map stuff? I'm quite interested in running everything without root. I tried nsinit a couple of times with the stock kernel(C720 on 3.8.11) without luck.
Currently I'm running everything unprivileged with "unshare -Umpf" on stock kernel with some custom scripts(pivot_root etc). Most stuff works, including Xserver, but it is kind of adhoc.
ccaapton
on 3 Jun 2015
@perpen, could you post the container.json you used with nsinit?
bendavis78
on 6 Jun 2015
Hi,
Just a quick heads up here about recent changes in the Chrome OS kernel. We've enabled some of the needed options to run Docker/LXC on R45 (current ToT).
We've only enabled this on kernel versions 3.14 and 3.18. Container support isn't deemed solid in kernels before that, so unfortunately those platforms won't see it enabled.
For reference, the platforms running the 3.14 kernel are Rockchip, the 2015 Chromebook Pixel and other Broadwell-based Chromebooks.
olofj
on 11 Jun 2015
_kaboom_ — @olofj you're da bomb!
Watney
on 11 Jun 2015
Is there any chance that these kernel versions will trickle down to current ARM-based Chromebooks?
mathuin
on 15 Jun 2015
Is there any chance that these kernel versions will trickle down to
current ARM-based Chromebooks?There are no plans to update to newer kernels on existing platforms. But
that's off topic for this thread.
Note that the recently introduced RK3288-based Chromebooks run 3.14.
olofj
on 15 Jun 2015
@olofj I'm really excited about your news, can't wait to try it!
@ccaapton no I didn't try uid remapping, I suppose we could remap chronos to root but I'm not sure about the benefit? I would quite like being able to load modules from the container for example (although I didn't test that).
Did you get xiwi working? That's where I got stuck, as iirc the chroot-side process was trying to communicate with the chrome side via /proc/<pid>/maps, which obv. doesn't work with the pid namespace.
@bendavis78 for the container config I have been using you can have a look at https://github.com/perpen/chromeos-nsinit. As I said earlier I did not test nsinit with the vanilla kernel, so if you try it please let me know the result.
perpen
on 19 Jun 2015
@olofj That's a good news. And it's nice to see you show up here to tell us that. :)
tonyxue
on 19 Jun 2015
wondering if anyone has written a howto for this yet? with latest docker, here is how far i get
sudo docker -d --storage-driver=vfs --iptables=false -g ~/data/docker
WARN[0000] You are running linux kernel version 3.8.11, which might be unstable running docker. Please upgrade your kernel to 3.10.0.
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
WARN[0000] Running modprobe bridge nf_nat failed with message: , error: exit status 1
WARN[0000] Your kernel does not support cgroup memory limit: mountpoint for memory not found
FATA[0000] Error mounting devices cgroup: mountpoint for devices not found
Any ideas on how to solve the cgroup mount error?
and with -D output
sudo docker -d --storage-driver=vfs --iptables
=false -g ~/data/docker -D
DEBU[0000] Registering GET, /containers/{name:._}/top
DEBU[0000] Registering GET, /containers/{name:._}/stats
DEBU[0000] Registering GET, /info
DEBU[0000] Registering GET, /version
DEBU[0000] Registering GET, /containers/ps
DEBU[0000] Registering GET, /images/get
DEBU[0000] Registering GET, /images/{name:._}/get
DEBU[0000] Registering GET, /images/{name:._}/history
DEBU[0000] Registering GET, /images/{name:._}/json
DEBU[0000] Registering GET, /_ping
DEBU[0000] Registering GET, /events
DEBU[0000] Registering GET, /images/json
DEBU[0000] Registering GET, /containers/{name:._}/changes
DEBU[0000] Registering GET, /containers/{name:._}/logs
DEBU[0000] Registering GET, /exec/{id:._}/json
DEBU[0000] Registering GET, /containers/{name:._}/json
DEBU[0000] Registering GET, /containers/{name:._}/attach/ws
DEBU[0000] Registering GET, /images/search
DEBU[0000] Registering GET, /containers/json
DEBU[0000] Registering GET, /containers/{name:._}/export
DEBU[0000] Registering POST, /build
DEBU[0000] Registering POST, /containers/{name:._}/wait
DEBU[0000] Registering POST, /containers/{name:._}/attach
DEBU[0000] Registering POST, /containers/{name:._}/copy
DEBU[0000] Registering POST, /containers/{name:._}/exec
DEBU[0000] Registering POST, /exec/{name:._}/start
DEBU[0000] Registering POST, /auth
DEBU[0000] Registering POST, /images/create
DEBU[0000] Registering POST, /images/load
DEBU[0000] Registering POST, /images/{name:._}/push
DEBU[0000] Registering POST, /images/{name:._}/tag
DEBU[0000] Registering POST, /containers/{name:._}/pause
DEBU[0000] Registering POST, /containers/{name:._}/rename
DEBU[0000] Registering POST, /commit
DEBU[0000] Registering POST, /containers/{name:._}/restart
DEBU[0000] Registering POST, /exec/{name:._}/resize
DEBU[0000] Registering POST, /containers/{name:._}/unpause
DEBU[0000] Registering POST, /containers/{name:._}/kill
DEBU[0000] Registering POST, /containers/{name:._}/start
DEBU[0000] Registering POST, /containers/{name:._}/stop
DEBU[0000] Registering POST, /containers/{name:._}/resize
DEBU[0000] Registering POST, /containers/create
DEBU[0000] Registering DELETE, /containers/{name:._}
DEBU[0000] Registering DELETE, /images/{name:.*}
DEBU[0000] Registering OPTIONS,
WARN[0000] You are running linux kernel version 3.8.11, which might be unstable running docker. Please upgrade your kernel to 3.10.0.
DEBU[0000] [graphdriver] trying provided driver "vfs"
DEBU[0000] Using graph driver vfs
DEBU[0000] Using default logging driver json-file
DEBU[0000] Creating images graph
DEBU[0000] Restored 0 elements
DEBU[0000] Creating repository list
DEBU[0000] docker group found. gid: 999
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
WARN[0000] Running modprobe bridge nf_nat failed with message: , error: exit status 1
DEBU[0000] /sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -D PREROUTING]
DEBU[0000] /sbin/iptables, [--wait -t nat -D OUTPUT]
DEBU[0000] /sbin/iptables, [--wait -t nat -F DOCKER]
DEBU[0000] /sbin/iptables, [--wait -t nat -X DOCKER]
WARN[0000] Your kernel does not support cgroup memory limit: mountpoint for memory not found
FATA[0000] Error mounting devices cgroup: mountpoint for devices not found
I've been using docker in virtualbox on my chrombook for a bit, and would love to be able to move out of virtualbox.
sudo docker version
Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): linux/amd64
wenlock
on 27 Jun 2015
Given that there was no answer since June 27th I guess we shouldn't be holding our breath for LXC to come to life inside a crouton chroot? No major architectural overhaul planned by chance. Will use chroot to code but will probably need to lxc-start my containers on a seperate box...can't give up on the sheer speed this chrome os chroot combo offers...
vevmesteren
on 8 Nov 2015
An interesting question has been posed in the 'Chromium OS Discuss forum' - Docker support in kernel.
Since it may depend on kernel 3.10+, it won't work on a lot of the older Chromebooks but it'll be interesting to see what the Developers say about it.
DennisLfromGA
on 21 Jun 2016
Did anyone get this working? I have chrome book pixel, which is supposedly supported by the update. I am on the beta channel, do I have to get on the dev channel for this? I can
t seem to find a way to get it working.
cat-turner
on 21 Nov 2016
In fact, Chrome OS is an obsolete platform. Google gave up on developers a long time ago. Let's see what it will be with the Andromeda project. But IMHO, you will lose less time by migrating to Linux/macOS/Windows platforms.
HLFH
on 21 Nov 2016
@HLFH bummer, I really liked working in it. It's been good. I got by working on c9 and crouton crust for a while but now I need to get real work done. Welp, guess I'm going with Linux. I'm diving into Linux Mint on my old PC.
cat-turner
on 21 Nov 2016
dnschneid
on 21 Nov 2016
I thought this is their way of getting developers to use their google cloud platform? I get it, but I can't pay into that. Maybe if their next chromebook/andromedia-like machine included X years of cloud support on it, but that still doesn't address the problem of dev needs being met secondarily. It's getting in the way of my work trying to find work around, but it's part of the dev life I suppose.
cat-turner
on 21 Nov 2016
@dnschneid Chrome Dev Editor is discontinued. No Arch Linux support for Crouton. And apparently, Google gave up not only on developers but also on the American people: Eric Schmidt & Google were supporting the globalist Clinton: we've seen a lot of people showing evidence that Google Search was biased in favor of Hillary Clinton during the Democratic Party presidential primaries, and during the US presidential election.
HLFH
on 21 Nov 2016
I think that this is not the appropriate forum for your political complaints.
fwip
on 21 Nov 2016
Chrome Dev Editor is discontinued. No Arch Linux support for Crouton. ≠Chrome OS is an obsolete platform
DennisLfromGA
on 21 Nov 2016
I was mostly just disappointed that no political candidate made a platform of pushing a timeline for crouton to use containers, and that nobody in the media commented on the candidates' preferred proportion of salad to croutons. Priorities, people!
dnschneid
on 21 Nov 2016
The media conspirators just didn't pay attention. LOCK THEM UP!
vevmesteren
on 21 Nov 2016
I don't see the big deal about lxc containers. Few chromebooks are powerful enough to really use containers and VMs IMHO. On an i7 desktop with 16GB of RAM, I use docker and core os, and virtualbox, but I wouldn't see a big use of these on chromebooks.
I'm waiting for my Toshiba 2 to be able to run Android apps. It's not a deal killer if it doesn't, as this doesn't have a touch screen.
I read that Trump (uggh) uses a Samsung phone, but his staffers use iphones. I am not sure what I'm going to get after my Nexus 6, but probably not the Pixel, just too pricey. I like the security of the iphones, but not a big apple fan either.
tedm
on 22 Nov 2016
re: arch - I got my Pine64 1.2Ghz / 1GB computer, only like 6 months after I paid for it... anyways, I was bummed that there was little Arch support for it, as that is what I would want to. But a BSP and version called Longsheep ubuntu loaded right up, and works fine. It's at work now as a dedicated nInvaders character based game system. Not bad for $19.00.
tedm
on 22 Nov 2016
Uh, this long and old thread pops up again.
I'd say Chrome OS isn't obsolete at all. Manufacturers are still building better and higher-end chromebooks.
Containers probably won't be possible on chromebooks in the short run as the hardware limitations and, more importantly, if containers are directly available on chromebook one day, we probably won't need crouton any more.
tonyxue
on 29 Nov 2016
@tonyxue,
I don't want to add too much more to this old thread but some recent development has added some fuel to it, inadvertantly maybe.
Containers probably won't be possible on chromebooks in the short run as the hardware limitations
In fact ARC++ is Android apps (Google Play) in a container on CrOS and it works extremely well, my Dell Chromebook 13 uses it almost flawlessly. I say almost because 'nougat' hasn't hit yet which allows resizing windows and such.
and, more importantly, if containers are directly available on chromebook one day, we probably won't need crouton any more.
Agree, probably not using chroots but hopefully something similar in a container.
Just my 2 cents,
-DennisL
DennisLfromGA
on 29 Nov 2016
Related issues
anonymouseprogrammer
·
4Comments
kgingging
·
5Comments
BRFNGRNBWS
·
3Comments
wymby
·
6Comments
jeremyckahn
·
5Comments
Most helpful comment
I think that this is not the appropriate forum for your political complaints.