Docker-alpine: No space left on device

[ibigpapa]

ERROR: Failed to create usr/libexec/gcc/x86_64-alpine-linux-musl/5.3.0/lto1: No space left on device
ERROR: gcc-5.3.0-r0: No space left on device

When attempting to use docker to build the image and passing in a proxy i'm seeing errors during Installing gcc (5.3.0-r0). I submitted an issue to docker thinking it was related to docker as i've got the free space. #https://github.com/docker/docker/issues/28404

I've tried this on 3 different implementations of docker:

• Fedora 24 4x kernel with overlay2 driver #https://github.com/docker/docker/issues/28404
• Docker for windows default driver
• Docker on Centos with devicemapper on LVM

It's not constant if i wait long enough before attempting the build again it works.

I pulled python's docker file to make sure it wasn't just mine doing it.

Python3 Docker file Modified slightly.

FROM alpine

ENV PATH=/usr/local/bin:$PATH LANG=C.UTF-8 PYTHON_VERSION=3.5.2 ENV=PYTHON_PIP_VERSION=9.0.1

COPY Python-3.5.2.tar.xz /python.tar.xz

RUN set -ex \
    && apk add --no-cache ca-certificates \
    && apk add --no-cache --virtual .fetch-deps \
        openssl \
        tar \
        xz \
    && mkdir -p /usr/src/python \
    && tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz \
    && rm python.tar.xz \
    && apk add --no-cache --virtual .build-deps \
        bzip2-dev \
        gcc \
        gdbm-dev \
        libc-dev \
        linux-headers \
        make \
        ncurses-dev \
        openssl \
        openssl-dev \
        pax-utils \
        readline-dev \
        sqlite-dev \
        tcl-dev \
        tk \
        tk-dev \
        xz-dev \
        zlib-dev \
    && apk del .fetch-deps \
    && cd /usr/src/python \
    && ./configure \
        --enable-loadable-sqlite-extensions \
        --enable-shared \
    && make -j$(getconf _NPROCESSORS_ONLN) \
    && make install \
    && if [ ! -e /usr/local/bin/pip3 ]; then : \
        && wget -O /tmp/get-pip.py 'https://bootstrap.pypa.io/get-pip.py' \
        && python3 /tmp/get-pip.py "pip==$PYTHON_PIP_VERSION" \
        && rm /tmp/get-pip.py \
    ; fi \
    && pip3 install --no-cache-dir --upgrade --force-reinstall "pip==$PYTHON_PIP_VERSION" \
    && find /usr/local -depth \
        \( \
            \( -type d -a -name test -o -name tests \) \
            -o \
            \( -type f -a -name '*.pyc' -o -name '*.pyo' \) \
        \) -exec rm -rf '{}' + \
        && runDeps="$( \
        scanelf --needed --nobanner --recursive /usr/local \
            | awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
            | sort -u \
            | xargs -r apk info --installed \
            | sort -u \
    )" \
    && apk add --virtual .python-rundeps $runDeps \
    && apk del .build-deps \
    && rm -rf /usr/src/python ~/.cache
    && cd /usr/local/bin \
    && { [ -e easy_install ] || ln -s easy_install-* easy_install; } \
    && ln -s idle3 idle \
    && ln -s pydoc3 pydoc \
    && ln -s python3 python \
    && ln -s python3-config python-config

CMD ["python3"]

Build command i use

set proxy=http://10.0.75.1:3128
docker build --build-arg HTTP_PROXY=%proxy% --build-arg HTTPS_PROXY=%proxy% --build-arg FTP_PROXY=%proxy% --build-arg ht
tp_proxy=%proxy% --build-arg https_proxy=%proxy% --build-arg ftp_proxy=%proxy% -t test/python-alpine .

Full docker build log

(60/70) Installing renderproto (0.11.1-r2)
(61/70) Installing libxrender-dev (0.9.10-r0)
(62/70) Installing expat-dev (2.1.1-r1)
(63/70) Installing libpng-dev (1.6.21-r0)
(64/70) Installing freetype-dev (2.6.3-r0)
(65/70) Installing fontconfig-dev (2.12.1-r0)
(66/70) Installing libxft-dev (2.3.2-r0)
(67/70) Installing tk-dev (8.6.5-r0)
(68/70) Installing xz-dev (5.2.2-r1)
(69/70) Installing .build-deps (0)
(70/70) Upgrading musl-utils (1.1.14-r12 -> 1.1.14-r14)
Executing busybox-1.24.2-r11.trigger
1 errors; 84 MiB in 84 packages
The command '/bi&& tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz pk add&& rm python.tar.xz     && apk add --no-cache --virtual .build-deps         bzip2-dev ir -p /ug
cc rc/python    readline-dev            sqlite-dev              tcl-dev aders           tk e            tk-dev -dev     xz-dev  openssl zlib-dev     && apk del .fetch-deps     && cd /
usr/src/python     && ./configure         --enab&& wget -O /tmp/get-pip.py 'https://bootstrap.pypa.io/get-pip.py'  make -j$(getc&& python3 /tmp/get-pip.py "pip==$PYTHON_PIP_VERSION" &
& rm /tmp/get-pip.py  ];; fi     && pip3 install --no-cache-dir -o pgrade --force-reinst\( -type f -a -name '*.pyc' -o -name '*.pyo' \) local -depth    \) -exec rm -rf '{}' +
&& runDeps="$( ame test scanelf --needed --noban| sort -u rsive )"     && apk add --virtual .python-rundeps $runDeps nso&& apk del .build-deps  && rm -rf /usr/src/python ~/.cache' ret
urned a non-zero code: 1talled

Another docker file that i can reproduce this with

FROM python:alpine

RUN apk --no-cache add nginx g++ musl-dev linux-headers && pip install flask uwsgi connexion raven[flask] && apk del g++ musl-dev linux-headers && mkdir -p /run/nginx && mkdir -p /app && rm -rf /var/cache/apk/*

COPY ./bootstrap /

RUN mv /nginx.conf /etc/nginx/ && chmod 744 run.sh

COPY ./app /app

CMD ["/bin/sh", "-c", "/run.sh"]

[ibigpapa]

Attempting the same commands from the python3 docker file interactively works just fine
just got it to do the same thing during interactive session
docker run --rm -it alpine /bin/sh
# apk add --no-cache -vv gcc
output of containers space df

# df -h
Filesystem                Size      Used Available Use% Mounted on
none                     55.1G     14.0G     38.3G  27% /
tmpfs                   993.4M         0    993.4M   0% /dev
tmpfs                   993.4M         0    993.4M   0% /sys/fs/cgroup
/dev/sda2                55.1G     14.0G     38.3G  27% /etc/resolv.conf
/dev/sda2                55.1G     14.0G     38.3G  27% /etc/hostname
/dev/sda2                55.1G     14.0G     38.3G  27% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                   993.4M         0    993.4M   0% /proc/kcore
tmpfs                   993.4M         0    993.4M   0% /proc/timer_list
tmpfs                   993.4M         0    993.4M   0% /proc/sched_debug

also tested alpine:edge with same results

[ibigpapa]

Tested by using a base of just python which doesn't use alpine and has a bigger image. It works no space issues. So it appears to be limited to just alpine.

[justincormack]

It is a bug in apk. It reuses the ENOSPC error code in some situations. I reported it upstream, not sure if there has been a fix yet cc @ncopa.

[jbergstroem]

@justincormack do you have a link to the bug in their bug tracker?

[jbergstroem]

Full strace here: https://gist.github.com/jbergstroem/5705f87b03a84f1786f6e0c78a4e1cde

So, fallocate 14mb, mmap, read and then freeze.

[jbergstroem]

Just an update: Tried baking new docker images with alpine 3.5. Had no problems for like 5-10 runs; but then started failing again. Things I've tried/troubleshooted so far:

• reboot (lols)
• increase available memory to docker
• increase qcow2 container with 10G

Strace is similar to above. Stops after reading for a while. Next step is fully reinstalling docker for macOS.

[jbergstroem]

Update: full reinstall of docker for macOS (both stable and beta) didn't help. At the time of the read "freeze", here's the memory/disk situation:

/ # free
             total       used       free     shared    buffers     cached
Mem:       3076140     309056    2767084     151564       1472     203972
-/+ buffers/cache:     103612    2972528
Swap:            0          0          0
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  62.7G      1.1G     58.4G   2% /
tmpfs                     1.5G         0      1.5G   0% /dev
tmpfs                     1.5G         0      1.5G   0% /sys/fs/cgroup
/dev/sda1                62.7G      1.1G     58.4G   2% /etc/resolv.conf
/dev/sda1                62.7G      1.1G     58.4G   2% /etc/hostname
/dev/sda1                62.7G      1.1G     58.4G   2% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                     1.5G         0      1.5G   0% /proc/kcore
tmpfs                     1.5G         0      1.5G   0% /proc/timer_list
tmpfs                     1.5G         0      1.5G   0% /proc/sched_debug
tmpfs                     1.5G         0      1.5G   0% /sys/firmware

[jbergstroem]

I switched from gcc to another large package (clang-libs); which showed the exact same behavior. running docker build with a Dockerfile containing apk add clang-libs worked fine for a few times then started failing..

[jbergstroem]

Steps to reproduce btw:

# seccomp for strace
$ docker run --security-opt seccomp:unconfined -it alpine:3.5 /bin/sh
/ $ apk update
/ $ apk add strace
/ $ strace apk add gcc

[isp14]

I have the same problem building a docker image from alpine 3.4 on ubuntu 14.04 with docker 1.11.2.

Up until today it would build with no issues. Now I get no space left on device, but I have 33G free.

(1/30) Installing ca-certificates (20160104-r4)
(2/30) Installing libssh2 (1.7.0-r0)
(3/30) Installing libcurl (7.51.0-r0)
(4/30) Installing curl (7.51.0-r0)
(5/30) Installing binutils-libs (2.26-r0)
(6/30) Installing binutils (2.26-r0)
(7/30) Installing gmp (6.1.0-r0)
(8/30) Installing isl (0.14.1-r0)
(9/30) Installing libgomp (5.3.0-r0)
(10/30) Installing libatomic (5.3.0-r0)
(11/30) Installing pkgconf (0.9.12-r0)
(12/30) Installing pkgconfig (0.25-r1)
(13/30) Installing mpfr3 (3.1.2-r0)
(14/30) Installing mpc1 (1.0.3-r0)
(15/30) Installing gcc (5.3.0-r0)
(16/30) Installing musl-dev (1.1.14-r14)
ERROR: Failed to create usr/lib/libc.a: No space left on device
ERROR: musl-dev-1.1.14-r14: No space left on device

Filesystem Size Used Avail Use% Mounted on
udev 483M 12K 483M 1% /dev
tmpfs 100M 904K 99M 1% /run
/dev/mapper/jenkins--server--1--vg-root 42G 7.8G 33G 20% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 497M 4.0K 497M 1% /run/shm
none 100M 0 100M 0% /run/user
/dev/vda1 236M 44M 180M 20% /boot

[jbergstroem]

@isp14: could you install strace (see above) and verify that it stops at read/write to fallocate? No need to paste; perhaps in a gist to keep this issue less scroll:y.

[ncopa]

@jbergstroem can you tell more about the "freeze"? When does it happen? during read? Can you run a new strace with strace -r to get the timestamps?

It looks to me that the fallocate calls are all successful. I think it is read that returns 0 which surprises apk and make it think that there is an error, and aborts. But it gives wrong error message.

[jbergstroem]

@ncopa:

     0.000484 read(11, "\366\352\226\356\353\26\230\21)k\250\314\31\"\212\2370\233RZY\353te\350\370\313\"\36\301\371V"..., 8192) = 8192
     0.000489 read(11, "g\2772\253,\\\332U\213\366N\363\274\377\360\221\216\265\\\26\304\375\276\25\v\231\250Wg\310\245\373"..., 8192) = 8192
     0.000468 read(11, "\1\257\331R\363Y\266T\316\322-\345\251\265\345\253\271w7\335\25\367l\227\364{\357\375\3525\267\375"..., 8192) = 8192
     0.000621 read(11, "", 8192)        = 0
   129.851135 munmap(0x7f97b255d000, 16867016) = 0
     0.000409 close(12)                 = 0
     0.000033 writev(1, [{iov_base="\0338\33[0KERROR: Failed to create us"..., iov_len=106}, {iov_base="\n", iov_len=1}],ERROR: Failed to create usr/libexec/gcc/x86_64-alpine-linux-musl/6.2.1/lto1: No space left on device
) = 107
     0.000098 setsockopt(11, SOL_TCP, TCP_NODELAY, [0], 4) = 0
     0.000068 writev(1, [{iov_base="ERROR: gcc-6.2.1-r1: No space le"..., iov_len=44}, {iov_base="\n", iov_len=1}], 2ERROR: gcc-6.2.1-r1: No space left on device

[isp14]

@jbergstroem : just tried with strace but now I´m having trouble reproducing the error. It seems to be installing without any errors. I haven´t changed anything, if the problem returns I´ll post the strace.

[fabled]

Yes, sounds like incorrect error message. read() = 0 indicates that the other end closed the socket. This possibly means proxy closed connection prematurely. The error should probably be something like "network connection reset" or "short read of archive".

[jbergstroem]

@fabled but theres not really anything being downloaded:

openat(3, "usr/libexec/gcc/x86_64-alpine-linux-musl/5.3.0/lto1.apk-new", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0755) = 9
fallocate(9, 0, 0, 14496416)            = 0
mmap(NULL, 14496416, PROT_READ|PROT_WRITE, MAP_SHARED, 9, 0) = 0x7f9bc469c000
...
read(7, "", 8192)                       = 0

[fabled]

@jbergstroem FD 7 is the socket connection, and read() is doing the downloading:

connect(7, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("151.101.92.249")}, 16) = 0
...
sendto(7, "GET /alpine/v3.4/main/x86_64/gcc"..., 56, MSG_NOSIGNAL, NULL, 0) = 56
sendto(7, "Host: dl-cdn.alpinelinux.org\r\n", 30, MSG_NOSIGNAL, NULL, 0) = 30
sendto(7, "User-Agent: libfetch/2.0\r\n", 26, MSG_NOSIGNAL, NULL, 0) = 26
sendto(7, "\r\n", 2, MSG_NOSIGNAL, NULL, 0) = 2
setsockopt(7, SOL_TCP, TCP_NODELAY, [1], 4) = 0
read(7, "HTTP/1.1 200 OK\r\nContent-Type: a"..., 1024) = 336
...
read(7, "", 8192) = 0

The final read that returns zero indicates that remote end has closed the socket (end-of-file).

[fabled]

I believe this commit http://git.alpinelinux.org/cgit/apk-tools/commit/?id=ca368916e0333bf24cdcbdbe42130ec6a92c3f6e should fix the error message.

[jbergstroem]

@fabled damn, sorry -- was reading thew wrong FD. Thanks for pointing me in the right direction.

[leonprou]

I have the same issue on ubuntu 16.04.

Is installing the newest version of apk-tools should fix it?

[metalcamp]

@leonprou
I'm experiencing same issue. Adding "apk add --upgrade apk-tools" does not help.

[aysark]

Getting same issue, but when installing a proprietary linux setup software;

latest: Pulling from arcgis_server_linux
failed to register layer: Error processing tar file(exit status 1): write /home/ubuntu/installer/server/CD_Linux/setup_resources/ArcGISServer.tgz: no space left on device

[andyshinn]

Is this still an issue? For those having the issue, can you try upgrading apk-tools to at least 2.7.0 and see if it is still happening?

[jbergstroem]

Been unable to reproduce for a good while!

[andyshinn]

Go ahead and re-open this if we can get a good way to reproduce it. It's not clear if this is a Docker or Alpine bug.

[cbrwizard]

Just had that error. Seems to be a wrong error message during the unstable network.

[raffraffraff]

Hmm. I think @cbrwizard is on to something. I got this error when building a relatively small container. TL;DR I retried a few times and it eventually finished. But the error occurred on different packages each time it failed. It's not related to disk space.

For what it's worth, I first suspected space issues as I hadn't cleared out the btrfs /var/lib/docker for months, so I did the usual btrfs crap:

docker rm -v -f $(docker ps -a -q)
docker rmi -f $(docker images -a -q)
docker volume prune
systemctl stop docker
mount -o remount,clear_cache /var/lib/docker
systemctl start docker
btrfs subvolume delete /var/lib/docker/btrfs/subvolume/*

With 30GB free I got the errror:

(11/11) Installing python3 (3.6.1-r2)
ERROR: Failed to create usr/lib/python3.6/xml/dom/__pycache__/expatbuilder.cpython-36.pyc: No space left on device
ERROR: python3-3.6.1-r2: No space left on device

...yet I successfully build a 400mb Ubuntu container.

[achimnol]

I came across similar issue, with random outputs such as "no space left on the device", "segmentation fault", or "bad archive" when running apk add --no-cache inside alpine:3.6 docker container. One of those errors comes from different packages whenever I repeat it.
My setup uses Docker for Mac 17.06.0-ce-rc1. I used apk add without any problem for a while, but today the index server looks very slow for me.
I strongly suspect this may be some side-effect error due to unstable networks or the package index server. Maybe one method to reproduce this error is to simulate a slow network connection, such as passing only one byte at a second.

[Astomat]

I have the same problem running on Kubernetes:
ERROR: Failed to create usr/libexec/gcc/x86_64-alpine-linux-musl/6.3.0/cc1obj: No space left on device
ERROR: gcc-6.3.0-r4: No space left on device

[amitripshtos]

looks like alpine 3.6 works better, but still has issues with alpine 3.4
How I reproduce:

1. docker pull python:3.5.3-alpine

FROM python:3.5.1-alpine

RUN apk add -U ---virtual build-dependencies musl-dev linux-headers

error is:
ERROR: Failed to create usr/libexec/gcc/x86_64-alpine-linux-musl/5.3.0/cc1obj: No space left on device
ERROR: gcc-5.3.0-r0: No space left on device

[ay-b]

Just for the purpose of bughunting by smarter people:

FROM php:fpm-alpine

RUN apk add --update --no-cache libpng-dev gcc

Resulting:
```...
(1/1) Installing gcc (5.3.0-r0)
ERROR: Failed to create usr/libexec/gcc/x86_64-alpine-linux-musl/5.3.0/cc1obj: No space left on device
ERROR: gcc-5.3.0-r0: No space left on device
1 errors; 161 MiB in 59 packages

/var/www/html # df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 65792556 3479896 58940884 6% /
tmpfs 1023520 0 1023520 0% /dev
tmpfs 1023520 0 1023520 0% /sys/fs/cgroup
osxfs 243924992 217265848 26403144 89% /var/run
shm 65536 0 65536 0% /dev/shm
/dev/sda1 65792556 3479896 58940884 6% /etc/resolv.conf
/dev/sda1 65792556 3479896 58940884 6% /etc/hostname
/dev/sda1 65792556 3479896 58940884 6% /etc/hosts
osxfs 243924992 217265848 26403144 89% /var/www/html
osxfs 243924992 217265848 26403144 89% /var/log/php-fpm
osxfs 243924992 217265848 26403144 89% /usr/share/nginx/html
tmpfs 1023520 0 1023520 0% /proc/kcore
tmpfs 1023520 0 1023520 0% /proc/timer_list
tmpfs 1023520 0 1023520 0% /proc/sched_debug
tmpfs 1023520 0 1023520 0% /sys/firmware

/var/www/html # uname -a
Linux c548c29c5ce0 4.9.36-moby #1 SMP Fri Jul 14 17:47:34 UTC 2017 x86_64 Linux
```

UPDATE:
I've changed edge repo to http://dl-cdn.alpinelinux.org/alpine/edge/main and suddenly gcc successfully installed. Will report if this will work after repeated rebuilding the container

UPDATE2:
RUN echo "http://dl-3.alpinelinux.org/alpine/edge/main" >> /etc/apk/repositories &&\
apk add --update --no-cache gcc
works like a charm for 10 times already

[stevenkaras]

If someone else runs into this trying to use base images based on 3.4, such as python:2.7-alpine, this reproduces cleanly using:

FROM python:2.7-alpine

RUN apk add --no-cache \
      build-base

This can be worked around by using python:2.7-alpine3.6 as a base image

[johan974]

Docker reset (via the client) worked for me.
$ docker sytsem prune -a ... didn't solve any problems.