Docker-alpine: Temporary error (try again later)

[rommik]

I run into this issue when building a docker image on Ubuntu Host.
The Same build on Windows 10 using docker-tools CLI (so technically inside a VirtualBox VM) has no issues and Docker images are built correctly.

I have reinstalled Docker on my Ubuntu completely to have a fresh version just in case.
Any suggestions what else I can do to troubleshoot this issue?

Build command output
````
Sending build context to Docker daemon 572.4kB
Step 1/15 : FROM alpine:edge
edge: Pulling from library/alpine
cc5efb633992: Pull complete
Digest: sha256:2b796ae57cb164a11ce4dcc9e62a9ad10b64b38c4cc9748e456b5c11a19dc0f3
Status: Downloaded newer image for alpine:edge
---> f96c4363411f
Step 2/15 : RUN apk add --update nodejs
---> Running in 12865b082f34
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/edge/main: temporary error (try again later)
WARNING: Ignoring APKINDEX.066df28d.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/edge/community: temporary error (try again later)
WARNING: Ignoring APKINDEX.b53994b4.tar.gz: No such file or directory
ERROR: unsatisfiable constraints:
nodejs (missing):
required by: world[nodejs]
The command '/bin/sh -c apk add --update nodejs' returned a non-zero code: 1

````

My dockerfile
````
FROM alpine:edge
RUN apk add --update nodejs

create an app directory

RUN mkdir -p /app
WORKDIR /app

create some useful folders

RUN mkdir -p /data

Copy apps build in another folder

COPY main.js /app
COPY package.json /app
COPY version.json /app
COPY dist /app/dist
COPY data /data
COPY app_modules /app/app_modules
COPY node_modules /app/node_modules

expose the port

EXPOSE 5555

start the app

ENV NODE_ENV=development

ENV debug=true

WORKDIR /app

comment if need a console in the container

CMD [ "node", "main.js" ]

````

[andyshinn]

This keeps popping up in other threads and I wonder if there might be a change in the way Fastly is doing DNS. To debug this we really need to see the output of something like curl -v -s http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz > /dev/null to get headers and the HTTP response from within a container. My guess is a temporary issue with a Fastly POP or a change in POP IP address that hasn't yet properly timed out in Docker DNS proxy yet.

[rommik]

@andyshinn I'll see if I can provide you with the output. However, I believe the issue is in Docker Engine, and possibly in DNS caching. My Host Machine was on the corporate network 1, but when we switched it to corporate network 2, the issue was gone.
After speaking to networking people, they weren't able to tell how the difference between networks could have affected the Docker Build. It points to a combination of DNS and Docker Engine. However, we ruled out the DNS, because on both networks we were able to do curl from the host and download the .gz files.

I checked my docker config and its DNS is set to 8.8.8.8 and 8.8.4.4. From both networks these IPs are reachable.

[andyshinn]

I think it is related to #279 which has more history. I think maybe when Fastly changes server IPs they may be redirecting with a 301, which causes apk to choke. But this is why I want to output to see where it is redirecting. I'm not sure if it comes from Fastly or a broken upstream server.

[rlewkowicz]

So I'm having dns issues, but my results are different. My stuff just hangs at getting packages.
https://www.whatsmydns.net/#A/mirrors.aliyun.com

This is the mirror mine tries to use.

curl -vvvv -s http://mirrors.aliyun.com/alpine/v3.5/main/x86_64/APKINDEX.tar.gz > /dev/null
*   Trying 124.47.11.88...
* Connected to mirrors.aliyun.com (124.47.11.88) port 80 (#0)
> GET /alpine/v3.5/main/x86_64/APKINDEX.tar.gz HTTP/1.1
> Host: mirrors.aliyun.com
> User-Agent: curl/7.47.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0

It will just hang here indefinitely in docker (at least for an hour+) and the curl times out. This is from my laptop:

recipes: curl -vvvv -s http://mirrors.aliyun.com/alpine/v3.5/main/x86_64/APKINDEX.tar.gz > /dev/null
*   Trying 219.147.157.84...
* TCP_NODELAY set
* Connected to mirrors.aliyun.com (219.147.157.84) port 80 (#0)
> GET /alpine/v3.5/main/x86_64/APKINDEX.tar.gz HTTP/1.1
> Host: mirrors.aliyun.com
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Tengine
< Content-Type: application/octet-stream
< Content-Length: 732690
< Connection: keep-alive
< Date: Wed, 27 Sep 2017 13:45:47 GMT
< Last-Modified: Thu, 21 Sep 2017 10:13:30 GMT
< ETag: "59c390ca-b2e12"
< Expires: Wed, 27 Sep 2017 13:50:47 GMT
< Cache-Control: max-age=300
< Accept-Ranges: bytes
< Via: cache14.l2nu29[36,304-0,H], cache32.l2nu29[38,0], cache2.cn547[0,200-0,H], cache6.cn547[1,0]
< Age: 18
< X-Cache: HIT TCP_MEM_HIT dirn:3:81977809 mlen:-1
< X-Swift-SaveTime: Wed, 27 Sep 2017 13:45:48 GMT
< X-Swift-CacheTime: 120
< Timing-Allow-Origin: *
< EagleId: db939d4615065199657165411e
<
{ [12630 bytes data]
* Curl_http_done: called premature == 0
* Connection #0 to host mirrors.aliyun.com left intact

@andyshinn this may be a different issue. But it's DNS related, and works one place but not the other. This is from my system, but if you need it from inside docker, I can provide it.

[andyshinn]

Have you tried another mirror? From that debug output, the IP address resolved from Docker and from the laptop are different, so it could definitely be an issue with just that host. Have you actually tried using the same IP on your laptop that Docker resolves to (something like curl -v -H 'Host: mirrors.aliyun.com' http://124.47.11.88/alpine/v3.5/main/x86_64/APKINDEX.tar.gz > /dev/null)?

[rlewkowicz]

Other mirrors do work. I suppose I have to go dig and see if they hard coded a mirror or something. It's odd though that it breaks apk so hard. I suppose I would have thought it to be more resilient

Its in general, related to DNS/apk fragility, but I think perhaps a different issue then op:

https://github.com/laradock/laradock/issues/749#issuecomment-293296687

[matkam]

I was seeing this error on my Drone build server, running on Kubernetes. By default it users "Docker-in-Docker" (dind) for builds. The error went away after I disabled dind on the Drone server.

[Zest-nikunj]

@matkam - Could you detail your solution ? We seem to having the same issue with our drone server.

[matkam]

@Zest-nikunj sure. My Drone server was deployed by a Helm chart to a Kubernetes cluster: https://github.com/kubernetes/charts/tree/master/incubator/drone
I just redeployed it, setting dind.enabled to false. I'd imagine if you're not using Helm or Kubernetes, you'd have to mount the host's docker socket on the drone agent's container: https://github.com/kubernetes/charts/blob/master/incubator/drone/templates/deployment-agent.yaml#L52

[Zest-nikunj]

Thanks @matkam. We are not on Kubernetes and the settings seem correct but we continue to have the issue. We will investigate further. If needed, I shall come back to you. Thanks again.

[publicocean0]

same problem ... before i resolved with reboot. Now it is not working

[tdhawan]

Same problem, just kept on re-trying, and 3rd time it went thru. seems like there is a bad server which the load balancer hits.

[rogerluiz]

same error here

[markk12]

same problem here . I tried many many times ... it is not a bad server ... it seams a bad code line , a bug

[ghost]

I got same. I tried to change dns on docker with no luck.
I'm on Ubuntu 18.04, my colleague can run successfully exactly same build in the same network on Arch

[wlechowicz]

Hit the same problem (again) on Windows + Docker and tried many suggested (in #279) solutions, involving changing apk settings in Dockerfile and such. but nothing helped. What did help finally was to restart Docker and then apk just pulled the APKINDEX like nobody's business.

[CharlesStover]

I get this as well on my workstation using Docker for Windows. Using a mirror did not help.

[brunowdev]

I had this error with Gitlab, i just change docker build command to run with --network host option and it works!

[EvgeniGordeev]

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf helped on Ubuntu 16 build agent to make CI jobs build alpine based docker images again as per https://github.com/gliderlabs/docker-alpine/issues/279#issuecomment-411982620

[montao]

@EvgeniGordeev That solution worked for me too, running Windows 10 host + Ubuntu 18.04 with Vagrant and alpinelinux in a docker container in the Ubuntu.

[hezzze]

@wlechowicz restart docker service solved my issue 👍

[jbcurtin]

@andyshinn , RE: https://github.com/gliderlabs/docker-alpine/issues/334#issuecomment-330096922

It was a DNS error for me. By setting /etc/docker/daemon.json with,

{
  "dns": ["8.8.8.8"]
}

and then restarting docker with,

sudo service docker restart

I was able to build images again.

[hackaugusto]

Just for reference, I got a similar error, but in my case it was a problem with nftables:

 ---> Running in 2522cf9dc903
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
ERROR: unsatisfiable constraints:

Stopping nftables and restarting docker will give the container internet access.

[ShayBox]

Same problem, neither of the previous fixes it.

[shintard]

Since there are times when the bridge is disturbing, it is successful if you execute it as host.
docker build -t hoge:latest . --network=host

[When-Jay]

restarting docker service maybe help you

[akarimvnera]

Run docker build command to run with --network=host option and it works

[rvillane]

any idea how to add this workaround using docker-compose ? the --network flag is not available there

[dacoburn]

I just ran into this, everything had been working fine on:

Windows 10 host with Hyper-V
Ubuntu 18.04 Guest
Docker in the Ubuntu 18.04

Suddenly apk and go mod download started failing. The APK issue was the temporary error. Did the fix suggested in this comment https://github.com/gliderlabs/docker-alpine/issues/334#issuecomment-450598069 and everything was working again. DNS Was working fine on the guest and the host, only when building the docker containers was I having issues.

[zaunist]

same error

[zhuyuanzy]

same error

Ubuntu in VMware
Docker in the Ubuntu 18.04

change virtual machine's network setting to NAT mode, and it works

[zaunist]

same error

Ubuntu in VMware
Docker in the Ubuntu 18.04

change virtual machine's network setting to NAT mode, and it works

It is usually caused by the local network, try to reboot

[SvetaO]

any idea how to add this workaround using docker-compose ? the --network flag is not available there

same problem. It will be great if someone can advise. Restarting docker or reboot is no go for my case

[samuelfusato]

Since there are times when the bridge is disturbing, it is successful if you execute it as host.
docker build -t hoge:latest . --network=host

Solved using:

$ docker build . -t rancher-demo --network=host

Thank YOU!

[Tawliew]

any idea how to add this workaround using docker-compose ? the --network flag is not available there

Same problem. I'm using Manjaro and none of the solutions worked for me

[zaunist]

I have encountered this problem before. The reason is that docker cannot access the external network. Maybe the internal network has designated a private DNS server, so it is necessary to configure daemon.json

[Tawliew]

I solved using this: https://github.com/docker/for-linux/issues/957#issuecomment-627166787

Using Manjaro, one of the first things I do is install firewalld, but it's configured with nftable by default.
I changed it to iptables and then it was fixed.

1. vim /etc/firewalld/firewalld.conf
2. change FirewallBackend=nftables to FirewallBackend=iptables
3. restart the service of firewalld with sudo systemctl restart firewalld.service

[AnatoleLucet]

If you're building an image via docker-compose you might try the following to use the host network at build time:

version: "3.7"

services:
  my_service:
    build:
      context: .
      network: host # this line is the trick

[chanon]

On WSL2 I've been building docker images successfully for months, but today suddenly I'm getting this error. Tried --network=host and it is still not working. Rebooted a few times too.

A wget of the url from inside WSL2 command line works, so WSL2 network connectivity is fine.

Not sure how to fix it.

EDIT: Edited the dns in the daemon.json file to local router one, and made sure the network in Windows was set to Private and then rebooted and it worked!

[knrchowdary]

fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
ERROR: unable to select packages:
python (no such package):
required by: world[python]

Getting above error while running command from .gilab-ci.yml

FYI command: apk add --no-cache python py-pip bash gawk sed grep bc coreutils

[minecraftchest1]

I took a quick look, and apparently there is a 301 from HTTPS to HTTP.

[minecraftchest1]

I am still seeing this error. I have a docker image that I CANNOT BUILD because of this. If the problem is with the CDN, *USE ANOTHER CDN! * Software users should not be forced to suffer from upstream issues just because upstream won't do anything about it.

[ryukoposting]

@minecraftchest1 Just got the 'temporary error' problem myself:

fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
ERROR: unsatisfiable constraints:
  ...

I tried a few different solutions, including explicitly defining DNS servers in the docker-compose file and in /etc/docker/daemon.json, to no avail.

For me, adding network: host to my docker-compose file fixed the problem:

version: "3.9"
services:
  backend:
    build:
      context: .
      dockerfile: ...
      network: host    <--- this fixed it

[minecraftchest1]

I have tried --network host with my docker build command which still doesn't solve it for me. I don't use docker compose.

[tropicoo]

For Raspberry Pi:

The answer is this: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.13.0#time64_requirements
(alpine 3.13 isn't compatible with raspberry pi os by default due to some outdated package)

[[source](https://github.com/docker/for-linux/issues/1196#issuecomment-774490539)]

For Raspberry Pi, you can upgrade libseccomp2 to a supported version from the Debian Repos eg.

wget http://ftp.de.debian.org/debian/pool/main/libs/libseccomp/libseccomp2_2.5.1-1_armhf.deb
dpkg -i libseccomp2_2.5.1-1_armhf.deb

[[source](https://github.com/docker/for-linux/issues/1196#issuecomment-816600988)]