Docker-alpine: Repository problem?

[Skorpyon]

http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz

Error 503 Connection timed out
Connection timed out
Guru Mediation:
Details: cache-hhn1533-HHN 1494522331 3223824506
Varnish cache server

[JWinkler05]

I am getting the same thing.

Also, to note. This is happening on more than just v3.4. I tried on 3.5 and 3.3 and there was a 503 response from each.

[pkhxcp]

Also having this issue

[QSonx]

Also having this problem. Right when I was tweaking my ci pipeline.

[borourke]

+1 on having this issue

fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/main: temporary error (try again later)

[Zebradil]

Use one of the mirrors:

http://dl-cdn.alpinelinux.org/alpine/
http://nl.alpinelinux.org/alpine/
http://dl-2.alpinelinux.org/alpine/
http://dl-3.alpinelinux.org/alpine/
http://dl-4.alpinelinux.org/alpine/
http://dl-5.alpinelinux.org/alpine/
http://dl-6.alpinelinux.org/alpine/
http://dl-8.alpinelinux.org/alpine/
http://distrib-coffee.ipsl.jussieu.fr/pub/linux/alpine/alpine/
http://mirror.yandex.ru/mirrors/alpine/
http://mirrors.gigenet.com/alpinelinux/
http://repos.lax-noc.com/alpine/
http://repos.dfw.lax-noc.com/alpine/
http://repos.mia.lax-noc.com/alpine/
http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/
http://liskamm.alpinelinux.uk/
http://mirrors.2f30.org/alpine/
http://mirror.leaseweb.com/alpine/
http://repository.fit.cvut.cz/mirrors/alpine/
http://alpine.mirror.far.fi/
http://lasca.ic.unicamp.br/pub/alpine/
http://alpinelinux.c3sl.ufpr.br/

I mean, you could modify your Dockerfile with something like this:

RUN echo http://mirror.yandex.ru/mirrors/alpine/v3.5/main > /etc/apk/repositories; \
    echo http://mirror.yandex.ru/mirrors/alpine/v3.5/community >> /etc/apk/repositories

RUN apk update \
 && apk add --no-cache \
...

[mattatcha]

@andyshinn and I are looking into this.

[mbelang]

Getting the same issue here as well.

@Zebradil You should not have to change mirror like that because one is down...

[Zebradil]

@mbelang What should I do then? Just wait?)
It's effective solution while the main mirror is down.

[mbelang]

Well it depends if it is an urgency I guess, but for me it is one commit that will automatically be deployed and another commit to revert that duct tape that will also be deployed. All of that for no change..

[andyshinn]

The upstream mirror is down. I am updating the configuration to another one. It should be back in the next 5 minutes.

[mbelang]

I confirm it is back online. Thanks a lot

[Zebradil]

@mbelang I agree.

[Skorpyon]

@Zebradil
Thank you, man! You save my ass )))

[QSonx]

All good here too. Thanks for the quick response to this.

[Kapoor-uh]

Hi I am trying to build up container using alpine:3.5, however,i am not able to move forward after these steps
Sending build context to Docker daemon 3.588GB
Step 1/17 : FROM alpine:3.5
---> 074d602a59d7
Step 2/17 : MAINTAINER blabla
---> Using cache
---> e8bafda9c208
Step 3/17 : ENV http_proxy http://10.144.1.10:8080/
---> Using cache
---> b02224fcf625
Step 4/17 : ENV https_proxy http://10.144.1.10:8080/
---> Using cache
---> 6f7ebd10ff06
Step 5/17 : RUN apk add --update git py-virtualenv python-dev py-pip openjdk7-jre-base && pip install Flask && pip install gevent && pip install greenlet && pip install numpy && pip install requests RUN apk --update add gcc make g++ zlib-dev
---> Running in 5230e54c83ae
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz

The URL which docker build is trying to access (http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz) is up. I have tried it couple of times and nothing moves after fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz, even i don't get any error.

[scarothers]

I know this issue is closed, but I'm seeing the same error now:

fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

When I open the URL in my browser, I see this:

Error 503 Backend is unhealthy

Backend is unhealthy

Guru Mediation:

Details: cache-dca17737-DCA 1501279301 2934117002

Varnish cache server

Our build is failing and we can't push out a change that needs to go out :(

[andyshinn]

That is an error from Fastly, the CDN. Can you provide the curl output including the headers?

$ curl -v -s -o /dev/null http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
*   Trying 151.101.48.249...
* TCP_NODELAY set
* Connected to dl-cdn.alpinelinux.org (151.101.48.249) port 80 (#0)
> GET /alpine/v3.6/community/x86_64/APKINDEX.tar.gz HTTP/1.1
> Host: dl-cdn.alpinelinux.org
> User-Agent: curl/7.51.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: application/octet-stream
< Last-Modified: Fri, 14 Jul 2017 22:20:57 GMT
< ETag: "596943c9-5b68a"
< Accept-Ranges: bytes
< Content-Length: 374410
< Accept-Ranges: bytes
< Date: Fri, 28 Jul 2017 22:40:58 GMT
< Via: 1.1 varnish
< Connection: keep-alive
< X-Served-By: cache-dfw1837-DFW
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Timer: S1501281658.867729,VS0,VE296

[btgoodwin]

I'm getting the same error right now for v3.6. I was able to get one update through my CI pipeline a few minutes ago, and now it's dead again.

$ curl -v -s -o /dev/null http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
* About to connect() to dl-cdn.alpinelinux.org port 80 (#0)
*   Trying 151.101.200.249...
* Connected to dl-cdn.alpinelinux.org (151.101.200.249) port 80 (#0)
> GET /alpine/v3.6/community/x86_64/APKINDEX.tar.gz HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dl-cdn.alpinelinux.org
> Accept: */*
> 
< HTTP/1.1 503 Backend is unhealthy
< Server: Varnish
< Retry-After: 0
< Content-Type: text/html; charset=utf-8
< Content-Length: 455
< Accept-Ranges: bytes
< Date: Wed, 02 Aug 2017 16:03:29 GMT
< Via: 1.1 varnish
< Connection: close
< X-Served-By: cache-dca17733-DCA
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Timer: S1501689810.553383,VS0,VE1
< 
{ [data not shown]
* Closing connection 0

[dineshauti]

Its still down. Any updates?

[andyshinn]

Can you also provide the headers output so we can try and correlate what might be going on here? So far, the last couple failures were all in the DCA region. I wonder if Fastly DCA is having some issues. I'll open up a thread with them if that seems to be the common denominator.

[bob798]

I'm also getting the same error now for v3.6 .

WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz: remote server returned error (try 'apk update')
ERROR: unsatisfiable constraints:

[thomas-oo]

Still getting the same error when I do RUN apk add --update curl
[INFO] fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
[INFO] http://dl-cdn.alpinelinux.org/alpine/v3.6/main: temporary error (try again later)
[INFO] WARNING: Ignoring APKINDEX.84815163.tar.gz: No such file or directory

I'm also getting a 301 moved permanently when I curl to http://dl-cdn.alpinelinux.org/alpine/v3.6/main

[aquacode]

Receiving same exact error output as of this moment:

fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.6/main: temporary error (try again later) WARNING: Ignoring APKINDEX.84815163.tar.gz: No such file or directory ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.6/community: temporary error (try again later) WARNING: Ignoring APKINDEX.24d64ab1.tar.gz: No such file or directory OK: 4 MiB in 11 packages fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.6/main: temporary error (try again later) WARNING: Ignoring APKINDEX.84815163.tar.gz: No such file or directory

[MarkTiedemann]

Same error here.

fetch http://mirror.yandex.ru/mirrors/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
ERROR: http://mirror.yandex.ru/mirrors/alpine/v3.6/main: temporary error (try again later)
fetch http://mirror.yandex.ru/mirrors/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
fetch http://mirror.yandex.ru/mirrors/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://mirror.yandex.ru/mirrors/alpine/v3.6/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
ERROR: http://mirror.yandex.ru/mirrors/alpine/v3.6/community: temporary error (try again later)
fetch http://mirror.yandex.ru/mirrors/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://mirror.yandex.ru/mirrors/alpine/v3.6/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

[codevbus]

Getting this as well:

WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

What's strange is that I can do a wget from the host node and it downloads the APKINDEX.tar.gz file perfectly fine.

[cpettifer]

Same / similar problems here. I've tried updating repositories to use explicit mirror too and tried several that all show the same problem such as:

fetch http://repository.fit.cvut.cz/mirrors/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
+ apk update
    fetch http://repository.fit.cvut.cz/mirrors/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
ERROR: http://repository.fit.cvut.cz/mirrors/alpine/v3.6/main/: temporary error (try again later)
WARNING: Ignoring APKINDEX.33685f67.tar.gz: No such file or directory
    ERROR: http://repository.fit.cvut.cz/mirrors/alpine/v3.6/community/: temporary error (try again later)
WARNING: Ignoring APKINDEX.3aec0ff2.tar.gz: No such file or directory`

I can download the files directly on my host computer (running windows) and browse the directories fine in a browser.

Any clues? This is blocking me building and updating containers that are based on Alpine.

[cpettifer]

OK, oddly I was looking through similar issues and issue #155 suggested restarting Docker to resolve the issue. I didn't believe it, but tried it anyway.

It resolved my issue!

[rcorre]

I tried @Zebradil's suggestion to use a different mirror, but no luck:

fetch http://mirror.yandex.ru/mirrors/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
ERROR: http://mirror.yandex.ru/mirrors/alpine/v3.4/main: temporary error (try again later)
WARNING: Ignoring APKINDEX.89669904.tar.gz: No such file or directory

This is only happening when I try to docker build a new image. If I start a shell inside a running image I can apk update just fine.

[bhgames]

I tried the different mirror too. However a shell also causes the issue.

MacBook-Air:opslab_cnc arkavon$ docker run -it 6471729507b9 sh
/app #  apk add --no-cache postgresql-dev git
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz: 
temporary error (try again later)
/app #

Just began all of a sudden.

[jmealo]

I'm having the same issue (starting this morning). I'm probably going to switch to Debian due to the history of this happening for long periods of time.

You should be able to fix this using: DNS (better) or anycast (best).

I haven't used them personally but KeyCDN offers free hosting for Open Source. That should speed things up considerably and give you better reliability than whatever is going on here (as they offer anycast at multiple POPs).

[adriandugan]

@Zebradil response here worked for me this morning. Thanks! It would be good to see this resolved for good.

[npxquynh]

I also get similar issue:

15:54:22 fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
15:54:27 ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/main: temporary error (try again later)
15:54:27 WARNING: Ignoring APKINDEX.167438ca.tar.gz: No such file or directory
15:54:27 fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
15:54:32 ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/community: temporary error (try again later)
15:54:32 WARNING: Ignoring APKINDEX.a2e6dac0.tar.gz: No such file or directory

For me it's really strange because I got error if I do not provide the slash character / at the end of the URL

curl -v -s -o /dev/null http://dl-cdn.alpinelinux.org/alpine/v3.4/main/ # 200 OK
curl -v -s -o /dev/null http://dl-cdn.alpinelinux.org/alpine/v3.4/main # 301 Moved Permanently

The solution of @Zebradil here that @adriandugan mentioned above did not work for me. I would appreciate any input on how to resolve this issue for good.

[andyshinn]

Hmm, what is it trying to redirect to? It is possible this is related to bad upstream server.

[laukotomi]

I had the same issue today. It turned out that the docker bridge network stopped working.

So I created a new bridge network with this command: _docker network create --driver bridge network_name_
And started the docker build with this additional parameter: _--network network_name_

I guess that parameter can also be used when starting a docker container.

[rlewkowicz]

This is still not fixed. I'm also having this issue.

[Grinzzly]

Yeah, not fixed yet. Waiting for solution.

[xelex]

I can confirm the problem.

[peternormann]

I had this issue as well this morning. I am running Docker CE on Windows 10 Pro.

I resolved it by changing the TCP IPv4 adapter settings on vEthernet (DockerNAT) to point to Google's DNS servers (8.8.8.8 and 8.8.4.4) and then restarting Docker.

[marcstreeter]

I have this issue too - seems as if it's a recurring issue from #264

[davisn9]

Got the same issue today.

[ejlp12]

Experiencing the same issue and was trying several alternative mirrors mentioned by @Zebradil but did not work. Curl/wget to the same URL from the host is working fine.

I found that it works on different machine (same OS version RHEL 7 and same Docker version 1.12.6, build 85d7426/1.12.6). Strange...

[davisn9]

Thanks for the info.

[gabrielmirandat]

Do what @laukotomi suggested works here

[sibil]

It is happening again.

[ss@webdev ~]# wget http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
--2017-12-27 15:39:24-- http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
Resolving dl-cdn.alpinelinux.org (dl-cdn.alpinelinux.org)... 151.101.0.249, 151.101.64.249, 151.101.128.249, ...
Connecting to dl-cdn.alpinelinux.org (dl-cdn.alpinelinux.org)|151.101.0.249|:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2017-12-27 15:39:25 ERROR 503: Service Unavailable.

[kotekaman]

WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

stilll get error like this

[nicbou]

Currently, http://dl-4.alpinelinux.org/alpine/3.6/community/x86_64/APKINDEX.tar.gz does not work either. Why are these repository URLs suddenly breaking? It's breaking docker's promise of reproducible builds.

EDIT: v3.6 instead of 3.6 fixed the URL for me. It used to work, but it doesn't anymore. Perhaps it's a different issue, but I'm leaving this here in case it helps someone else.

[kvz]

Same here, have to retry building 5 times and then it works. Each time it errors out on a different package like:

ERROR: make-4.1-r1: temporary error (try again later)
(2/22) Installing binutils-libs (2.26-r1)
ERROR: binutils-libs-2.26-r1: temporary error (try again later)

Sites go down all the time, that's fine, but is there a way to tell apk to retry itself a few times?

[nicbou]

@kvz, I think this is a separate problem. URLs should be semi-permanent (or at least return HTTP 301/302 when they change) to maintain reproducible builds, but the stability of third-party repositories is not an issue this project and its maintainers can tackle.

[froodley]

I have been getting these errors intermittently all week

[javacraft]

For what it is worth; this issue also arises when DNS cannot resolve repository URLs. Our users experience the same ERROR ... temporary error... because: (1) /etc/resolv.conf has incorrect or no nameserver entries; (2) upstream DNS server is down; and (3), believe it or not, their network cable is unplugged; the latter always deserving a /facepalm :)

[christhomas]

I am still getting this error, I don't think I should manually configure this, it should be resolved on the server side with no changes my side

[shaider0802]

I was having this issue on Windows 10, resolved it by following peternormann 's suggestion to point the DNS settings of DockerNAT to google's..

[wisprp]

fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.5/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

Worked fine 15 min ago, but now I'm getting this error.

[KristopherWindsor]

I am getting this error now.
http://dl-cdn.alpinelinux.org/alpine/v3.6/main says:

Error 503 Backend is unhealthy
Backend is unhealthy

Guru Mediation:
Details: cache-sjc3146-SJC 1518737504 1886963746

Varnish cache server

[wisprp]

Back online.

[Francesco149]

changing /etc/resolv.conf to point to google DNSes fixed the issue for me (gentoo linux, docker 17.09.1-ce)

[dwaiba]

3.7 mirrors are down again

[barthwalsamarth]

Getting this error
/ # apk add vim WARNING: Ignoring APKINDEX.70c88391.tar.gz: No such file or directory WARNING: Ignoring APKINDEX.5022a8a2.tar.gz: No such file or directory ERROR: unsatisfiable constraints: vim (missing): required by: world[vim]
How can I fix it?

ok found the solution

echo http://dl-cdn.alpinelinux.org/alpine/edge/testing >> /etc/apk/repositories
apk update
apk add vim
this worked

[roffe]

Been seeing this a couple of times over the past days making my automatic builds break :(

[fr33ek]

Same now.

[ChrisChiasson]

Getting same error as everyone else in docker. I'm on google compute engine in a VM on Ubuntu 16.04 LTS.
In Chrome from my workstation: http://dl-cdn.alpinelinux.org/alpine/v3.7/main
Error 503 Backend is unhealthy
Backend is unhealthy

Guru Mediation:
Details: cache-mia17625-MIA 1521467300 1442441019

Varnish cache server

[ghost]

Same here, now

[dewey]

The mirror of the latest mirrors is up in case anyone wants to switch over:

http://mirror.yandex.ru/mirrors/alpine/MIRRORS.txt

[ChrisChiasson]

As requested:

<user>@<machine>:~/mingw-w64-clang$ curl -vvvv http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 151.101.0.249...
* Connected to dl-cdn.alpinelinux.org (151.101.0.249) port 80 (#0)
> GET /alpine/v3.7/main/x86_64/APKINDEX.tar.gz HTTP/1.1
> Host: dl-cdn.alpinelinux.org
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 503 Backend is unhealthy
< Server: Varnish
< Retry-After: 0
< Content-Type: text/html; charset=utf-8
< Content-Length: 455
< Accept-Ranges: bytes
< Date: Mon, 19 Mar 2018 13:51:15 GMT
< Via: 1.1 varnish
< Connection: close
< X-Served-By: cache-mdw17322-MDW
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Timer: S1521467475.266664,VS0,VE0
< 
{ [455 bytes data]
100   455  100   455    0     0   8315      0 --:--:-- --:--:-- --:--:--  8425

[dewey]

It's back up for me now!

[ChrisChiasson]

The main started working again.

[lorintotahh]

same here :(

[wstrange]

http://dl-cdn.alpinelinux.org/alpine/v3.7/main is down again

[JPWKU]

I would like to report that we are also seeing timeouts updating against dl-cdn.alpinelinux.org.
I have sidestepped the issue by adding references to one of the mirrors in our Dockerfile.

[JamesKyburz]

Mirror for 3.6 seems to be done at the moment :(

http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

[zfeng8]

Any update on this? I have used a mac computer to do the work, it works fine. However, every time I switch to an Ubuntu system, the problem occurs.

[Necmttn]

looks like down again ?

[JoaoCarabetta]

I am still getting this error in Ubuntu. As @zfeng8 mentioned, it works fine on mac but fails on an Ubuntu. I also tried a bunch of mirrors, slashes and restarting the docker daemon.

Does anyone know how to fix it?

[zfeng8]

@JoaoCarabetta if you have a mac computer, try to build the image and somehow move to the ubuntu. Then you should be fine

[johnkoehn]

Getting this error currently :(

[LuckyTyagi]

I'm facing 'timed out' error on this. Following Kubernetes up and running book.

# make build
container: kuard-build
Sending build context to Docker daemon  11.23MB
Step 1/10 : FROM golang:1.7-alpine
 ---> 974aa102bae2
Step 2/10 : WORKDIR /data
 ---> Using cache
 ---> 03e6eee40878
Step 3/10 : ENV GOPATH /data/go
 ---> Using cache
 ---> a4263084a348
Step 4/10 : ENV npm_config_cache /data/npm_cache
 ---> Using cache
 ---> 1d1067eb3a33
Step 5/10 : ARG ALL_ARCH
 ---> Using cache
 ---> c5ed0dff6d95
Step 6/10 : ENV ALL_ARCH ${ALL_ARCH}
 ---> Using cache
 ---> 50872afbf603
Step 7/10 : RUN for ARCH in ${ALL_ARCH}; do       ln -s -f "/data/std/${ARCH}" "/usr/local/go/pkg/linux_${ARCH}_static" ;     done
 ---> Using cache
 ---> 98b54c180d31
Step 8/10 : RUN apk update && apk upgrade && apk add --no-cache git nodejs bash
 ---> Running in 549a74a0a473
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/main: operation timed out
WARNING: Ignoring APKINDEX.167438ca.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/community: operation timed out
WARNING: Ignoring APKINDEX.a2e6dac0.tar.gz: No such file or directory
2 errors; 12 distinct packages available
The command '/bin/sh -c apk update && apk upgrade && apk add --no-cache git nodejs bash' returned a non-zero code: 2
make: *** [.kuard-build-container] Error 2
[root@rsvm63 kuard]#

[ncopa]

@LuckyTyagi I can not reproduce it. Can you please tell me what you get from:
sh curl --head http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz

[LuckyTyagi]

@ncopa It goes like this:

# curl --head  http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
HTTP/1.1 200 OK
Server: nginx
Content-Type: application/octet-stream
Last-Modified: Mon, 30 Apr 2018 18:50:38 GMT
ETag: "5ae7657e-a5b18"
Via: 1.1 varnish
Accept-Ranges: bytes
Date: Mon, 07 May 2018 13:01:59 GMT
Via: 1.1 varnish, 1.1 mc-loc-web-gw.alcatel-lucent.com
Age: 13700
X-Served-By: cache-jfk8145-JFK, cache-atl6228-ATL
X-Cache: HIT, HIT
X-Cache-Hits: 1, 1
X-Timer: S1525698119.381417,VS0,VE22
Content-Length: 678680

[neclimdul]

I was just having this problem repeatedly over the last couple hours. I fired up wireshark to see if there was something to be seen and it started working. schroedinbug

Because I didn't confirm it I don't know that its the same problem but I've seen a similar problem with fastly mirrors quite often before though. Sometimes its alpine, sometimes its npmjs.org. Wireshark shows the TCP session stuck with fastly sending packets sized very closed to 1500 MTU with a no fragment flag and the linux bridge failing to route because of a lower MTU. Repeat until it eventually dies.

Its especially easy to recreate in k8s because k8s sets a lower MTU on the bridge to to allow for routing overlay network traffic and I can consistently create the problem using DinD to build containers in k8s.

The log looks something like this:

... misc traffic 
ack:                   container -> node -> fastly
Transmission:          container  X node <- fastly (2924 no fragment)
ICMP unreachable:                   node -> fastly
Transmission:          container  X node <- fastly (1496 no framgment)
ICMP unreachable:                   node -> fastly
Transmission:          container  X node <- fastly (1496 no framgment)
.... repeat

I have a saved wireshark packet logs of this happening on a k8s node though its talking to npm's fastly mirror because I was using the workaround to bypass the alpine fastly mirrors. Its the same behavior though.

[LuckyTyagi]

Strange that nobody was able to reproduce the issue or provide the solution even after these many days..!!

[neclimdul]

there are half a dozen similar issues so i think people are reproducing it just not the people that can fix it.

[ncopa]

@neclimdul thank you for sharing that wireshark info which explains what is going on. It looks like it is PMTU that is broken. My guess is that the icmp packet with DF set if filtered by something along the way. It could be your ISP or it could be fastly. In either case, this is not something can can be fixed by changing anything in the alpine linux docker image.

[ncopa]

I was able to reproduce it. I ran alpine in a docker container on my linux host. I manually set mtu on the veth interface to 1450, and apk update just hangs. Switching repository to dl-4.alpinelinux.org makes it work, and it also sends ICMP needs frag. So it looks like fastly is filtering out ICMP need frag packets and break PMTU by doing so.

Again, this is not something Alpine specific and I am pretty sure this affects anything using fastly and should be reported to them.

[neclimdul]

Great, that matches what I've been seeing and yes it does affect other things as the log I've got stored is actually the npm js mirror on fastly. Someone connected to alpine's infrastructure might have better contacts to work with Fastly but lacking a better method I shot them a tweet for now.

https://twitter.com/neclimdul/status/994274338072248320

[ncopa]

I sent them a support mail, but they claim nothing is wrong with their network. Then they recommended the users to set the MTU manually. A better workaround would be to configure tcp mss clamping.

[neclimdul]

The MTU is manually set...

[neclimdul]

FWIW, opened the wireshark documented earlier. Checked the SYN request for the session, MSS set to 1460. so... not sure watch options there are.

[JoFAM]

I'm having the issue again with http://nl.alpinelinux.org and it suddenly started even though my dockers built perfectly fine this morning. The warnings about temporary errors point to the main repo dl-cdn.alpinelinux.org . Changing the repo to any other mirror gives the exact same errors.

The problem was solved after restarting Docker, but the problem is specific to alpine. Debian-based containers didn't have any problem.

Here's (part of) the docker output:

Step 4/6 : RUN echo '@edge http://nl.alpinelinux.org/alpine/edge/community' >> /etc/apk/repositories
      && echo '@edge http://nl.alpinelinux.org/alpine/edge/main' >> etc/apk/repositories
      && apk add --upgrade --no-cache apk-tools@edge
 ---> Running in 0b9d70ecba5a
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://nl.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://nl.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://nl.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
WARNING: Ignoring http://nl.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
WARNING: The repository tag for world dependency 'apk-tools@edge' does not exist
ERROR: Not committing changes due to missing repository tags. Use --force-broken-world to override.

[zment]

Down again :(
With mirrors, alpine is pullable, but cannot apk add openldap

fetch http://dl-2.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
ERROR: unsatisfiable constraints:
openldap (missing):
required by: world[openldap]

[reeshabh90]

I am facing the issue with alpine v3.4. Any solutions?

[reeshabh90]

My Solutions: Configure Docker DNS on UBUNTU, it will fix the problem.

Refer the link below for reference:

https://development.robinwinslow.uk/2016/06/23/fix-docker-networking-dns/

[publicocean0]

same problem

[publicocean0]

in my case no solution works (reboot or dns config in daemon.json)

[neclimdul]

I'm guessing the DNS fixes some people are getting shake up the IP for the fastly mirror being used and maybe get a different path avoiding routers causing the ICMP issue. Using a global DNS versus a local one can really mess up CDNs localization so that's a very real possibility.

@publicocean0 a couple people have reached out to fastly now about this and they don't seem to be acknowledging the problem. Maybe reach out as well? I wish migrating off alpine was even an option but since I see the same problem with node and other fastly hosted projects I'm just :confounded:

[LugaMuga]

I found that docker alpine in Docker cannot resolve repo url when I've added iptables rule:

sudo iptables -t nat -F
sudo iptables -t mangle -F
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

And fix this issue can only remove all rules from iptables and restat docker:

sudo iptables -F && sudo iptables -X && sudo iptables -S
sudo service docker restart

[knaggit]

For everyone who's still facing that problem. It seems, there are DNS probs in Alpine DNS records. It affects somehow other repositories as well. I tried other repos and still had the problem. On which URL everything seems fine is, when you RUN sed before getting packages as below:

RUN sed -i -e 's/dl-cdn/dl-4/' /etc/apk/repositories && \
    apk add --no-cache \
        bash \
        build-base \

I read all the threads and finally that got me help. Good luck!

[igwejk]

@andyshinn This issue has come up again, is there anything you can do about it?

fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.8/main: IO ERROR
WARNING: Ignoring APKINDEX.adfa7ceb.tar.gz: No such file or directory
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.8/community: IO ERROR
WARNING: Ignoring APKINDEX.efaa1f73.tar.gz: No such file or directory
2 errors; 13 distinct packages available
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.8/main: IO ERROR
WARNING: Ignoring APKINDEX.adfa7ceb.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.8/community: IO ERROR
WARNING: Ignoring APKINDEX.efaa1f73.tar.gz: No such file or directory

[igwejk]

Can you recommend a way to ensure this issue doesn't occur frequently?

[us108y]

I am also getting the same Error as of today (Aug -09-2018)

FROM alpine

fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)

`curl -v -s -o /dev/null http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz

• Trying 151.101.48.249...
• TCP_NODELAY set
• Connected to dl-cdn.alpinelinux.org (151.101.48.249) port 80 (#0)

GET /alpine/v3.6/community/x86_64/APKINDEX.tar.gz HTTP/1.1
Host: dl-cdn.alpinelinux.org
User-Agent: curl/7.58.0
Accept: /

< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: application/octet-stream
< Last-Modified: Tue, 24 Jul 2018 15:26:34 GMT
< ETag: "5b57452a-5b5d7"
< Via: 1.1 varnish
< Content-Length: 374231
< Accept-Ranges: bytes
< Date: Thu, 09 Aug 2018 18:35:17 GMT
< Via: 1.1 varnish
< Age: 1
< Connection: keep-alive
< X-Served-By: cache-jfk8123-JFK, cache-dfw18646-DFW
< X-Cache: HIT, HIT
< X-Cache-Hits: 1, 1
< X-Timer: S1533839718.760208,VS0,VE38
<
{ [1260 bytes data]

• Connection #0 to host dl-cdn.alpinelinux.org left intact`

while other cdn works

[T0T4R4]

Same issue on ubuntu server 18.04 with
Docker version 18.06.0-ce, build 0ffa825

Surprisingly, It worked on my fedora 27 workstation with
Docker version 1.13.1, build 30c1ca9-unsupported

The error message is :

---> cc8b89f74863
Step 5/34 : RUN     apk add --update --no-cache nginx nodejs nodejs-npm git curl wget gcc ca-certificates                                     python-dev py-pip musl-dev libffi-dev cairo supervisor bash                                     py-pyldap py-rrd                                                                 
&&        wget -q -O /etc/apk/keys/sgerrand.rsa.pub   https://raw.githubusercontent.com/sgerrand/alpine-pkg-glibc/master/sgerrand.rsa.pub              
&&        wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.26-r0/glibc-2.26-r0.apk                &&        apk add --no-cache  glibc-2.26-r0.apk                                                                        
&&        rm glibc-2.26-r0.apk                                                                                         
&&        adduser -D -u 1000 -g 'www' www                                                                              
&&        pip install -U pip pytz gunicorn six --no-cache-dir                                                          
&&  npm config set proxy http://proxy.usq.edu.au:8080                                
&&  npm config set https-proxy http://proxy.usq.edu.au:8080                              
&&        npm install -g wizzy                                                                                         
&&        npm cache clean --force

 ---> Running in fb046f39414f
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz: temporary error (try again later)
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz: temporary error (try again later)
ERROR: unsatisfiable constraints:
  bash (missing):
    required by: world[bash]
  ca-certificates (missing):
    required by: world[ca-certificates]
  cairo (missing):
    required by: world[cairo]
  curl (missing):
    required by: world[curl]

[T0T4R4]

Found the issue. DNS resolution was broken within the container because of the host configuration. Host is Ubuntu 18.04. So I had to do the following :

Ubuntu 18.04 changed to use systemd-resolved to generate /etc/resolv.conf. Now by default it uses a local DNS cache 127.0.0.53. That will not work inside a container, so Docker will default to Google's 8.8.8.8 DNS server, which may break for people behind a firewall.

/etc/resolv.conf is actually a symlink (ls -l /etc/resolv.conf) which points to /run/systemd/resolve/stub-resolv.conf (127.0.0.53) by default in Ubuntu 18.04.

Just change the symlink to point to /run/systemd/resolve/resolv.conf, which lists the real DNS servers:
sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Verify on the host: cat /etc/resolv.conf

Credits: https://serverfault.com/questions/642981/docker-containers-cant-resolve-dns-on-ubuntu-14-04-desktop-host (see post from wisbucky on 27/07/2018 at 23:06)

[JoeUX]

Hi all, I'm having the same issue as @igwejk and others, with ultimately the "no such file or directory" errors. All I'm doing is running apk update after putting the main, community, and edge mirrors in the repositories file.

One problem with configuring DNS as per the instructions is that the /etc/resolv.conf file does not exist in Alpine 3.8 as far as I can tell. I'm using the Alpine "virt" ISO. Maybe it exists in other ISOs, like Standard? I don't understand why the instructions refer to a file that doesn't exist.

[smilemakc]

@T0T4R4 Thank you!

[ahawkins]

I'm having this problem on OSX. The only solution was to change the mirror. Also, this problem seems to be network specific since images built fine on CI but not my local machine.

git:fix/docker-build* >> docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     false

[johndatserakis]

Had this issue on one machine but not the other.

The one that worked was using the cloudflare 1.1.1.1 DNS. I set the non-working one to also use the cloudflare DNS, and it started working too.

Try having your computer use that DNS if you have this type of issue.

[montao]

I had a similar error running docker-alpine from Ubuntu 18.04 (in a Vagrant box). After running the following command in the Ubuntu 18.04, things started to work:

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

[Pterosaur]

I had a similar error running docker-alpine from Ubuntu 18.04 (in a Vagrant box). After running the following command in the Ubuntu 18.04, things started to work:

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

@montao Thanks for your answer, it solve my problem at Ubuntu 18.04.

[syedsaadqamar]

It's DNS issue you can solve it by using the following steps:

1) Find your DNS ip using below command
nmcli dev show | grep 'IP4.DNS'

IP4.DNS[1]: 192.168.0.1

2) Create a file in the desktop which name is deamon.json and file has below data:
{
"dns": ["192.168.0.1", "8.8.8.8"]
}

note: replace that ip "192.168.0.1" to your DNS ip

3) Go that directory /etc/docker/ in your system and paste file deamon.json that you just created

4) Then restart your docker/your system, it will solve your issue.

[HasanAhmadQ7]

running networksetup -getdnsservers returned that I have no dns IPs (using mac).

I ran networksetup -setdnsservers Wi-Fi 8.8.8.8

and included it in the daemon json file. it did not work.

[ncopa]

This issue is closed. Please open a new issue if there is is a problem with alpine docker image.

[tekintian]

DNS问题， 重新设置一下宿主机的DNS为 8.8.8.8 8.8.4.4 ， 然后重启docker 解决问题

[bbaassssiiee]

I run docker 1.13 on centos7 in a VirtualBox on Mac. No DNS problems on the Mac, no DNS problems on the centos7 guest, but running 'apk update' in a container in the guest showd this problem. I get a DNS from DHCP, but the network setup of docker in the guest was the cause

After removing /etc/docker/daemon.json the build works again.

[tekintian]

setup dns with static , 8.8.8.8 8.8.4.4

[Mihailoff]

Well explained solution that worked for me https://development.robinwinslow.uk/2016/06/23/fix-docker-networking-dns/

[cmaliwal]

facing the same issue :

Step 3/11 : RUN apk update
 ---> Running in 7a001721b812
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz

[jamestenglish]

Getting the same issue, I am running in Jenkins on an EKS cluster that I don't have the ability to change the DNS on

[sepastian]

I could also download APKINDEX.tar.gz on the host, but not when building the image.

In my case, the problem was a VPN connection, which modified /etc/resolv.conf, which somehow prevented Docker from updating apk. Closing the VPN connection solved the issue.

[JostCrow]

For me a restart of docker fixed the problem.

sudo systemctl restart docker

[caoyj1991]

solution 1: check your environment
docker ->dns (/etc/default/docker)
solution 2: update the repository URL

[NuKESec]

I'm getting this issue in 3.10. like what the actual fuck devs? not just broken mirrors (all of term are broken 100% of the time), but broken EVERYTHING. Alpine had hopes of being great, but turned into shit. the devs don't know what they're doing apparently

[sea-erkin]

If you're using custom iptables, you may run into this issue and have to adjust your rules to allow this traffic to route.

[MyZeD]

I had this problem, sometime and how you tell with combination on alpine.
The main-problem here is, that ipv6 is prefer activated (before ipv4) and dl-cdn.alpinelinux.org
# dig dl-cdn.alpinelinux.org
has no AAAA-Record. So it cant resolve the request.
You need to force IPv4 to download all ressources.

// Or Sometimes the Client (with) Docker cant connect through Ipv6

Had the same problem with some SMTP-Server.

[marceloavf]

@MyZeD How to force IPv4?

[MyZeD]

@marceloavf i took this: https://askubuntu.com/questions/32298/prefer-a-ipv4-dns-lookups-before-aaaaipv6-lookups