Flux: Flux images from Quay.io being unavailable

[tycoles]

I'm currently trying to reinstall flux via helm and I'm getting this error when the pods are attempting to pull the images:

Warning Failed 1m (x3 over 2m) kubelet, Failed to pull image "quay.io/weaveworks/flux:1.12.0": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/weaveworks/flux/manifests/1.12.0: unknown: Namespace weaveworks has been disabled. Please contact a system administrator.

It happens when I attempt a docker pull quay.io/weaveworks/flux:1.12.0 as well, but I'm not sure if there's an authentication mechanism I'm unaware of.

Updated on 2019-04-23 by @stefanprodan

We've recovered our images from Quay and all sem ver releases are now on Docker Hub.

Flux releases: https://hub.docker.com/r/weaveworks/flux/tags
Flux prereleases (master builds): https://hub.docker.com/r/weaveworks/flux-prerelease/tags

Helm Operator releases: https://hub.docker.com/r/weaveworks/helm-operator/tags
Helm Operator prereleases (master builds): https://hub.docker.com/r/weaveworks/helm-operator-prerelease/tags

[arturo-c]

Had same issue, looks like its a quay.io issue http://status.quay.io/incidents/4lb8mr3fmpqg

[tycoles]

Good lookin out, @arturo-c!

[stefanprodan]

Flux 1.12 can be pulled from Docker Hub weaveworks/flux:1.12.0. To use it with Helm when installing do --set image.repository=weaveworks/flux.

PS. We'll get the operator on Docker Hub soon. Thanks

[alexanderbuhler]

A little different on this side, I was just wondering since I can get a decent response in my browser from this endpoint.

On top, I don't even understand why he's monitoring the flux repo since its a helm chart which is not automated anyway?

ts=2019-04-16T17:47:16.55232456Z caller=images.go:18 component=sync-loop msg="polling images"
ts=2019-04-16T17:47:19.574153741Z caller=warming.go:192 component=warmer canonical_name=quay.io/weaveworks/flux auth={map[]} err="requesting tags: Get https://quay.io/v2/weaveworks/flux/tags/list: unknown: Namespace weaveworks has been disabled. Please contact a system administrator."

[hiddeco]

On top, I don't even understand why he's monitoring the flux repo since its a helm chart which is not automated anyway?

Flux monitors all images in your cluster, not just the ones that are automated.

[stefanprodan]

Flux Helm Operator is on Docker Hub now at weaveworks/helm-operator:0.8.0.

To use it with Helm when installing do --set helmOperator.repository=weaveworks/helm-operator.

[dvelitchkov]

Any update on this?
The quay.io website says the incident is resolved, but:

$ docker pull quay.io/weaveworks/flux:1.12.0
Error response from daemon: Get https://quay.io/v2/weaveworks/flux/manifests/1.12.0: unknown: Namespace weaveworks has been disabled. Please contact a system administrator.
$ 

[hiddeco]

We moved our images to Docker Hub, see Stefan's comments above.

[benhartley]

It would be good to get some more context here - is the move permanent? If so - why?

[hiddeco]

Sorry about the radio silence and my very short answer earlier about our sudden move to Docker Hub.

Quay.io currently isn't working for us, until this has been resolved the move will be semi-permanent. We are working very hard to get all the older releases on Docker Hub and although I can't give you all an exact time frame, I promise they will be available as soon as possible and that we will take measures so this will not happen again.

Last, my apologies for the abrupt change and any inconvenience it may (have) cause(d), I am well aware this may not be the quality of support you are used to from us.

Will keep you posted :postbox: :rocket:

[isen-ng]

Switching to 0.9.1 version of the charts fixed this.

[hiddeco]

Hiya folks,

We just pushed a first batch of >=1.8.0 tags of weaveworks/flux to Docker Hub, please take a look at the listed tags to see the ones that are available right now.

We also pushed the 0.4.0 release of weaveworks/helm-operator for people still running <0.5.0 versions of the operator so you will not have to take the upgrade path due to the major change in 0.5.0.

We are (still) working hard to get all other tags available, thanks for your patience :tulip:

[gtseres]

Can we also have 0.7.1 in Dockerhub? Thanks!

[hiddeco]

Can we also have 0.7.1 in Dockerhub?

It will be available eventually but I am afraid I can not give you an exact time.

In case you really need it and have good reasons not to upgrade to a newer version, as I would highly recommend this (the upgrade should be quite easy), I would be able to provide you with a rebuild version.

[gtseres]

The problem is that we have to test the new version in staging first, which can take a couple of days. On the other hand, given that we run 0.7.1 in production, what will happen if GKE tries to evict the Flux pods from the node? Won't it block us altogether from having Flux up in the new node, given that the image cannot be downloaded?

[hiddeco]

what will happen if GKE tries to evict the Flux pods from the node?

You will run into much bigger problems.

I will have the rebuild ready for you in approximately one hour.

Update: weaveworks/helm-operator:0.7.1-rebuild is available since 16:35 UTC.

[gtseres]

Thanks a lot! Is it possible to have the Flux 1.11.1 version as well?

[stealthybox]

@gtseres, I've pushed some builds to:

stealthybox/flux:1.11.1
stealthybox/helm-operator:1.11.1  # <-- don't use this tag, it's wrong

Here's the script I used to sandbox, checkout, build, and publish:

#!/bin/bash
# Usage:
#   repo=${USER} tag=1.11.1 ./publish-flux.sh
#
# See https://github.com/weaveworks/flux/issues/1948#issuecomment-485405828

tag="${tag:-1.11.1}"
repo="${repo:-stealthybox}"

set -euo pipefail


info() {
  echo -e "\e[32m"
  echo -e $@ "\e[0m"
}


info "Creating a fresh GOPATH ..."
mkdir -p "tmpgo/{bin,src,pkg}"
GOPATH="${PWD}/tmpgo"
curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh

info "Checking out flux @ ${tag} ..."
go get github.com/weaveworks/flux
cd "${GOPATH}/src/github.com/weaveworks/flux"
git checkout "${tag}"

info "Running dep ensure --vendor-only ..."
rm -rf ./vendor
"${GOPATH}/bin/dep" ensure --vendor-only

info "Building flux ..."
make clean
make

info "Pushing images ..."
tag="$(./docker/image-tag)"
for image in flux helm-operator; do
  docker tag "quay.io/weaveworks/${image}:${tag}" "${repo}/${image}:${tag}"
  docker push "${repo}/${image}:${tag}"
done

info "Finished :)"

Note you'll need to auth to your target container registry, or the push will fail.
(docker login / gcloud auth configure-docker / etc)

[stealthybox]

~note that weaveworks/flux:1.11.1 already exists~:
https://hub.docker.com/r/weaveworks/flux/tags

correction 1.11.0 exists, but not 1.11.1

[stealthybox]

You can checkout a tag such as helm-0.7.x to build a specific helm-operator.

[stealthybox]

I just updated the above script to:

• clear the vendor directory on every run
• use dep ensure --vendor-only
• run make clean before building

I'm now hard-resetting to each ref and publishing to my personal dockerhub.

[stealthybox]

The following tags are up on my personal registry.
These new images are all fresh builds (using dep with Gopkg.lock).

We are still working with Quay to recover the original images.

https://hub.docker.com/r/stealthybox/flux/tags

# stealthybox/flux
1.12.0
1.11.1
1.11.0
1.10.1
1.10.0
1.9.0
1.8.2
1.8.1
1.8.0
1.7.1
1.7.0
1.6.0
1.5.0
1.4.2
1.4.1
1.4.0
1.3.1
1.3.0
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.0

https://hub.docker.com/r/stealthybox/helm-operator/tags

# stealthybox/helm-operator
0.8.0
0.7.1
0.7.0
0.6.0
0.5.3
0.5.2
0.5.1
0.5.0
0.4.0
0.3.0
0.2.1
0.2.0
0.1.1-alpha
0.1.0-alpha

# stealthybox/helm-operator  (accidental -- don't use)
1.11.1

[stealthybox]

I did not build flux for the following tags.
They're very old and require gvt instead of dep:

0.1.0
0.2.0
0.3.0
1.0.0
1.0.0-beta
1.0.1
1.0.2
master-0d109dd
master-6cc08e4
master-ccb9a99
pre-split

[stefanprodan]

We've recovered our images from Quay and all sem ver releases are now on Docker Hub.

Flux releases: https://hub.docker.com/r/weaveworks/flux/tags
Flux prereleases (master builds): https://hub.docker.com/r/weaveworks/flux-prerelease/tags

Helm Operator releases: https://hub.docker.com/r/weaveworks/helm-operator/tags
Helm Operator prereleases (master builds): https://hub.docker.com/r/weaveworks/helm-operator-prerelease/tags

[runningman84]

@stefanprodan the helm operator release docker hub links are returning a 404 error

[stefanprodan]

@runningman84 fixed! there was a typo sorry