Onpremise: ./install.sh fails: 404 Client Error: Not Found

[killarny]

A number of images fail to download when running the install script (using a fresh clone of this repo, as of dffd597 on Jan 14):

• snuba-cleanup-onpremise-local:latest
• sentry-onpremise-local:latest
• symbolicator-cleanup-onpremise-local:latest

All of these fail with 404 Client Error: Not Found ("pull access denied for XXXXXXXXXXXX, repository does not exist or may require 'docker login'")

I did see some mention of a similar problem in #278, but the solutions listed there did not fix this iteration of the issue. (Also, I am not doing an upgrade, this is a fresh installation. Also, also, I am using dffd597 which includes the fix for that issue.)

Here is my log:

Checking minimum requirements...
Removing network onpremise_default
Network onpremise_default not found.
Removing network sentryonpremise_default
Network sentryonpremise_default not found.

Creating volumes for persistent storage...
Created sentry-data.
Created sentry-postgres.
Created sentry-redis.
Created sentry-zookeeper.
Created sentry-kafka.
Created sentry-clickhouse.
Created sentry-symbolicator.

Creating sentry/sentry.conf.py...
sentry/config.yml already exists, skipped creation.
Creating sentry/requirements.txt...

Generating secret key...
Secret key written to sentry/config.yml

Building and tagging Docker images...

Pulling redis (redis:5.0-alpine)...
5.0-alpine: Pulling from library/redis
Digest: sha256:a4e0b7bff7ecec0dc0be95d185d6c99323a92a51065d9563a5bafbc1cf6b3497
Status: Downloaded newer image for redis:5.0-alpine
Pulling postgres (postgres:9.6)...
9.6: Pulling from library/postgres
Digest: sha256:b5f6e8a1369ec3ce48c8419c65dffa083c3eb4a507cb8c75b50bdf3480588a07
Status: Downloaded newer image for postgres:9.6
Pulling smtp (tianon/exim4:latest)...
latest: Pulling from tianon/exim4
Digest: sha256:b23482ddb3584af9f14fc207ae167181ee4d2bb44ae328250219e277d4fcc452
Status: Downloaded newer image for tianon/exim4:latest
Pulling zookeeper (confluentinc/cp-zookeeper:5.1.2)...
5.1.2: Pulling from confluentinc/cp-zookeeper
Digest: sha256:c63871c17b038e1685bf708e7bc2d32ae518f40137c86a6148fbdcdd1c78fe83
Status: Downloaded newer image for confluentinc/cp-zookeeper:5.1.2
Pulling kafka (confluentinc/cp-kafka:5.1.2)...
5.1.2: Pulling from confluentinc/cp-kafka
Digest: sha256:c5ff41b494329e9dea0f2d2d942a391a8d5968f597e9ba4a10846f511ba85220
Status: Downloaded newer image for confluentinc/cp-kafka:5.1.2
Pulling memcached (memcached:1.5-alpine)...
1.5-alpine: Pulling from library/memcached
Digest: sha256:3cfb2eee0b618722a62f4cc907fa0ede848efa87d773bb9384044f490926e7ce
Status: Downloaded newer image for memcached:1.5-alpine
Pulling symbolicator (getsentry/symbolicator:latest)...
latest: Pulling from getsentry/symbolicator
Digest: sha256:a60e8262766cd94a1c0d5463c3e02df57271ee5bf1c94e1853c5b6f7254fa98e
Status: Downloaded newer image for getsentry/symbolicator:latest
Pulling clickhouse (yandex/clickhouse-server:19.4)...
19.4: Pulling from yandex/clickhouse-server
Digest: sha256:a83b98bc73ee7ba1b7e5dedcff84990f8d903716cb0ea352fddb8ec51678713d
Status: Downloaded newer image for yandex/clickhouse-server:19.4
Pulling snuba-api (getsentry/snuba:latest)...
latest: Pulling from getsentry/snuba
Digest: sha256:ab8177e8be8bc54e78cfb5d9957684e784d16af53f13e9f6f2f139578035c033
Status: Downloaded newer image for getsentry/snuba:latest
Pulling snuba-replacer (getsentry/snuba:latest)...
latest: Pulling from getsentry/snuba
Digest: sha256:ab8177e8be8bc54e78cfb5d9957684e784d16af53f13e9f6f2f139578035c033
Status: Image is up to date for getsentry/snuba:latest
Pulling snuba-cleanup (snuba-cleanup-onpremise-local:latest)...
404 Client Error: Not Found ("pull access denied for snuba-cleanup-onpremise-local, repository does not exist or may require 'docker login'")
Pulling snuba-consumer (getsentry/snuba:latest)...
latest: Pulling from getsentry/snuba
Digest: sha256:ab8177e8be8bc54e78cfb5d9957684e784d16af53f13e9f6f2f139578035c033
Status: Image is up to date for getsentry/snuba:latest
Pulling post-process-forwarder (sentry-onpremise-local:latest)...
404 Client Error: Not Found ("pull access denied for sentry-onpremise-local, repository does not exist or may require 'docker login'")
Pulling worker (sentry-onpremise-local:latest)...
404 Client Error: Not Found ("pull access denied for sentry-onpremise-local, repository does not exist or may require 'docker login'")
Pulling cron (sentry-onpremise-local:latest)...
404 Client Error: Not Found ("pull access denied for sentry-onpremise-local, repository does not exist or may require 'docker login'")
Pulling sentry-cleanup (sentry-cleanup-onpremise-local:latest)...
404 Client Error: Not Found ("pull access denied for sentry-cleanup-onpremise-local, repository does not exist or may require 'docker login'")
Pulling web (sentry-onpremise-local:latest)...
404 Client Error: Not Found ("pull access denied for sentry-onpremise-local, repository does not exist or may require 'docker login'")
Pulling symbolicator-cleanup (symbolicator-cleanup-onpremise-local:latest)...
404 Client Error: Not Found ("pull access denied for symbolicator-cleanup-onpremise-local, repository does not exist or may require 'docker login'")
latest: Pulling from getsentry/sentry
8d691f585fa8: Pulling fs layer
3fd6980f9df6: Pulling fs layer
c4f890363b98: Pulling fs layer
b8cfb9853b0a: Pulling fs layer
ad50c051941d: Pulling fs layer
d5f911626281: Pulling fs layer
85598e244c1f: Pulling fs layer
ce6ec8873cec: Pulling fs layer
8a1f8f6f2eaa: Pulling fs layer
08cad7172a8d: Pulling fs layer
5c733cc21c91: Pulling fs layer
b8cfb9853b0a: Waiting
ad50c051941d: Waiting
d5f911626281: Waiting
85598e244c1f: Waiting
ce6ec8873cec: Waiting
8a1f8f6f2eaa: Waiting
08cad7172a8d: Waiting
5c733cc21c91: Waiting
3fd6980f9df6: Verifying Checksum
3fd6980f9df6: Download complete
c4f890363b98: Verifying Checksum
c4f890363b98: Download complete
8d691f585fa8: Verifying Checksum
8d691f585fa8: Download complete
ad50c051941d: Verifying Checksum
ad50c051941d: Download complete
d5f911626281: Verifying Checksum
d5f911626281: Download complete
b8cfb9853b0a: Verifying Checksum
b8cfb9853b0a: Download complete
8a1f8f6f2eaa: Verifying Checksum
8a1f8f6f2eaa: Download complete
85598e244c1f: Verifying Checksum
85598e244c1f: Download complete
08cad7172a8d: Verifying Checksum
08cad7172a8d: Download complete
5c733cc21c91: Verifying Checksum
5c733cc21c91: Download complete
8d691f585fa8: Pull complete
3fd6980f9df6: Pull complete
c4f890363b98: Pull complete
b8cfb9853b0a: Pull complete
ad50c051941d: Pull complete
d5f911626281: Pull complete
85598e244c1f: Pull complete
ce6ec8873cec: Verifying Checksum
ce6ec8873cec: Download complete
ce6ec8873cec: Pull complete
8a1f8f6f2eaa: Pull complete
08cad7172a8d: Pull complete
5c733cc21c91: Pull complete
Digest: sha256:f79263c7120df76329c5f12fdab4b45b4c86ca482088ecbc5b6357ade2eeacaf
Status: Downloaded newer image for getsentry/sentry:latest
Building web
Step 1/8 : ARG SENTRY_IMAGE
Step 2/8 : FROM ${SENTRY_IMAGE:-getsentry/sentry:latest}
 ---> 333bef514bd5
Step 3/8 : WORKDIR /usr/src/sentry
Removing intermediate container a1fa3b7161af
 ---> 45ba25037e52
Step 4/8 : ENV PYTHONPATH /usr/src/sentry
 ---> Running in b1955653738b
Removing intermediate container b1955653738b
 ---> e91cdd755596
Step 5/8 : COPY . /usr/src/sentry
 ---> 6384282b4c22
Step 6/8 : RUN if [ -s requirements.txt ]; then pip install -r requirements.txt; fi
 ---> Running in 867e5ef40498
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Removing intermediate container 867e5ef40498
 ---> 1f5dc2109676
Step 7/8 : RUN if [ -s setup.py ]; then pip install -e .; fi
 ---> Running in a15ebaa7ddb2
Removing intermediate container a15ebaa7ddb2
 ---> bf5a4f0f279f
Step 8/8 : RUN if [ -s sentry.conf.py ]; then cp sentry.conf.py $SENTRY_CONF/; fi   && if [ -s config.yml ]; then cp config.yml $SENTRY_CONF/; fi
 ---> Running in cb9955096839
Removing intermediate container cb9955096839
 ---> 9f7e41b29ce1

Successfully built 9f7e41b29ce1
Successfully tagged sentry-onpremise-local:latest
Build or rebuild services.

Services are built once and then tagged as `project_service`,
e.g. `composetest_db`. If you change a service's `Dockerfile` or the
contents of its build directory, you can run `docker-compose build` to rebuild it.

Usage: build [options] [--build-arg key=val...] [SERVICE...]

Options:
    --force-rm              Always remove intermediate containers.
    --no-cache              Do not use cache when building the image.
    --pull                  Always attempt to pull a newer version of the image.
    -m, --memory MEM        Sets memory limit for the build container.
    --build-arg key=val     Set build-time variables for one service.
Cleaning up...

[merlin9527]

me too

[itpimp]

Me three...

[BYK]

Turns out you all need at least [email protected] now. See #353.

[itpimp]

I'm on: docker-compose version 1.24.1, build 4667896
Still same issue.

[BYK]

@itpimp can you share your logs too?

[itpimp]

richard@sentry01-upgrade-test:/opt/sentry_10/onpremise$ cat sentry_install_log-2020-01-17_19-27-03.txt
Checking minimum requirements...
Removing network onpremise_default
Network onpremise_default not found.
Removing network sentry_onpremise_default
Network sentry_onpremise_default not found.

Creating volumes for persistent storage...
Created sentry-data.
Created sentry-postgres.
Created sentry-redis.
Created sentry-zookeeper.
Created sentry-kafka.
Created sentry-clickhouse.
Created sentry-symbolicator.

sentry/sentry.conf.py already exists, skipped creation.
sentry/config.yml already exists, skipped creation.
sentry/requirements.txt already exists, skipped creation.

Generating secret key...
Secret key written to sentry/config.yml

Building and tagging Docker images...

Pulling redis ...
Pulling postgres ...
Pulling smtp ...
Pulling zookeeper ...
Pulling kafka ...
Pulling memcached ...
Pulling symbolicator ...
Pulling clickhouse ...
Pulling snuba-api ...
Pulling snuba-replacer ...
Pulling snuba-cleanup ...
Pulling snuba-consumer ...
Pulling post-process-forwarder ...
Pulling worker ...
Pulling cron ...
Pulling sentry-cleanup ...
Pulling web ...
Pulling symbolicator-cleanup ...
404 Client Error: Not Found ("pull access denied for sentry-onpremise-local, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
404 Client Error: Not Found ("pull access denied for sentry-onpremise-local, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
Pulling post-process-forwarder ... done
Pulling cron ... done
Pulling memcached ... pulling from library/memcached
Pulling clickhouse ... pulling from yandex/clickhouse-se...
Pulling clickhouse ... pulling fs layer
Pulling clickhouse ... pulling fs layer
Pulling clickhouse ... pulling fs layer
Pulling clickhouse ... pulling fs layer
Pulling clickhouse ... pulling fs layer
Pulling clickhouse ... pulling fs layer
Pulling clickhouse ... pulling fs layer
Pulling clickhouse ... pulling fs layer
Pulling clickhouse ... pulling fs layer
Pulling clickhouse ... pulling fs layer
Pulling clickhouse ... pulling fs layer
Pulling clickhouse ... waiting
Pulling clickhouse ... waiting
Pulling clickhouse ... waiting
Pulling clickhouse ... waiting
Pulling clickhouse ... waiting
Pulling clickhouse ... waiting
Pulling clickhouse ... waiting
Pulling clickhouse ... waiting
Pulling symbolicator ... pulling from getsentry/symbolicator
Pulling symbolicator ... pulling fs layer
Pulling symbolicator ... pulling fs layer
Pulling symbolicator ... pulling fs layer
Pulling symbolicator ... pulling fs layer
Pulling symbolicator ... pulling fs layer
Pulling symbolicator ... pulling fs layer
Pulling symbolicator ... pulling fs layer
Pulling symbolicator ... waiting
Pulling symbolicator ... waiting
Pulling symbolicator ... waiting
Pulling symbolicator ... waiting
Pulling symbolicator ... waiting
Pulling symbolicator ... waiting
Pulling symbolicator ... waiting
Pulling memcached ... pulling fs layer
Pulling memcached ... pulling fs layer
Pulling memcached ... pulling fs layer
Pulling memcached ... pulling fs layer
Pulling memcached ... pulling fs layer
Pulling memcached ... pulling fs layer
Pulling memcached ... waiting
Pulling memcached ... waiting
Pulling memcached ... waiting
Pulling memcached ... waiting
Pulling memcached ... waiting
Pulling memcached ... waiting
Pulling clickhouse ... downloading (55.7%)
Pulling clickhouse ... downloading (100.0%)
Pulling clickhouse ... verifying checksum
Pulling clickhouse ... download complete
Pulling clickhouse ... downloading (86.5%)
Pulling clickhouse ... download complete
Pulling clickhouse ... downloading (1.0%)
Pulling clickhouse ... downloading (21.4%)
Pulling clickhouse ... downloading (44.7%)

After seeing the failure, I cancelled the install.

[omaraltayyan]

i had the same issue, installed docker-compose using the following commands

sudo curl -L "https://github.com/docker/compose/releases/download/1.25.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

sudo chmod +x /usr/local/bin/docker-compose

this downloads version 1.25 which is the latest version currently on the docs

and it worked

[BYK]

After seeing the failure, I cancelled the install.

Yes, there is your problem: just don't :) The 404s are normal, that's why we have the --ignore-pull-failures flag on that docker-compose pull command. The issue @killarny is actually experiencing is due to docker-compose build command not having the --parallel flag before version 1.23.0, thus bailing at that point in the flow.

[omaraltayyan]

After seeing the failure, I cancelled the install.

Yes, there is your problem: just don't :) The 404s are normal, that's why we have the --ignore-pull-failures flag on that docker-compose pull command. The issue @killarny is actually experiencing is due to docker-compose build command not having the --parallel flag before version 1.23.0, thus bailing at that point in the flow.

must be the case that running ./install.sh simply removes these errors right?

[martijnrondeel]

If 404s are expected behavior, including a note in the readme about this would be nice (and possibly prevent new issues about this quirk in the future).

[killarny]

Updating my docker-compose did indeed solve the problem over here. Thanks, @BYK!

I'm not sure what the etiquette is for closing my own issues on someone else's project, so I'll leave that up to you. 👍 (Oh, presumably it'll get closed when the associated MR is accepted, I see..)

[itpimp]

@omaraltayyan BRAVO! Worked!.... but... didn't upgrade my existing sentry.

Reloading VM and going to try it again.

[BYK]

@omaraltayyan

must be the case that running ./install.sh simply removes these errors right?

Not necessarily. We use both the build context and image name tagging as we make use of the same image in a few services. This prevents multiple redundant builds of the same image. But when we try to do docker-compose pull it tries to pull these images, which are only meant to be locally. This is the cause of the 404 errors. /cc @martijnrondeel

@killarny - yeah, I'd wait for the PR to auto-close this as there is a legitimate issue, just not what your original title implies :)