Cypress: All chrome docker image versions upgraded to chrome 80

[jrichard0725]

Current behavior:

Pulling the latest cypress/browsers:node12.13.0-chrome78-ff70 and checking the version of google-chrome shows version 80 (latest)

docker run -it cypress/browsers:node12.13.0-chrome78-ff70 google-chrome --version

Returns: Google Chrome 80.0.3987.87

Desired behavior:

Should be running the version the image is tagged with

docker run -it cypress/browsers:node12.13.0-chrome78-ff70 google-chrome --version

Should return: Google Chrome 78.xxxxx

Test code to reproduce

Looking at https://circleci.com/gh/cypress-io/cypress-docker-images/2480#config/containers/0 of the latest ci build I can see that the docker image is rebuilt for every version every time even though those dockerfiles have not been updated.

Looking at https://github.com/cypress-io/cypress-docker-images/blob/master/browsers/node12.13.0-chrome78-ff70/Dockerfile, if the image is rebuilt it will apt-get install google-chrome-stable which is the latest version.

Looking at https://hub.docker.com/r/cypress/browsers/tags?page=1&name=12.13.0-chrome78-ff70 you can see the image was updated 4 hours ago (corresponding to the last ci build time).

From what I can tell this impacts almost every chrome docker image version since https://hub.docker.com/r/cypress/browsers/tags?page=1&ordering=last_updated shows many chrome versions updated in the last 4 hours.

Versions

cypress/browsers:node12.13.0-chrome78-ff70

[jrichard0725]

possibly related to https://github.com/cypress-io/cypress/issues/6334

[bahmutov]

Investigating. CI checks if an image exists on Docker hub and seems today the check failed for a bunch of cypress/browsers* images, creating them again, but installing Chrome 80 since it is the latest. Ughh, what a mess.

Let me investigate to see what and how it happened.

[jrichard0725]

Also doesn't look like all the versions are available to lock in place for apt: https://www.ubuntuupdates.org/pm/google-chrome-stable

[bahmutov]

Hmm, example bad image

#!/bin/bash -eo pipefail
if npx docker-image-exists --quiet --repo cypress/browsers:node12.13.0-chrome78-ff70; then
  echo Found image cypress/browsers:node12.13.0-chrome78-ff70
  circleci-agent step halt
else
  echo Did not find Docker image cypress/browsers:node12.13.0-chrome78-ff70
fi
Did not find Docker image cypress/browsers:node12.13.0-chrome78-ff70

So the CI build for cypress/browsers:node13.6.0-chrome-80-ff72 (correct) also rebuilt with Chrome 80 the following images (I am marking the images I have restored from local Docker)

• cypress/browsers:node12.13.0-chrome78-ff70 ✅ restored
• cypress/browsers:chrome69 ✅ restored
• cypress/browsers:node8.9.3-npm6.10.1-chrome75 ✅ restored
• cypress/browsers:node10.2.1-chrome74 ✅ restored
• cypress/browsers:node12.6.0-chrome75 cypress/browsers:node12.6.0-chrome75

I think this might be due to bug in docker-image-exists or more likely Docker Hub API being down / rate limiting

[bahmutov]

Update

Root cause

We have automated building and pushing Docker images in https://github.com/cypress-io/cypress-docker-images using CircleCI. For each image, we do the following:

• check if the image with that tag exists on Docker Hub.
  
  
  
  • if the image exists, stop the CI job, all done
    
    otherwise:
  
  
• build the Docker image locally
• test the image
• push the image to Docker Hub

Today, we have merged a pull request https://github.com/cypress-io/cypress-docker-images/pull/220 that added a single image with Chrome 80. When CI ran through the image folders, all of the sudden the Docker Hub responded with an error or "no" when asked about 5 other Docker images (that already exist). Then the CI went on to build these images. Unfortunately, when installing Chrome, the official installation only has the latest stable Chrome version, which is v80. These 5 images go new Chrome stable (v80) and then were pushed, happily overwriting the existing images on Docker Hub.

We have restored the accidentally overwritten images by pushing old builds from a backup.

Future plans

• add more checks into building the container to avoid installing Chrome version X while building the image with the tag -chrome-Y
• archive versions of Chrome ourselves or find a source of older Chrome versions to install, like https://www.slimjet.com/chrome/download-chrome.php
• check Docker Hub several times before pushing an image
• mark the Docker image as built as soon as we pushed. Then we should not be building the image again.