Aws-codebuild-docker-images: codebuild agent 4.0 is missing the aws binary

[renperez]

hey guys, we use codebuild and we are using aws command to login to ecr. this stopped working today on projects that are using the latest run time version. (4.0) Can you guys please re-add the aws binary to this version.

[tyliggity]

Just ran into the same issue. Strange cuz last change was days ago and a URL change but both the old and new url's seem to still work from direct browser navigation: #318

[tyliggity]

wow, it just came back for me...not even kidding

[jgriffith]

I'm seeing the same issue. The AWS binary is missing from Ubuntu Standard 4.0 on us-east-1

++ aws sts get-caller-identity --query Account --output text
--
30 | build.sh: line 5: aws: command not found

Edit: previous builds that were failing, now appear to be working 9:00PM EST

[ChrisRut]

We also just ran into this out of no where, a build passed 1hr ago and the same build is failing now.

[jeffborg]

Same here - trying aws/codebuild/standard:3.0-19.11.26 hopefully it's still ok.

[galenhuntington]

Yeah I worked around this by using standard:3.0-19.11.26.

[ChrisRut]

Spoke w/ AWS support they said they are pushing out a fix shortly.

[ChrisRut]

They also suggested adding yum -y install awscli to your buildspec.yaml as a temporary work-around.

[whereisaaron]

We didn't change our standard build image version, and there is still only one version of 4.0 from 13 March, yet the 'aws' install still disappeared a few hours ago. It is a bit disconcerting the even with a locked image it is not a stable build.

[whereisaaron]

I was looking at the Dockerfile's to understand why my builds were breaking with 4.0 suddenly and it appears this stanza to install it that was in previous versions Dockerfile was removed in 4.0. Although you get a side-effect install if you specify a python runtime version.

standard 3.0:

RUN set -ex \
    && pip3 install awscli boto3

Side-effect awscli install if using python runtime in 4.0:

COPY tools/runtime_configs/python/$PYTHON_38_VERSION /root/.pyenv/plugins/python-build/share/python-build/$PYTHON_38_VERSION
RUN   pyenv install $PYTHON_38_VERSION; rm -rf /tmp/*
RUN   pyenv global  $PYTHON_38_VERSION
RUN set -ex \
    && pip3 install --no-cache-dir --upgrade --force-reinstall "pip==$PYTHON_PIP_VERSION" \
    && pip3 install --no-cache-dir --upgrade "PyYAML==5.1.2" \
    && pip3 install --no-cache-dir --upgrade setuptools wheel aws-sam-cli awscli boto3 pipenv virtualenv

[whereisaaron]

AWS tells me it is fixed.

Between 4:25 PM and 6:45 PM PDT we experienced increased build errors due to a missing dependency in the build workflow. The issue has been resolved and the service is operating normally.

[subinataws]

This should be fixed now. Let us know if you still face any issues.

[2ps]

@subinataws : this has reared its head for other python binaries installed by pip install. For example, after running pip3.7 install invoke, the command invoke --version will not be able to find the invoke binary in the path.

[subinataws]

@2ps - Not sure I understand the issue you are trying to highlight. A "pip3 install invoke && invoke --version" works just fine for me.

output:

[Container] Running command pip3 install invoke
Collecting invoke
Downloading https://files.pythonhosted.org/packages/2c/16/f00efa99ae9f255142a230ce6819c37ae9dd29a7144477c1161cc72d01ed/invoke-1.4.1-py3-none-any.whl (210kB)
Installing collected packages: invoke
Successfully installed invoke-1.4.1
WARNING: You are using pip version 19.3.1; however, version 20.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[Container] Running command invoke --version
Invoke 1.4.1

Why do you have "pip3.7"?

[2ps]

@subinataws Because pip3 is the ubuntu / system pip, and pip3.7 is the pip that is associated with the codebuild PYTHON_37_VERSION that we specify in our buildspec. Do which pip3 and compare with which pip3.7.

[subinataws]

There are multiple non-EOL version on this image. You could use "python3.7 -m pip install invoke && python3.7 -m invoke --version", after selecting the runtime as python 3.7.

https://stackoverflow.com/questions/2812520/dealing-with-multiple-python-versions-and-pip

[2ps]

Sure, just try adding that in front of every call to the aws cli . . . Please just add the shimmed python module bin path to the path, or stop using pyenv entirely. We've gone through any number of bugs from the incorrect usage of pyenv, several of which keep reappearing and regressing. As a community member and a customer, please move away from it or figure out how to use it correctly.