Salt: Various issues with dockermod and docker_image
Description of Issue/Question
dockermod and docker_image recently underwent some changes that appear to have resulted in various issues. Below is an executable example.
Setup
#! /bin/bash
LOG_LVL=error
ROOT_DIR=$(mktemp -d)
DOCKER_DIR=$(mktemp -d)
DOWNLOAD_DIR=$(mktemp -d)
python3 -m venv $ROOT_DIR > /dev/null
source $ROOT_DIR/bin/activate
git clone https://github.com/saltstack/salt $DOWNLOAD_DIR/salt > /dev/null
pip install $DOWNLOAD_DIR/salt > /dev/null
pip install docker > /dev/null
CONFIG_DIR=$ROOT_DIR/etc/salt
FILE_ROOT_BASE=$ROOT_DIR/srv/salt/base
FILE_ROOT_FOO=$ROOT_DIR/srv/salt/foo
PILLAR_ROOT_BASE=$ROOT_DIR/srv/pillar/base
PILLAR_ROOT_FOO=$ROOT_DIR/srv/pillar/foo
mkdir -p $CONFIG_DIR
cat <<EOF > $CONFIG_DIR/minion
root_dir: $ROOT_DIR
file_roots:
base:
- $FILE_ROOT_BASE
foo:
- $FILE_ROOT_FOO
pillar_roots:
base:
- $PILLAR_ROOT_BASE
foo:
- $PILLAR_ROOT_FOO
pillar_source_merging_strategy: none
top_file_merging_strategy: none
pillarenv_from_saltenv: True
EOF
cat <<EOF > $CONFIG_DIR/minion_id
my_minion
EOF
mkdir -p $PILLAR_ROOT_BASE
cat <<EOF > $PILLAR_ROOT_BASE/top.sls
base:
'my_minion':
- base_pillar
EOF
mkdir -p $PILLAR_ROOT_FOO
cat <<EOF > $PILLAR_ROOT_FOO/top.sls
foo:
'my_minion':
- foo_pillar
EOF
cat <<EOF > $PILLAR_ROOT_BASE/base_pillar.sls
A:
B: 'curl'
EOF
cat <<EOF > $PILLAR_ROOT_FOO/foo_pillar.sls
A:
B: 'fake_package_name_should_break_things'
EOF
mkdir -p $FILE_ROOT_BASE
cat <<EOF > $FILE_ROOT_BASE/outside_docker.sls
salt-docker-base:
docker_image.present:
- force: True
- tag: latest
- build: $DOCKER_DIR
salt-docker-test:
docker_image.present:
- tag: latest
- base: salt-docker-base
- sls: inside_docker
- saltenv: foo
- force: True
- require:
- salt-docker-base
install_pkgs_outside:
pkg.installed:
- name: {{ salt['pillar.get']('A:B') }}
EOF
mkdir -p $FILE_ROOT_FOO
cat <<EOF > $FILE_ROOT_FOO/inside_docker.sls
install_pkgs_inside:
pkg.installed:
- name: {{ salt['pillar.get']('A:B') }}
EOF
mkdir -p $DOCKER_DIR
cat <<EOF > $DOCKER_DIR/Dockerfile
FROM ubuntu:zesty
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y python3 python3-pip
RUN apt-get clean
RUN ln -s /usr/bin/python3 /usr/bin/python
RUN ln -s /usr/bin/pip3 /usr/bin/pip
CMD []
ENTRYPOINT []
EOF
salt-call --local --log-file-level=$LOG_LVL --config-dir=$CONFIG_DIR state.apply outside_docker
salt-call --local --log-file-level=$LOG_LVL --config-dir=$CONFIG_DIR docker.sls_build img_built_with_sls_build base=salt-docker-base mods=inside_docker saltenv=foo
# salt --versions-report
rm -rf $ROOT_DIR $FILE_ROOT_BASE $FILE_ROOT_FOO $DOWNLOAD_DIR
Steps to Reproduce Issue
Clear your docker containers (docker ps) and docker images (docker images) and run the script. Example output:
[WARNING ] /tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/modules/dockermod.py:3925: DeprecationWarning: The 'image' argument to docker.build has been deprecated, please use 'repository' instead.
[ERROR ] An exception occurred in this state: Traceback (most recent call last):
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/docker/api/client.py", line 222, in _raise_for_status
response.raise_for_status()
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localunixsocket/v1.26/images/salt-docker-base:latest/json
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/modules/dockermod.py", line 630, in _client_wrapper
ret = func(*args, **kwargs)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/docker/api/image.py", line 262, in inspect_image
self._get(self._url("/images/{0}/json", image)), True
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/docker/api/client.py", line 228, in _result
self._raise_for_status(response)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/docker/api/client.py", line 224, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("No such image: salt-docker-base:latest")
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/states/docker_image.py", line 326, in present
__salt__['docker.inspect_image'](full_image)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/modules/dockermod.py", line 1813, in inspect_image
ret = _client_wrapper('inspect_image', name)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/modules/dockermod.py", line 421, in wrapper
ret = wrapped(*args, **kwargs)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/modules/dockermod.py", line 636, in _client_wrapper
exc.explanation)
salt.exceptions.CommandExecutionError: Error 404: No such image: salt-docker-base:latest
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/state.py", line 1934, in call
**cdata['kwargs'])
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/loader.py", line 1799, in wrapper
return f(*args, **kwargs)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/states/docker_image.py", line 329, in present
msg = exc.__str__()
UnboundLocalError: local variable 'exc' referenced before assignment
local:
----------
ID: salt-docker-base
Function: docker_image.present
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/docker/api/client.py", line 222, in _raise_for_status
response.raise_for_status()
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localunixsocket/v1.26/images/salt-docker-base:latest/json
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/modules/dockermod.py", line 630, in _client_wrapper
ret = func(*args, **kwargs)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/docker/api/image.py", line 262, in inspect_image
self._get(self._url("/images/{0}/json", image)), True
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/docker/api/client.py", line 228, in _result
self._raise_for_status(response)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/docker/api/client.py", line 224, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("No such image: salt-docker-base:latest")
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/states/docker_image.py", line 326, in present
__salt__['docker.inspect_image'](full_image)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/modules/dockermod.py", line 1813, in inspect_image
ret = _client_wrapper('inspect_image', name)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/modules/dockermod.py", line 421, in wrapper
ret = wrapped(*args, **kwargs)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/modules/dockermod.py", line 636, in _client_wrapper
exc.explanation)
salt.exceptions.CommandExecutionError: Error 404: No such image: salt-docker-base:latest
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/state.py", line 1934, in call
**cdata['kwargs'])
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/loader.py", line 1799, in wrapper
return f(*args, **kwargs)
File "/tmp/tmp.8wcYIPL3yV/lib/python3.5/site-packages/salt/states/docker_image.py", line 329, in present
msg = exc.__str__()
UnboundLocalError: local variable 'exc' referenced before assignment
Started: 18:41:01.508113
Duration: 71137.274 ms
Changes:
----------
ID: salt-docker-test
Function: docker_image.present
Result: False
Comment: One or more requisite failed: outside_docker.salt-docker-base
Changes:
----------
ID: install_pkgs_outside
Function: pkg.installed
Name: curl
Result: True
Comment: All specified packages are already installed
Started: 18:42:15.223992
Duration: 48.651 ms
Changes:
Summary for local
------------
Succeeded: 1
Failed: 2
------------
Total states run: 3
Total run time: 71.186 s
Error running 'docker.sls_build': Error 404: repository salt-docker-base not found: does not exist or no pull access
Versions Report
Salt Version:
Salt: 2018.2.0-2-g98755b9
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: Not Installed
docker-py: 2.6.1
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.5.3 (default, Nov 23 2017, 11:34:05)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.3
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.2
ZMQ: 4.1.6
System Versions:
dist: Ubuntu 17.04 zesty
locale: UTF-8
machine: x86_64
release: 4.12.0-041200rc6-generic
system: Linux
version: Ubuntu 17.04 zesty
brianthelion
All 24 comments
diff --git a/salt/states/docker_image.py b/salt/states/docker_image.py
index 81100972e0..f01d6b721b 100644
--- a/salt/states/docker_image.py
+++ b/salt/states/docker_image.py
@@ -258,7 +258,8 @@ def present(name,
try:
# map values passed from the state to the build args
build_args['path'] = build
- build_args['image'] = full_image
+ build_args['repository'] = name
+ build_args['tag'] = tag
build_args['dockerfile'] = dockerfile
image_update = __salt__['docker.build'](**build_args)
except Exception as exc:
@@ -322,18 +323,21 @@ def present(name,
# Only add to the changes dict if layers were pulled
ret['changes'] = image_update
+ error = True
try:
__salt__['docker.inspect_image'](full_image)
error = False
- except CommandExecutionError:
+ except CommandExecutionError as exc:
msg = exc.__str__()
- if '404' not in msg:
- error = 'Failed to inspect image \'{0}\' after it was {1}: {2}'.format(
+ if '404' in msg:
+ error_msg = 'Failed to inspect image \'{0}\' after it was {1}: {2}'.format(
full_image, action, msg
)
+ else:
+ raise exc
if error:
- ret['comment'] = error
+ ret['comment'] = error_msg
else:
ret['result'] = True
if not ret['changes']:
gets us to
[ERROR ] UnicodeDecodeError while decoding output of cmd cat "/tmp/tmp.88AzIk6sWg/var/cache/salt/minion/thin/thin.tgz" | docker exec -i c7ebed21ccea867d82fd16331637792fa5bee3c8f87c7443afafe41312493d2e env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin tee "/tmp/salt.docker.c16787/thin.tgz"
[ERROR ] UnicodeDecodeError while decoding output of cmd cat "/tmp/__salt.tmp.uvscuuoy" | docker exec -i c7ebed21ccea867d82fd16331637792fa5bee3c8f87c7443afafe41312493d2e env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin tee "/tmp/salt.docker.b73b13/salt_state.tgz"
[ERROR ] UnicodeDecodeError while decoding output of cmd cat "/tmp/tmp.88AzIk6sWg/var/cache/salt/minion/thin/thin.tgz" | docker exec -i c7ebed21ccea867d82fd16331637792fa5bee3c8f87c7443afafe41312493d2e env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin tee "/tmp/salt.docker.9684ff/thin.tgz"
local:
----------
ID: salt-docker-base
Function: docker_image.present
Result: True
Comment: Image 'salt-docker-base:latest' was built
Started: 18:47:35.593692
Duration: 63214.185 ms
Changes:
----------
Id:
5f02fd898f73
Image:
- salt-docker-base:latest
Status:
Downloaded newer image for ubuntu:zesty
Time_Elapsed:
0.06360554695129395
----------
ID: salt-docker-test
Function: docker_image.present
Result: True
Comment: Image 'salt-docker-test:latest' was built
Started: 18:48:38.808522
Duration: 49715.548 ms
Changes:
----------
Id:
sha256:69ab3364685fb7c9412c700549cf33df036eee8b17e794823eaeb6bdf1bea66b
Image:
None
Time_Elapsed:
1.0145847797393799
----------
ID: install_pkgs_outside
Function: pkg.installed
Name: curl
Result: True
Comment: All specified packages are already installed
Started: 18:49:31.149498
Duration: 52.903 ms
Changes:
Summary for local
------------
Succeeded: 3 (changed=2)
Failed: 0
------------
Total states run: 3
Total run time: 112.983 s
[ERROR ] UnicodeDecodeError while decoding output of cmd cat "/tmp/tmp.88AzIk6sWg/var/cache/salt/minion/thin/thin.tgz" | docker exec -i 7e45e9dab69bfe0fe1ddc3b0d91bcf786622e90870e3666ea2d6476f51cfc704 env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin tee "/tmp/salt.docker.67fc3d/thin.tgz"
[ERROR ] UnicodeDecodeError while decoding output of cmd cat "/tmp/__salt.tmp.a3ajal71" | docker exec -i 7e45e9dab69bfe0fe1ddc3b0d91bcf786622e90870e3666ea2d6476f51cfc704 env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin tee "/tmp/salt.docker.2903aa/salt_state.tgz"
[ERROR ] UnicodeDecodeError while decoding output of cmd cat "/tmp/tmp.88AzIk6sWg/var/cache/salt/minion/thin/thin.tgz" | docker exec -i 7e45e9dab69bfe0fe1ddc3b0d91bcf786622e90870e3666ea2d6476f51cfc704 env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin tee "/tmp/salt.docker.ad9cb4/thin.tgz"
local:
----------
Id:
sha256:f40b2a04666a05ec5027e9ec061e934f0c3b6c0b7034071510645c447bf551c7
Image:
None
Time_Elapsed:
0.7810461521148682
Remaining issues:
- [x]
UnicodeDecodeErrors. - [ ] Both
salt-call --localruns should fail ifpillarenv_from_saltenv: Trueis being respected.
- (Note the
'fake_package_name_should_break_things'in thefoopillarenv)
- (Note the
brianthelion
on 13 Dec 2017
ping @terminalmage mind taking a look here and work with @brianthelion on these issues?
Also thanks for bringing these to our attention :)
Ch3LL
on 13 Dec 2017
@terminalmage Any input on the pillar_from_saltenv line-item would be appreciated as I am thoroughly blocked on pillar targeting for docker images.
The pillar logic's entrypoint assumes pillarenv = saltenv here:
https://github.com/saltstack/salt/blob/53eee476ac8df7dfd3e14ad617b827527196ee08/salt/modules/dockermod.py#L6608
And continues here:
https://github.com/saltstack/salt/blob/53eee476ac8df7dfd3e14ad617b827527196ee08/salt/modules/dockermod.py#L6467-L6484
brianthelion
on 14 Dec 2017
https://github.com/saltstack/salt/pull/45038 fixes the UnboundLocalError. Can you test with that one-line change?
terminalmage
on 15 Dec 2017
Actually I didn't read down further. Hold on.
terminalmage
on 15 Dec 2017
OK, https://github.com/saltstack/salt/pull/45038 is updated to include the proper argument passing for docker.build.
terminalmage
on 15 Dec 2017
Working on the pillarenv stuff. It looks like whomever added this support didn't do it properly.
terminalmage
on 15 Dec 2017
@brianthelion can you test with the updates to that PR? You can use this walkthrough to fetch the PR into a local branch and run Salt from the updated code.
terminalmage
on 15 Dec 2017
Will do in the next few hours. Thanks @terminalmage !
brianthelion
on 15 Dec 2017
OK, if you feel it's good to go, then comment in that PR. Otherwise, we can make whatever changes still need to be made. I'm currently in the middle of a much bigger project so I unfortunately did not have the opportunity to test this locally.
terminalmage
on 15 Dec 2017
@brianthelion Since #45038 has been merged now, is this issue OK to close?
rallytime
on 27 Dec 2017
@rallytime We still need to deal with the contents_pillar issues. Do you want to give those another ticket?
brianthelion
on 28 Dec 2017
@brianthelion I just spoke with @terminalmage and he says we can keep the contents_pillar problem in this issue. Can you post a reproducible test case here so we can take a look?
rallytime
on 2 Jan 2018
@rallytime Sorry, I've got a ton going on this week, post-holidays. I can probably address this next week sometime.
brianthelion
on 4 Jan 2018
Ok @brianthelion - Keep us posted when you can. Thanks!
rallytime
on 4 Jan 2018
@rallytime Just ran my older test suite and caught one thing unrelated to the contents_pillar issue:
diff --git a/salt/client/ssh/state.py b/salt/client/ssh/state.py
index b912e8ab12..7c30ae8230 100644
--- a/salt/client/ssh/state.py
+++ b/salt/client/ssh/state.py
@@ -206,7 +206,7 @@ def prep_trans_tar(opts, file_client, chunks, file_refs, pillar=None, id_=None,
os.makedirs(env_root)
for ref in file_refs[saltenv]:
for name in ref:
- short = salt.utils.url.parse(name)[0]
+ short = salt.utils.url.parse(name)[0].lstrip('/')
cache_dest = os.path.join(cache_dest_root, short)
try:
path = file_client.cache_file(name, saltenv, cachedir=cachedir)
That's against your Oxygen branch.
brianthelion
on 8 Jan 2018
Can you be more specific about the issue you're seeing?
rallytime
on 10 Jan 2018
Oh, sorry, I think @terminalmage and I talked about this on IRC or something.
There is an unhandled exception in the ssh client when using a non-standard root_dir. Under that condition, file_refs[saltenv] gets populated with paths that have a leading "/". That would be fine except that os.path.join(...) has really specific semantics around leading "/" that break the logic in the patched code block, above.
Again, this is unrelated to the docker stuff; I just happened to hit it when I was testing the attached PR in a /tmp sandbox.
brianthelion
on 11 Jan 2018
@rallytime In addition to the above, one of my devs is seeing an issue with Docker entrypoints not being inherited from the base image in Oxygen. The issue is not present in Nitrogen, possibly due to this PR. We'll try to pull together an example in the next few days.
brianthelion
on 18 Jan 2018
OK @brianthelion, I've been super busy with other issues affecting the upcoming release, but by later this week or early the next, I'll be free to tackle these issues.
Specifically with regard to the entrypoints issue, since at least 2017.7.0 we don't pass arguments if they're not present in the SLS, with very few exceptions, and I don't think that entrypoints is one of them. I'd be interested to see a use case and would definitely want to get that fixed though.
If you can also get an example of the contents_pillar issue, I'd definitely want to look at those as well.
Once you get examples, could you please open a new issue and @ mention me? Once we have that new issue opened, I will close this issue, but for now I will keep it open to remind me to follow up.
terminalmage
on 22 Jan 2018
I've also opened https://github.com/saltstack/salt/pull/45606 to fix that issue with the path joining. Thanks, that was a nice catch!
terminalmage
on 22 Jan 2018
Since the main issue that was raised here that was blocking for Oxygen has been resolved, I am removing the "oxygen" and "blocker" labels. We can keep this issue open in order to address the remaining problems raised, however.
rallytime
on 29 Jan 2018
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.
![stale[bot] picture](https://avatars.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 14 May 2019
Related issues
sfozz
路
3Comments
icycle77
路
3Comments
sagetherage
路
3Comments
twangboy
路
3Comments
saurabhnemade
路
3Comments