Salt: 2016.11.1 - archive.extracted fails when symlink is in a tar file
Description of Issue/Question
After upgrading to 2016.11.1 I've noticed archive.extracted was causing fatal errors in the state execution with the debug log posted below.
After checking the logs on the minion I've also noticed this specific error being thrown every time I tried using _archive.extracted_ state:
The below paths (relative to /opt/java/jre/8u112/) exist, but are the incorrect type (i.e. file instead of directory or vice-versa).To proceed with extraction, set 'force' to True. Note that this will remove these paths before extracting.
- man/ja
The aforementioned file is actually a symlink that points to a folder in the same folder level:
lrwxrwxrwx. 1 root root 11 Sep 22 22:20 ja -> ja_JP.UTF-8
drwxr-xr-x. 3 root root 17 Sep 22 22:20 ja_JP.UTF-8
drwxr-xr-x. 2 root root 4096 Sep 22 22:20 man1
Once I removed the symlink from the filesystem, the state executed successfully.
Setup
CentOS 7.3 x86_64 (Server and Minion)
Steps to Reproduce Issue
Trying to extract the Oracle JDK 8u112 which has a symlink in the folder _man_
java.jre-archive:
file.directory:
- name: {{ java.home }}/{{ java.version }}
- user: root
- group: root
- makedirs: True
- mode: 755
- require_in:
- file: java.jre-home-dir
archive.extracted:
- name: {{ java.home }}/{{ java.version }}
- source: salt://{{ slspath }}/files/server-jre-{{ java.version }}-linux-x64.tar.gz
- options: --strip 1
- list_options: --strip 1
- enforce_ownership_on: {{ java.home }}/{{ java.version }}
- enforce_toplevel: False
- user: root
- group: root
- require:
- file: java.jre-archive
The minion function caused an exception: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1412, in _thread_return
return_data = executor.execute()
File "/usr/lib/python2.7/site-packages/salt/executors/direct_call.py", line 28, in execute
return self.func(*self.args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 544, in apply_
return highstate(**kwargs)
File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 816, in highstate
orchestration_jid=orchestration_jid)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 3454, in call_highstate
return self.state.call_high(high, orchestration_jid)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 2294, in call_high
ret = dict(list(disabled.items()) + list(self.call_chunks(chunks).items()))
File "/usr/lib/python2.7/site-packages/salt/state.py", line 1810, in call_chunks
running = self.call_chunk(low, running, chunks)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 2084, in call_chunk
running = self.call_chunk(chunk, running, chunks)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 2095, in call_chunk
running = self.call_chunk(low, running, chunks)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 2095, in call_chunk
running = self.call_chunk(low, running, chunks)
... (truncating the log output to keep it short, the above error line repeats like a thousand times)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 2095, in call_chunk
running = self.call_chunk(low, running, chunks)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 1992, in call_chunk
low = self._mod_aggregate(low, running, chunks)
File "/usr/lib/python2.7/site-packages/salt/state.py", line 762, in _mod_aggregate
agg_opt = self.functions['config.option']('state_aggregate')
File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1086, in __getitem__
func = super(LazyLoader, self).__getitem__(item)
File "/usr/lib/python2.7/site-packages/salt/utils/lazy.py", line 91, in __getitem__
if self._missing(key):
RuntimeError: maximum recursion depth exceeded
_
Versions Report
Salt Version:
Salt: 2016.11.1
Dependency Versions:
cffi: Not Installed
cherrypy: 3.2.2
dateutil: 2.5.3
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.5 (default, Nov 6 2016, 00:28:07)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: centos 7.3.1611 Core
machine: x86_64
release: 3.10.0-514.2.2.el7.x86_64
system: Linux
version: CentOS Linux 7.3.1611 Core
danlsgiga
All 12 comments
Just found that setting an if_missing in the state overcomes the issue... Anyway, this issue seems to be critical for the new archive state implementation.
danlsgiga
on 16 Jan 2017
@danlsgiga I have tagged this as a bug.
@terminalmage you might be interested in this.
Thanks,
Daniel
gtmanfred
on 17 Jan 2017
I'm afraid I'll need to see more of the log (preferably the entire thing), since the only parts you've exposed have to do with the state system, and I can't tell what is actually being run.
Also, you're using list_options incorrectly, which might be causing a problem.
This seems like it could be related to https://github.com/saltstack/salt/issues/38228 which had to do with the --strip option, the fix for which is expected in 2016.11.2. If you can, I would try testing against the head of the 2016.11 branch, as the fix for that issue was merged a few weeks ago.
terminalmage
on 17 Jan 2017
Hi @terminalmage, that's everything I got in the logs.
I'm guessing the issue is in here: https://github.com/saltstack/salt/blob/develop/salt/states/archive.py#L1040
for path_list, func in ((contents['dirs'], stat.S_ISDIR),
(contents['files'], stat.S_ISREG)):
The state is not accounting for symlinks, only regular files and directories.
danlsgiga
on 17 Jan 2017
stat.S_ISREG() refers to the destination of the symlink when invoked on an os.stat of a symlink, so you may be on to something with that.
I still can't reproduce this at all though, so if you can come up with a docker/vagrant/etc. instance where this can be reproduced, it'll go a long way towards finding what is causing the recursion.
terminalmage
on 18 Jan 2017
Thanks for the clarification on the os.stat.
So, I just created a docker container with 2016.11.1 HEAD.
First run:
[root@salt-archive salt]# salt-call --local state.apply java_jre
local:
----------
ID: java.jre-archive
Function: file.directory
Name: /opt/java/jre/8u112
Result: True
Comment: Directory /opt/java/jre/8u112 updated
Started: 00:23:53.596141
Duration: 5.477 ms
Changes:
----------
/opt/java/jre/8u112:
New Dir
----------
ID: java.jre-archive
Function: archive.extracted
Name: /opt/java/jre/8u112
Result: True
Comment: /tmp/server-jre-8u112-linux-x64.tar.gz extracted to /opt/java/jre/8u112/
Started: 00:23:53.602070
Duration: 3533.261 ms
Changes:
----------
extracted_files:
no tar output so far
updated ownership:
True
Summary for local
------------
Succeeded: 2 (changed=2)
Failed: 0
------------
Total states run: 2
Total run time: 3.539 s
Any subsequent run:
[root@salt-archive salt]# salt-call --local state.apply java_jre
[ERROR ] The below paths (relative to /opt/java/jre/8u112/) exist, but are the incorrect type (i.e. file instead of directory or vice-versa).To proceed with extraction, set 'force' to True. Note that this will remove these paths before extracting.
- man/ja
local:
----------
ID: java.jre-archive
Function: file.directory
Name: /opt/java/jre/8u112
Result: True
Comment: Directory /opt/java/jre/8u112 is in the correct state
Started: 00:25:56.650161
Duration: 3.96 ms
Changes:
----------
ID: java.jre-archive
Function: archive.extracted
Name: /opt/java/jre/8u112
Result: False
Comment: The below paths (relative to /opt/java/jre/8u112/) exist, but are the incorrect type (i.e. file instead of directory or vice-versa).To proceed with extraction, set 'force' to True. Note that this will remove these paths before extracting.
- man/ja
Started: 00:25:56.654593
Duration: 1497.275 ms
Changes:
Summary for local
------------
Succeeded: 1
Failed: 1
------------
Total states run: 2
Total run time: 1.501 s
Here is the output when force is set to True on the state:
[root@salt-archive salt]# salt-call --local state.apply java_jre
[ERROR ] One or more paths existed by were the incorrect type (i.e. file instead of directory or vice-versa), but could not be removed. The following errors were observed:
- Cannot call rmtree on a symbolic link
local:
----------
ID: java.jre-archive
Function: file.directory
Name: /opt/java/jre/8u112
Result: True
Comment: Directory /opt/java/jre/8u112 is in the correct state
Started: 00:27:26.224907
Duration: 4.04 ms
Changes:
----------
ID: java.jre-archive
Function: archive.extracted
Name: /opt/java/jre/8u112
Result: False
Comment: One or more paths existed by were the incorrect type (i.e. file instead of directory or vice-versa), but could not be removed. The following errors were observed:
- Cannot call rmtree on a symbolic link
Started: 00:27:26.229415
Duration: 1486.556 ms
Changes:
Summary for local
------------
Succeeded: 1
Failed: 1
------------
Total states run: 2
Total run time: 1.491 s
Here is my java_jre.sls:
java.jre-archive:
file.directory:
- name: /opt/java/jre/8u112
- user: root
- group: root
- makedirs: True
- mode: 755
archive.extracted:
- name: /opt/java/jre/8u112
- source: /tmp/server-jre-8u112-linux-x64.tar.gz
- options: --strip 1
- enforce_toplevel: False
- user: root
- group: root
Sorry, forgot to send you the docker image I've created for debugging.
First run:
$ docker run -t --name salt-archive danlsgiga/salt-archive:38711
Subsequent runs:
$ docker start -a salt-archive
Here is the Dockerfile:
FROM centos:latest
MAINTAINER Daniel Santos <[email protected]>
RUN yum -y update && \
curl -o /tmp/bootstrap-salt.sh -L https://bootstrap.saltstack.com && \
sh /tmp/bootstrap-salt.sh -X git 2016.11 && \
mkdir -p /srv/salt && \
sed -i 's/#file_client: remote/file_client: local/g' /etc/salt/minion && \
cd /tmp && curl -L -O -H "Cookie: oraclelicense=accept-securebackup-cookie" -k http://download.oracle.com/otn-pub/java/jdk/8u112-b15/server-jre-8u112-linux-x64.tar.gz
COPY jre.sls /srv/salt/
CMD ["/usr/bin/salt-call", "--local", "state.apply", "jre"]
jre.sls:
java.jre-archive:
file.directory:
- name: /opt/java/jre/8u112
- user: root
- group: root
- makedirs: True
- mode: 755
archive.extracted:
- name: /opt/java/jre/8u112
- source: /tmp/server-jre-8u112-linux-x64.tar.gz
- options: --strip 1
- enforce_toplevel: False
- user: root
- group: root
OK, I've opened a pull request (https://github.com/saltstack/salt/pull/38832) that I've confirmed fixes this. Thanks a lot for the Dockerfile, that helped. I made the following changes to it, which allow you to mount a git checkout and use it instead of the version installed from the bootstrap.
FROM centos:latest
MAINTAINER Daniel Santos <[email protected]>
RUN yum -y update && \
curl -o /tmp/bootstrap-salt.sh -L https://bootstrap.saltstack.com && \
sh /tmp/bootstrap-salt.sh -X git 2016.11 && \
mkdir -p /srv/salt && \
sed -i 's/#file_client: remote/file_client: local/g' /etc/salt/minion && \
cd /tmp && curl -L -O -H "Cookie: oraclelicense=accept-securebackup-cookie" -k http://download.oracle.com/otn-pub/java/jdk/8u112-b15/server-jre-8u112-linux-x64.tar.gz
COPY jre.sls /srv/salt/
ENV PYTHONPATH=/testing/:/testing/salt-testing/
ENV PATH=/testing/scripts/:/testing/salt/tests/:$PATH
VOLUME /testing
Just run docker run --rm -it -v /path/to/git/checkout:/testing <image_name> and then salt-call --local state.apply jre to execute the state.
terminalmage
on 19 Jan 2017
Awesome! Will this PR make into 2016.11.2 in time?
danlsgiga
on 20 Jan 2017
@danlsgiga yes, it will. We have not yet tagged that release and the pull request was just merged into the release branch.
terminalmage
on 20 Jan 2017
Thanks @terminalmage, I think you can close this issue. I've confirmed it is fixed with https://github.com/saltstack/salt/pull/38832
danlsgiga
on 20 Jan 2017
Great, thanks for the confirmation.
terminalmage
on 20 Jan 2017
Related issues
commutecat
路
3Comments
mooperd
路
3Comments
icycle77
路
3Comments
golmaal
路
3Comments
Arguros
路
3Comments