Salt: init.sls with included states fails with more than one, 'Template was specified incorrectly: False'

[dvenckus]

Description of Issue/Question

This applies to a problem with the 2018.3.1 version and I have a 2nd server built identically except for having a different version of salt. The same state files exist on each server. Each server runs as a masterless, stand-alone minion.

I have 2 separate Virtualbox Ubuntu 18.04 VMs:
Server 1: Salt 2018.3.1 installed
Server 2: Salt 2017.7.4 installed

I have a directory named 'common' containing an init.sls file and several sls files that are included by init.sls in the same directory.

common/init.sls:
     include: 
        - common.file1
        - common.file2
        - common.fille3

SERVER 1 & SERVER2:

I am able to run each of the included sls files individually with success

salt-call state.sls common.file1

However, when I call the directory for the init.sls file

salt-call state.sls common

it fails.

[INFO    ] Loading fresh modules for state activity
[INFO    ] Fetching file from saltenv 'base', ** done ** 'common/init.sls'
[ERROR   ] Template was specified incorrectly: False
local:
    Data failed to compile:
----------
    Specified SLS common.file2 in saltenv base is not available on the salt master or through a configured fileserver
    Specified SLS common.file3 in saltenv base is not available on the salt master or through a configured fileserver

I have tried changing the order of the included states.... no difference.
I commented out all but 1 included file (I rotated which state I kept).... then it worked.

ALL of these states work without issue on SERVER 2 with salt 2017.7.4

Versions Report

SERVER 1

Salt Version:
Salt: 2018.3.1

Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: 2.6.1
docker-py: Not Installed
gitdb: 2.0.3
gitpython: 2.1.8
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.7
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
python-gnupg: 0.4.1
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: 2.0.3
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.5

System Versions:
dist: Ubuntu 18.04 bionic
locale: ISO-8859-1
machine: x86_64
release: 4.15.0-20-generic
system: Linux
version: Ubuntu 18.04 bionic

SERVER 2

Salt Version:
Salt: 2017.7.4

Dependency Versions:
cffi: 1.11.5
cherrypy: unknown
dateutil: 2.6.1
docker-py: Not Installed
gitdb: 2.0.3
gitpython: 2.1.10
ioflo: Not Installed
Jinja2: 2.10
libgit2: 0.26.0
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.18
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.26.2
Python: 3.6.5 (default, Apr 1 2018, 05:46:30)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: 2.0.3
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.5

System Versions:
dist: Ubuntu 18.04 bionic
locale: ISO-8859-1
machine: x86_64
release: 4.15.0-20-generic
system: Linux
version: Ubuntu 18.04 bionic

[garethgreenaway]

@dvenckus Thanks for the report. Unfortunately I haven't been able to reproduce this so far. I have the init file as you have it above:

include: 
    - common.file1
    - common.file2
    - common.fille3

the common directory is directly below /srv/salt.

[dvenckus]

First, I overlooked the fact that we have 2 saltenv's defined, base, dev. We set 'environment' to 'base' on the minion and override with saltenv as needed.

I was also checking into the salt code (state.py) and found where the error is happening:

line 3603

 if not resolved_envs:
                            msg = ('Unknown include: Specified SLS {0}: {1} is not available on the salt '
                                   'master in saltenv(s): {2} '
                                   ).format(env_key,
                                            inc_sls,
                                            ', '.join(matches) if env_key == xenv_key else env_key)

It looks like the salt environment is not being resolved.

To be honest, I'm guessing. I've been staring at this for far too long.

[garethgreenaway]

@dvenckus This is a good clue! Thanks!

[dvenckus]

The salt file structure is:

/srv/salt/
    base/
        assets/
               [ grains, modules, templates, scripts ]
        pillars/
            ...
            top.sls
        states/
            ...
            top.sls
    dev/
        assets/
             [ grains, modules, templates, scripts ]
        pillars/
            ...
            top.sls
        states/
            ...
            top.sls

[garethgreenaway]

@dvenckus Thanks. When you're defining your environments on the Salt master you including the states directory in the file root for each environment?

[dvenckus]

This is the same for both VMs.

file_roots:
  base:
    - /srv/salt/base/states
    - /srv/salt/base/assets
    - /srv/salt/formulas/mariadb-saltstack-formula

  dev:
    - /srv/salt/dev/states
    - /srv/salt/base/states
    - /srv/salt/dev/assets
    - /srv/salt/base/assets
    - /srv/salt/formulas/mariadb-saltstack-formula



pillar_roots:
  base:
    - /srv/salt/base/pillars
  dev:
    - /srv/salt/dev/pillars
    - /srv/salt/base/pillars

And other relevant settings from the minion.conf:

# default to base environment, unless saltenv is specified
environment: base

top_file_merging_strategy: 'same'
env_order: ['base','dev']
default_top: 'base'

pillar_source_merging_strategy: 'smart'
pillarenv_from_saltenv: True
pillar_safe_render_error: True

[garethgreenaway]

@dvenckus Thanks. Which environment does the common directory live in?

[dvenckus]

At the moment 99% of our states and pillars are in 'base'. The 'dev' environment allows for easy testing of specific states and pillars before we fully deploy them. Only some simple test pillars and a test state in 'dev'. This is the same file configuration for both 2017.7.4 and 2018.3.1 installations of salt

[garethgreenaway]

@dvenckus And when you're seeing the issue when you're specifying the saltenv as dev?

[dvenckus]

The issue happens whether I specify saltenv as 'base' or 'dev' or omit it and allow it to default

[dvenckus]

These are vagrant/virtualbox VMs. I have destroyed and rebuilt the 2018.3.1 VM multiple times to the same end result.

[garethgreenaway]

@dvenckus So far still haven't been able to duplicate. Would it be possible to provide a snapshot of one of the VMs where this is happening?

[dvenckus]

I will try to pull that together for you ASAP

[dvenckus]

In putting together the VM, I removed the 'dev' environment and dumbed it down to just the 'base' environment. That seemed to have no effect. The same errors occurred. I included 5

states in the common init.sls so that running....

salt-call state.sls common

should reproduce the same outcome. I also removed any personal or sensitive info.

I shared a link with you to the file archive using the email address on your account.

[garethgreenaway]

@dvenckus Got the image and taking a look. And I see the errors now. Thanks!

[dvenckus]

I downgraded to version 2017.7.6 and the same problem is happening.

However, version 2017.7.4 is working as expected.

[garethgreenaway]

Thanks. I've narrowed it down to something in your custom module calling pillar.get with the saltenv parameter. Without that parameter it works as expected but with it it does not. With the versions I'll run through a git bisect and try to narrow down the issue. Thanks!

[garethgreenaway]

Quick question. How were you using this setup with 2017.7.4 and the calls to pillar.get? That parameter wasn't available for that function in that release.

[dvenckus]

We've been successfully using it since 2016.3. It's in the documentation. Have been using it in 2017.7.4 as well.

No issues until the recent migration from centos 6 to Ubuntu 18.04

[garethgreenaway]

@dvenckus Apologies, I had checked out the wrong tag. Seeing it working as expected with 2017.7.4 now. Thanks!

[garethgreenaway]

@dvenckus I believe we've found the root cause and with the help of @terminalmage come up with a fix. If you're able to, could you try the patch in either the 2017.7 or 2018.3 pull requests and see if it fixes the issue for you?

[dvenckus]

Goooooooooaaaaaalllll !!!!!
Both versions are working with multiple included states. Well done!