Salt: pillar include stopped working

[nocturo]

Description of Issue/Question

I have used git for ext_pillar for a while now and when upgrading to 2019.2.0 no pillars get rendered for me. As a quick test I downgraded to 2018.3.4 and everything works as expected.

ext_pillar config:

ext_pillar:
  - git:
    - master [email protected]:user/pillar.git:
       - root: pillar
       - privkey: /root/.ssh/id_rsa
       - pubkey: /root/.ssh/id_rsa.pub

This straight up doesn't load up the pillar top.sls if I have root: specified. If I remove the option and put top.sls in git "root" it then loads it and I get error that now it can't find the SLS (as expected since the path is not relative anymore to the specified root in the config which I've been using).

Steps to Reproduce Issue

Configure exit_pillar git pillar with root: and top.sls in the root.

Versions Report

```Salt Version:
Salt: 2019.2.0

Dependency Versions:
cffi: 1.9.1
cherrypy: unknown
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.8
libgit2: 0.24.6
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.24.2
Python: 3.4.9 (default, Feb 5 2019, 14:36:09)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.4.2
ZMQ: 4.1.4

System Versions:
dist: centos 7.6.1810 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-957.1.3.el7.x86_64
system: Linux
version: CentOS Linux 7.6.1810 Core
```

[Ch3LL]

im not able to replicate this with:

ext_pillar:                                                                                                                                                                                          
  - git:                                                                                                                                                                                             
    - master https://github.com/Ch3LL/Ch3LLScripts.git:                                                                                                                                              
      - root: salt/pillars

although i did have to patch mine with https://github.com/saltstack/salt/pull/51304/files but thats because i have a newer pygit version then you do.

Do you see any error's or further information in the debug logs when this occurs?

[nocturo]

OK I was vague as it seems more involved than I first thought. I have some include:s happening in the pillar and it looks like this: https://github.com/nocturo/pillar

If you put that as your ext_pillar, with root:

ext_pillar:
  - git:
    - master https://github.com/nocturo/pillar.git
      - root: pillar

and that's the root cause of this. I've just initially assumed since it errored out due to SLS path when trying top.sls directly at top level that it was working, but it doesn't on that repo with or without root when include: is called to call a pillar.

--- edit
Actually this is not related to git at all, I've just tried cloning that repo into /srv and using pillar_roots:

pillar_roots:
  base:
    - /srv/pillar

and it still doesn't render, so the include with init.sls seems broken?

2019-02-26 23:00:41,597 [salt.template    :120 ][DEBUG   ][25064] Rendered data from file: /srv/pillar/top.sls:
base:
  '*':
    - pillar/test/roottestdir

2019-02-26 23:00:41,598 [salt.loaded.int.render.yaml:68  ][DEBUG   ][25064] Results of YAML rendering:
OrderedDict([('base', OrderedDict([('*', ['pillar/test/roottestdir'])]))])
2019-02-26 23:00:41,608 [salt.template    :59  ][DEBUG   ][25064] compile template: /srv/pillar/pillar/test/roottestdir/init.sls
2019-02-26 23:00:41,608 [salt.utils.jinja :70  ][DEBUG   ][25064] Jinja search path: ['/srv/pillar']
2019-02-26 23:00:41,624 [salt.template    :26  ][PROFILE ][25064] Time (in seconds) to render '/srv/pillar/pillar/test/roottestdir/init.sls' using 'jinja' renderer: 0.015855073928833008
2019-02-26 23:00:41,625 [salt.template    :120 ][DEBUG   ][25064] Rendered data from file: /srv/pillar/pillar/test/roottestdir/init.sls:
include:
  - pillar/test/test

2019-02-26 23:00:41,625 [salt.loaded.int.render.yaml:68  ][DEBUG   ][25064] Results of YAML rendering:
OrderedDict([('include', ['pillar/test/test'])])
2019-02-26 23:00:41,626 [salt.template    :26  ][PROFILE ][25064] Time (in seconds) to render '/srv/pillar/pillar/test/roottestdir/init.sls' using 'yaml' renderer: 0.0004544258117675781

[Ch3LL]

thanks for the additional information. looks like i was able to replicate this and bisect it to fbab73a35c2a36e54ace814f2b45119e320b61e3

which is this PR: https://github.com/saltstack/salt/pull/45269

also here is a docker container to quickly replicate the issue:

1. docker run -it -v ~/git/salt/:/testing/ ch3ll/issues:51832 (where ~/git/salt is a local cloned git repo of salt)
2. salt-master -d; salt-minion -d
3. salt '*' pillar.items (and it will be empty on 2019.2.0 and return pillar on 2018.3.3)

[Ch3LL]

I realized that the reason its failing is because of this:

include:
  - pillar/test/test

if you change that to:

include:
  - pillar.test.test

it will work. Since we were previously supporting the / notation we still need to get this fixed up still though

[nocturo]

Ok, thanks for the insight, not that of a big deal to change to keep using fluorine. Thanks!

[dmurphy18]

Actually this is easy to reproduce and it is breaking the nightly builds when upgraded to 2019.2.0 salt-master.
root@c7master:/usr/lib/python2.7/site-packages/salt# cat /srv/pillar/pkgbuild.sls

set version to build

{% set build_version = '2019_2_0' %}

{% if build_version != '' %}
include:
- .versions.{{build_version}}.pkgbuild
{% endif %}

Problem is that it is not recursively processing the jinja2 derived include file .versions.2019_2_0.pkgbuild

area is in render_pstate:
'''
matched_pstates = []
for sub_sls in state.pop('include'):
if isinstance(sub_sls, dict):
sub_sls, v = next(six.iteritems(sub_sls))
defaults = v.get('defaults', {})
key = v.get('key', None)
else:
key = None

                        try:
                            matched_pstates += fnmatch.filter(self.avail[saltenv], sub_sls)
                        except KeyError:
                            errors.extend(
                                ['No matching pillar environment for environment '
                                 '\'{0}\' found'.format(saltenv)]
                            )

                    for sub_sls in set(matched_pstates):

'''
The matched_pstates += fnmatch.filter(self.avail[saltenv], sub_sls) will come back empty

salt-call --local pillar.items

when minion installed on same machine as master

[nandudheeraj]

Pillar include is not working in the recent version of salt as well
Not using ext pillar. Simple pillar root (file based)
debug mode output on master shows "include" is not being interpreted as a keyword but as a regular key

2020-09-27 00:26:01,147 [salt.template    :62  ][DEBUG   ][2005816] compile template: /srv/salt/pillar/users/import-ops.sls
2020-09-27 00:26:01,148 [salt.utils.jinja :92  ][DEBUG   ][2005816] Jinja search path: ['/srv/salt/pillar']
2020-09-27 00:26:01,149 [salt.template    :31  ][PROFILE ][2005816] Time (in seconds) to render '/srv/salt/pillar/users/import-ops.sls' using 'jinja' renderer: 0.0013480186462402344
2020-09-27 00:26:01,149 [salt.template    :123 ][DEBUG   ][2005816] Rendered data from file: /srv/salt/pillar/users/import-ops.sls:
include:
  - users.ops:
      key: sreusers

2020-09-27 00:26:01,150 [salt.loaded.int.render.yaml:85  ][DEBUG   ][2005816] Results of YAML rendering: 
OrderedDict([('include', [OrderedDict([('users.ops', OrderedDict([('key', 'sreusers')]))])])])
2020-09-27 00:26:01,150 [salt.template    :31  ][PROFILE ][2005816] Time (in seconds) to render '/srv/salt/pillar/users/import-ops.sls' using 'yaml' renderer: 0.0005915164947509766

salt --versions-report
Salt Version:
           Salt: 3001.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: 2.0.6
      gitpython: 3.0.7
         Jinja2: 2.10.1
        libgit2: 0.28.3
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.6.1
         pygit2: 1.0.3
         Python: 3.8.2 (default, Jul 16 2020, 14:00:26)
   python-gnupg: 0.4.5
         PyYAML: 5.3.1
          PyZMQ: 18.1.1
          smmap: 2.0.5
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: ubuntu 20.04 focal
         locale: utf-8
        machine: x86_64
        release: 5.4.0-37-generic
         system: Linux
        version: Ubuntu 20.04 focal