Salt: permit file_roots to be defined as a relative path

[khaije1]

Please either
1) alter file_roots to respect a variable that _can_ be defined locally, such as root_dir or config_dir or using a new variable, (something like), file_roots_rela which would operate similarly to root_dir
-or-
2) add a complimentary means to define relative paths, that either defers entirely to file_roots or appends it's config to it

Example Use Case:
I'm creating a git repo that contains all the settings needed to manage my infrastructure. Using, salt-ssh -c ./etc/salt , I'm part of the way there. This is presently working except that - because file_roots requires an absolute path, it's paths must be altered with each clone or repo move.
This is further desirable because it can be stored, controlled and tracked using gitolite, handed off to junior admins as a decentralized infrastructure control-center kit and subsequently secure-deleted from a machine.

Note to the reader:
Sensitive data will be not stored in plain text. It is encrypted or otherwise obfuscated.
For example, the salt-ssh key and the GPG keys used are defined relative to the userhome, (using tilde, '~'), making the paths and the ensuing capabilities localized according to the "secrets" possessed by the invoking user.

[basepi]

This would make sense, thanks for the request!

[davidjb]

+1 for this. Essentially, I'd like to keep a Saltfile at the root of my repository, place my salt configuration somewhere in a subdirectory, and have all paths be correctly relative to avoid any and all absolute paths.

A recent change now allows file_roots to be relative, but relative to the pwd when you are running salt-ssh. This change happened recently, somwhere between 2014.1 and a commit on the 2014.7 branch, but I'm not sure where.

With how this now stands, I had expected any paths to be relative to the file they are defined in. However, they're relative to where the salt callable is being run. So, if I had a /home/david/salt/master containing:

file_roots:
  base:
    - ./roots/states
    - ./roots/formulas/mysql-formula

I expected that these paths would/should be resolved to be relative to the file they're in, resolving to files/directories next to this file (eg /home/david/salt/roots/states and /home/david/salt/roots/formulas/mysql-formula respectively).

At present, they're resolved to be relative to wherever I'm running salt-ssh, resulting in states that can't be found.

$ salt-ssh -c /home/david/salt  '*' state.show_sls mysql
example:
    - No matching sls found for 'mysql' in env 'base'

If I cd to the given directory, then the paths are resolved correctly, showing the paths are relative to where I execute salt-ssh.

$ cd /home/david/salt
$ salt-ssh -c .  '*' state.show_sls mysql
example:
    ... long mysql state YAML output ...

Can this issue be changed to a bug report, rather than a feature?

[khaije1]

FWIW I expected paths to be relative to the salt directory, (ie. where the master config file resides), by default.

I feel this is the most uniform and unambiguous option.
@davidjb would this satisfy your use case?

[davidjb]

@khaije1 Having paths relative to the file where file_roots is specified makes sense. Relative to the Salt configuration directory would work in that case, assuming the master config can never be specified as being outside the config directory.

[mariodpros]

I would suggest also to add support for environment variables and ~.

file_roots:
  base:
    - $HOME/salt/roots/states
    - ~/salt/roots/formulas/mysql-formula

[davidjb]

+1 to @mariodpr's suggestion! Both would help for sharing states/formulas between different configurations.

[pykler]

+1

@davidjb comment uptop regarding paths relative to salt-call or salt-ssh is being called is correct in most cases except for extension_modules. Relative paths just don't seem to work at all for that.

Update: seems extension_modules is special as it is already relative just leave out the ./ at the beginning. Referenced in the docs.

Update Again: My last update is wrong :( it is prepended with root_dir which is a problematic feature to mess with.

[3add3287]

Potentially related to #7883

[khaije1]

Just wanted to bump this. It's still coming up as a hurdle. The latest example is with portable Salt-provisioner configurations in Packer ... a file_roots path in one circumstance is often invalid in another and thus far requires the maintenance and frequent swapping of largely redundant path'ing configuration which often varies only by the path prefix.

This should be considered a significant violation of the principle of least surprise, as a distributed work-flow hurdle, and as a paper-cut for those new to Salt when seeking to create and share examples which allow them to reason about systems at a chosen level of abstraction.

_I think allowing/setting root_dir to be prepended, (where file_roots is a rela-path), would be a really lovely and elegant solution to my issue._

I'm using root_dir: . in some projects for a while now and have found that it works great!

Please consider marking this for the next release?

[baby-gnu]

Hello,

I'm configuring salt-ssh to replace dish, clusterssh, pssh or mussh I used before.

This point is really annoying.

I setup config_dir in ~/.salt/Saltfile and then I defined the following in my ~/.salt/master:

# -*- yaml -*-

root_dir: .salt
pki_dir: pki
cachedir: cache
sock_dir: run
pidfile: pids

log_file: logs/master.log
key_logfile: logs/key.log

extension_modules: extmods

rosters:
  - rosters

file_roots:
  base:
    - states

pillar_roots:
  base:
    - pillars

# Salt-SSH specific
ssh_log_file: logs/ssh.log
ssh_use_home_key: True

# Global level logged in file
log_level_logfile: debug

I wanted the file_roots to be relative to root_dir.

Any news on this issue?

Regards.

[baby-gnu]

I was looking at the source code and here are my thougths:

• we could define utility functions to expand paths in salt.utils.path
  
  
  
  • expand_path when the caller expect a single file
  
  
  • expand_glob_path when the caller expect a list
  
  
• each one take the optional argument root_dir and apply the same logic than prepend_root_dir if it's not None (better to delegate to a dedicated function like prepend_root_dir)
• use os.path.expanduser to expand the user HOME directory
• handle file://, for example file:///~user/ should be the same as just ~user/, os.path.expanduser do not handle file://
• every code dealing with configurable path will use expand_path or expand_glob_path

What do you think about this proposal?

[baby-gnu]

The non expansion of ~ for file:// URI is normal according to RFC8089.

[khaije1]

What about the possibility of specifying paths as: %(root_dir)s/my/relative/path? It wouldn't solve all the issues/questions raised but would be a nice step in this direction. I'm just not clear on where this type of syntax is supported and where it isn't.

[damon-atkins]

Might help the salt team if you provide some examples. I suspect your all talking about master-less implementations Or your running a master on a local work station for testing before pushing the code up to your git repo?

[baby-gnu]

@damon-atkins I have 2 scenarios.

The first is described in #issuecomment-365304565.

The second is to use salt as a master-less provisionning for cloud images where I can't use salt-ssh (in a chroot for example): I would like to be able to mount a directory in my chroot and make the provisionning working with salt-call.

For example, I want:

1. a /home/salt-provisionning directory with a hierarchy like

.salt/
├── cache
├── logs
├── master
├── pillar
│   ├── chroots.sls
│   └── top.sls
├── pki
└── state
    ├── openssh
    │   ├── auth_map.sls
    │   ├── auth.sls
    │   ├── banner.sls
    │   ├── client.sls
    │   ├── config_ini.sls
    │   ├── config.sls
    │   ├── defaults.yaml
    │   ├── files
    │   │   ├── banner
    │   │   ├── fire_banner
    │   │   ├── ssh_config
    │   │   ├── sshd_config
    │   │   └── ssh_known_hosts
    │   ├── init.sls
    │   ├── known_hosts.sls
    │   ├── map.jinja
    │   └── moduli.sls
    ├── ssh_keys
    │   ├── bar.pub
    │   ├── foo.pub
    │   └── quux.pub
    ├── systemd
    │   └── timesyncd
    │       └── files
    │           └── chroots
    │               └── timesyncd.conf
    └── top.sls

1. define the following configuration

# -*- yaml -*-

root_dir: ~/.salt
pki_dir: pki
cachedir: cache
sock_dir: run
pidfile: pids

log_file: logs/master.log
key_logfile: logs/key.log

# Global level logged in file
log_level_logfile: debug

# Configure formulas
file_roots:
  base:
    - state

pillar_roots:
  base:
    - pillar

1. mount that /home/salt-provisionning under /mnt/rootfs/root
2. run chroot /mnt/rootfs salt-call --local -c /root/.salt

This way, everything is self contained in a directory, it's easier to maintain and share.

Regards.

[basepi]

@gtmanfred Mentioning you so someone who still works at saltstack is on this issue

[baby-gnu]

Hello.

Just a note for people following this issue, my proposal is a little bit more complicated than I expected because of the way the tests are done.

But I'm still working on it 😉(on my spare time).

Regards.

[max-arnold]

Related issue: #32578 (Master ignores root_dir config value on startup)

[baby-gnu]

Thanks @max-arnold, I'm waiting for salt developpers to agree with my second proposal before taking quite some time to implement it and I think I'll need some advice about the way to do it.

Regards.

[network-shark]

Wasted a lot of time , until I found this bug... . please fix this ! I can't imagine nobody cares.

[baby-gnu]

Hello @network-shark

I really care about this but I haven't the skills to implement #51192 which looks to me a dependency for this feature. When it will be done, I can redo my #49140 PR.

Thanks for keeping this issue alive.

Regards.