Salt: [BUG] Documented options in minion/master config files don't match minion/master config documentation, and vice versa

[ScriptAutomate]

Description

There are options listed in the conf/minion file that are undocumented in the minion configuration documentation, doc/ref/configuration/minion.rst, and vice versa.

Setup

Clone the latest version of this repository.

Steps to Reproduce the behavior

See the information in Additional context below.

Expected behavior

The options listed in the minion configuration file and the minion configuration documentation should match, or at a minimum the minion configuration documentation should include information regarding all options listed in the conf/minion file. This is the bug.

An improvement/enhancement could be for the conf/minion file to also include all the options in the minion configuration documentation that are currently missing. This would be helpful for users, such as what was experienced in #57931

Additional context

I whipped up these scripts to find all options missing from the conf/minion file that are listed doc/ref/configuration/minion.rst, and vice versa.

Minion audit

#!/usr/bin/env bash
# Format for option names
cat doc/ref/configuration/minion.rst | \
    grep '\-\-\-' -B1 | \
    grep -v '\-\-' | \
    sed 's/``//g' | \
    sort > minion-options-docs.txt

# Grab all config options for minion conf from conf
cat conf/minion | \
    grep ':' | \
    egrep -v "^# |\." | \
    sed 's:#::g' | \
    cut -d':' -f1 | \
    sort | \
    uniq > minion-options-conf.txt

# What's potentially missing from minion conf docs?
diff minion-options-docs.txt minion-options-conf.txt | \
    grep '>' | \
    sed 's:>\ ::g' > settings-missing-from-minion-options-docs.txt

# What's missing from minion conf file?
diff minion-options-docs.txt minion-options-conf.txt | \
    grep '<' | \
    sed 's:<\ ::g' > settings-missing-from-minion-options-conf.txt

This ends up creating two files:

• settings-missing-from-minion-options-conf.txt
• settings-missing-from-minion-options-docs.txt

$ cat settings-missing-from-minion-options-docs.txt | wc -l
8
$ cat settings-missing-from-minion-options-conf.txt | wc -l
71

This seems to say that quite a few options are missing from the minion conf file, and some are missing from the minion configuration documentation.

Missing from conf/minion

Source file: https://github.com/saltstack/salt/blob/master/conf/minion

• conf/minion

always_verify_signature
cache_sreqs
cmd_blacklist_glob
cmd_whitelist_glob
color_theme
decrypt_pillar
decrypt_pillar_default
decrypt_pillar_delimiter
decrypt_pillar_renderers
default_include
default_top
docker.compare_container_networks
docker.update_mine
enable_fqdns_grains
enable_gpu_grains
enable_zip_modules
env_order
extmod_whitelist/extmod_blacklist
fibre_channel_grains
grains_blacklist
grains_cache_expiration
grains_dirs
http_connect_timeout
http_request_timeout
iscsi_grains
lock_saltenv
log_rotate_backup_count
log_rotate_max_bytes
master_sign_key_name
master_tops_first
master_type
master_uri_format
metadata_server_grains
minion_id_remove_domain
modules_max_memory
nvme_grains
on_demand_ext_pillar
optimization_order
pass_to_ext_pillars
pidfile
publish_port
recon_default
recon_max
recon_randomize
retry_dns
retry_dns_count
return_retry_timer
return_retry_timer_max
saltenv
snapper_states
snapper_states_config
source_address
source_interface_name
source_publish_port
source_ret_port
ssh_merge_pillar
state_top
state_top_saltenv
top_file_merging_strategy
transport
use_master_when_local
use_yamlloader_old
verify_master_pubkey_sign
winrepo_cache_expire_max
winrepo_cache_expire_min
winrepo_cachefile
winrepo_dir
winrepo_dir_ng
winrepo_remotes
winrepo_remotes_ng
winrepo_source_dir

Missing from doc/ref/configuration/minion.rst

Source file: https://github.com/saltstack/salt/blob/master/doc/ref/configuration/minion.rst

• doc/ref/configuration/minion.rst

color
environment
event_match_type
key_logfile
output
return
state_aggregate
state_output_profile

Other Thoughts

Are the minion configuration options auto-populated anywhere? If not, it would be a good idea to introduce automation that would autopopulate content in either the minion conf file, or the rst documentation page (or both, ideally). That, or audit to see whether there are currently conf options that aren't included in the minion conf, nor the rst documentation page.

[ScriptAutomate]

I also decided to audit the master related conf and docs, in a similar fashion, after seeing that an issue was previously opened back in 2016 concerning documenting both minion and master configs: #32400

Master audit

#!/usr/bin/env bash
# Format for option names
sed -n 1,5818p doc/ref/configuration/master.rst | \
    egrep '\-\-\-|\*\*\*' -B1 | \
    grep -v ':' | \
    egrep -v '\-\-|\*\*| ' | \
    sed 's/``//g' | \
    sort > master-options-docs.txt

# Grab all config options for master conf from conf
cat conf/master | \
    grep ':' | \
    egrep -v "^# |\." | \
    sed 's:#::g' | \
    cut -d':' -f1 | \
    sort | \
    uniq > master-options-conf.txt

# What's potentially missing from master conf docs?
diff master-options-docs.txt master-options-conf.txt | \
    grep '>' | \
    sed 's:>\ ::g' > settings-missing-from-master-options-docs.txt

# What's missing from master conf file?
diff master-options-docs.txt master-options-conf.txt | \
    grep '<' | \
    sed 's:<\ ::g' > settings-missing-from-master-options-conf.txt

This ends up creating two files:

• settings-missing-from-master-options-conf.txt
• settings-missing-from-master-options-docs.txt

$ cat settings-missing-from-master-options-docs.txt | wc -l
30
$ cat settings-missing-from-master-options-conf.txt | wc -l
114

This seems to say that quite a few options are missing from the master conf file, and some are missing from the master configuration documentation.

Missing from conf/master

Source file: https://github.com/saltstack/salt/blob/master/conf/master

• conf/master

api_logfile
api_pidfile
auth_events
autoreject_file
autosign_file
autosign_timeout
azurefs_update_interval
color_theme
con_cache
default_include
enable_gpu_grains
enforce_mine_cache
event_publisher_niceness
event_return_niceness
ext_job_cache
extmod_whitelist/extmod_blacklist
file_ignore_glob
fileserver_limit_traversal
fileserver_list_cache_time
fileserver_update_niceness
fileserver_verify_config
gitfs_base
gitfs_disable_saltenv_mapping
gitfs_global_lock
gitfs_mountpoint
gitfs_ref_types
gitfs_saltenv
gitfs_saltenv_blacklist
gitfs_saltenv_whitelist
gitfs_update_interval
git_pillar_includes
git_pillar_update_interval
git_pillar_verify_config
hgfs_base
hgfs_branch_method
hgfs_mountpoint
hgfs_remotes
hgfs_root
hgfs_saltenv_blacklist
hgfs_saltenv_whitelist
hgfs_update_interval
http_connect_timeout
http_request_timeout
include
interface
jinja_lstrip_blocks
jinja_trim_blocks
job_cache_store_endtime
log_rotate_backup_count
log_rotate_max_bytes
maintenance_niceness
master_id
master_job_cache
master_pubkey_signature
master_sign_key_name
master_sign_pubkey
master_use_pubkey_signature
max_minions
minion_data_cache_events
minionfs_blacklist
minionfs_env
minionfs_mountpoint
minionfs_update_interval
minionfs_whitelist
mworker_niceness
mworker_queue_niceness
optimization_order
pidfile
pillar_includes_override_sls
ping_on_rotate
presence_events
publish_session
pub_server_niceness
reactor_niceness
req_server_niceness
rest_timeout
roots_update_interval
roster_defaults
rotate_aes_key
s3fs_update_interval
skip_grains
sock_pool_size
ssh_priv_passwd
ssh_scan_timeout
state_top
state_top_saltenv
svnfs_branches
svnfs_mountpoint
svnfs_remotes
svnfs_root
svnfs_saltenv_blacklist
svnfs_saltenv_whitelist
svnfs_tags
svnfs_trunk
svnfs_update_interval
syndic_forward_all_events
syndic_pidfile
tcp_master_publish_pull
tcp_master_workers
transport
transport_opts
userdata_template
use_yamlloader_old
winrepo_branch
winrepo_cachefile
winrepo_insecure_auth
winrepo_passphrase
winrepo_password
winrepo_privkey
winrepo_provider
winrepo_pubkey
winrepo_ssl_verify
winrepo_user
yaml_utf8

Missing from doc/ref/configuration/master.rst

Source file: https://github.com/saltstack/salt/blob/master/doc/ref/configuration/master.rst

• doc/ref/configuration/master.rst

batch_safe_limit
batch_safe_size
client_acl_verify
default_top
event_match_type
event_return_queue_max_seconds
fileserver_events
gitfs_insecure_auth
gitfs_passphrase
gitfs_password
gitfs_privkey
gitfs_pubkey
gitfs_refspecs
git_pillar_insecure_auth
git_pillar_passphrase
git_pillar_password
git_pillar_privkey
git_pillar_pubkey
git_pillar_refspecs
git_pillar_user
gpg_cache
gpg_cache_backend
gpg_cache_ttl
key_cache
key_logfile
netapi_allow_raw_shell
pillar_gitfs_ssl_verify
return
ssh_run_pre_flight
ssh_update_roster

[ScriptAutomate]

Also: according to #32400, the following are completely undocumented at the moment and aren't to be found in the above audits of master/minion config files or config documentation:

auth_mode
bootstrap_delay
cluster_masters
cluster_mode
enable_lspci
http_max_body
http_request_timeout
pillar_version
queue_dirs
search
search_index_interval
serial
sign_pub_messages
sqlite_queue_dir
syndic_event_forward_timeout
syndic_jid_forward_cache_hwm
syndic_max_event_process_time
zmq_filtering

It looks like the best source for all available options is:

• https://github.com/saltstack/salt/blob/master/salt/config/__init__.py

Which, running an audit against all default options for master/minion here led to the following 49 options potentially needing to be documented (or removed from salt/config/__init__.py if they are deprecated?). I do not know whether these can be defined within master/minion conf files:

archive_jobs
beacons_before_connect
cmd_safe
disabled_requisites
discovery
django_auth_path
django_auth_settings
drop_messages_signature_fail
dummy_pub
eauth_tokens
engines
extmod_blacklist
extmod_whitelist
gitfs_fallback
gitfs_user
git_pillar_fallback
id_function
ipc_write_buffer
key_pass
local
log_fmt_jid
minion_jid_queue_hwm
minion_restart_command
minion_sign_messages
multifunc_ordered
password
permissive_acl
python2_bin
python3_bin
regen_thin
renderer_blacklist
renderer_whitelist
require_minion_sign_messages
resolve_dns_fallback
restart_on_error
salt_cp_chunk_size
schedule
scheduler_before_connect
signing_key_pass
ssh_config_file
ssh_sudo_user
state_auto_order
thoriumenv
thorium_interval
thorium_roots
thorium_top
unique_jid
username
winrepo_fallback

[nicholasmhughes]

ran into the need for sqlite_queue_dir today due to use case similar to #45854