Salt: [BUG] [DOCS] repo.saltstack.com incorrect links since SEP22 and needs more review of possible other dead/incorrect links

[mbochenk]

Description
repo.saltstack misses multiple older releases, for example http://repo.saltstack.com/apt/debian/9/amd64/archive/
This causes bootstrap issues, etc (master running on 2019.2.0, for ex) This also originally appears as incorrect GPG keys and confuses troubleshooting

Setup
Bootstrap CentOS7 or Debian9 (other repos might be also affected) with salt-bootstrap script, with for example these settings:
-F -P -I stable 2019.2.0

Steps to Reproduce the behavior
Results
CentOS

error: /tmp/salt-gpg-nMpYnO0T.pub: key 1 not an armored public key.
--
* ERROR: Failed to run install_centos_stable_deps()!!!
Process Process-5:
Traceback (most recent call last):
File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1288, in create
output = self.clouds[func](vm_)
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/opennebula.py", line 1155, in create
ret = __utils__['cloud.bootstrap'](vm_, __opts__)
File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 644, in bootstrap
deployed = deploy_script(**deploy_kwargs)
File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 1762, in deploy_script
if root_cmd(deploy_command, tty, sudo, **ssh_kwargs) != 0:
File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 2372, in root_cmd
retcode = _exec_ssh_cmd(cmd, allow_failure=allow_failure, **kwargs)
File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 2047, in _exec_ssh_cmd
cmd, proc.exitstatus

Debian

Warning: apt-key output should not be parsed (stdout is not a terminal)
--
gpg: no valid OpenPGP data found.
* ERROR: Failed to run install_debian_deps()!!!
Process Process-5:
Traceback (most recent call last):
File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1288, in create
output = self.clouds[func](vm_)
File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/opennebula.py", line 1155, in create
ret = __utils__['cloud.bootstrap'](vm_, __opts__)
File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 644, in bootstrap
deployed = deploy_script(**deploy_kwargs)
File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 1762, in deploy_script
if root_cmd(deploy_command, tty, sudo, **ssh_kwargs) != 0:
File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 2372, in root_cmd
retcode = _exec_ssh_cmd(cmd, allow_failure=allow_failure, **kwargs)
File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 2047, in _exec_ssh_cmd
cmd, proc.exitstatus

Expected behavior
Archived versions either available or point to latest.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: 1.13.2
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: 1.3.8
         Jinja2: 2.7.2
        libgit2: 0.26.3
        libnacl: 1.6.1
       M2Crypto: 0.31.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: 3.7.3
         pygit2: 0.26.4
         Python: 2.7.5 (default, Apr  9 2019, 14:30:50)
   python-gnupg: 0.4.3
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: 0.2.4
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 4.4.164-1.el7.elrepo.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

[mbochenk]

Updated description as I've finally figured out what actually happenned.

[xeacott]

Thanks for the report, @bryceml do you have any insight on this?

Edit, this would be more helpful : https://github.com/saltstack/salt-enhancement-proposals/blob/master/accepted/0022-old-releases.md

This SEP we introduced handles archive releases and the reasoning for it.

[bryceml]

It might be nice if whatever is downloading the key in the bootstrap script throw an error and immediately exit if it gets a 404 for the key. If it's curl, it would be adding the -f flag.

[mbochenk]

I see the notice now.
I am not sure that pinning to minor releases makes any sense without suggesting to use the archive repos too?

In the past I ran multiple times into update issues between minor versions and have found pinning to minor versions required, sometimes during for much longer than 2 releases. In short, I never know if the next update would be for non-archived or already archived minor version as the update might be trivial or blocked because of bugs or other reasons.

[bryceml]

https://github.com/saltstack/salt-bootstrap/pull/1478/files should improve the error message.

[RedXIV2]

Are there plans to add minor versions to the archive? we've been using 2019.2.3 as a fixed version since our incompatibility with salt 3000 is still causing problems. I've seen 2019.2.4 and 2019.2.5 in the archive and 2019.2 also in the parent directory. This would be great to get added also

[OrangeDog]

@RedXIV2 STOP USING 2019.2.3
It has a critical vulnerability with multiple active attacks.

[dhs-rec]

We're also still in the process of evaluating 3000. Since both 3000 AND 2019.2.x had compatibility issues, we still couldn't update from 2018.3.x. But those versions are gone now.
Please restore them.

[mbochenk]

I am in a similar situation here. For me pinning to major version does not work.
There are usually quite big changes between minor versions - either new bugs, incompatible changes or old bugs resurfacing. Sure, sometimes these are not bugs but issues with my code that were working for some time by that point.
As a result I always have to actually plan update between minor versions. In the end I do not update due to hassle unless there is something really urgent or relevant.

[OrangeDog]

To be clear, they aren't gone, they've just been moved.

SEP 22

Further, 2019.2.3 and 2018.3.x have a critical security vulnerability you must patch.

[dhs-rec]

OK, then fix the docs, please: At least on the Windows download page, it points to https://repo.saltstack.com/windows/archive/, which is empty.

And NO, there's nothing we MUST do. However, I understand it's recommended to patch ;-)

[mbochenk]

@OrangeDog and they were present until a couple of days ago and for years in the old location.

I've run multiple times into incompatibilities between minor salt stack versions that make you roll back to previous release. In the past both bugfixes and CVE were mixed in a minor release. This path would be valid if minor releases were only for CVE and maybe some really critical bugs. For example, let's pull https://docs.saltstack.com/en/latest/topics/releases/2018.3.4.html - it is long list that may affect a lot of things (in fact I vaguely remember that I had issues with that particular version and had to roll back).

I understand that there is a planned change to make minor releases less broken in between releases and renaming to 3000 is a part of that initiative. I agree that security is important. However due to how releases for salt stack are (or were since migrating to 3000 versioning/release) I cannot "just" upgrade to a latest minor release, I have to plan a whole activity with testing verifying in multiple environments.

This also basically makes ability to pin to a minor release useless as there will be only two of them anyway.

Right now I've moved to 2019.2.5 and hopefully I can get a windows to attempt migrating to 3000.x sometime before this autumn.

[OrangeDog]

Yes I know, I've had all of the same problems.
I've repeatedly asked to have a single repo with all the currently-supported versions in it, but saltstack apparently don't want to.

All that has happened is unsupported releases have been moved. They have not been removed.
There was a proposal announced, the community approved it, it was announced again, and there is a highlighted warning banner on the download site.

For anyone who didn't patch the CVE: it is almost certain you have already been compromised if you had a publicly-accessible salt master. If you didn't the risk is lower, but any unprivileged user in your network could take control of the entire system.

[sagetherage]

@saltstack/docs-working-group @barbaricyawps @ScriptAutomate

[sagetherage]

Great points, I will update this issue to get the site reviewed as well as other documentation that may have these links. We do have a Working Group and they were going to help with running a script to look at dead links, but not sure if that was decided or helpful here. I am putting this in the next major release to plan and then execute. I will update this issue with more as we discover more and can detail our plan to execute updates.

[dhs-rec]

Thank you, @sagetherage. While you're at it, there's another one which requires a documentation update: #56199.

[rossengeorgiev]

I have similar issues with aftermath of SEP22. Obviously, it broke many links, but really the archive should simply have ALL point releases. That would still achieve the original idea, while also makes link updates and templates simple. See #57640

[ScriptAutomate]

Also related to archive links and installs: #57885