Salt: [FEATURE REQUEST] SaltStack APT package repository reorganization

[heini]

Is your feature request related to a problem? Please describe.
In large setups which lots of machines, admins tend to setup repository caches for the APT repositories used on Debian family systems, for example using the Nexus 3 Repository Manager. For most of the external packages installed on a Debian system it's sufficient to setup one cache in the repository manager, for example one for PostgreSQL, one for SyncThing, one for MongoDB, etc.

However, due to the internal organization of the SaltStack APT repository, which requires to use different URLs for different versions (2019.2, 2019.2.3, ...), OS's (Debian, Ubuntu) or flavours (Py2, Py3) one would need to setup different caches, depending on which version/flavour/os combination of the Salt packages should be installed.

Describe the solution you'd like
With a small reorganization of the SaltStack APT repositories, admins would be able to setup one single cache repository on their Nexus (or Artifactory, whatever) server, but still be able to pin their Salt version to major or minor relases, or flavours like Py2 and Py3.

Instead of encoding all this into the URL, it should go into the "distribution" and "component" fields of the repository description. So instead of using

deb https://repo.saltstack.com/py3/debian/10/amd64/2019.2 buster main

which

1. contains redundant information ("debian/10" and "buster")
2. has the architecture encoded in the URL
3. always uses "main" as component

one should be able to use, for example

deb [arch=amd64] https://repo.saltstack.com/apt buster/2019.2 python3

or

deb [arch=amd64] https://repo.saltstack.com/apt py3/buster/2019.2 main

or ... You get the idea.

This way, admins would only need to setup one single cache repository (since the URL part is always the same), but still be able to setup their system's sources.list so that it allows pinning of salt versions.

[heini]

BTW: Isn't Salt written in pure Python? If so, the package architecture could (or should?) be "all", to indicate that it can be installed on any architecture.

[dmurphy18]

@heini The SaltStack repo's layout has to accommodate not only Debian / Ubuntu but also Redhat and Amazon families and other operating systems, hence the current layout is to accommodate other operating systems and was considered at the time, the best compromise. For example: it is also used for AIX and Solaris releases which are not publicly available, they are available to Enterprise customers.

Hence while the current solution is not optimal for Debian / Ubuntu, it does serve it purpose for more than one family of operating systems.

Also note that while Salt itself is architecturally independent, not all of its dependencies on a particular release are, for example: libsodium on Debian 8, and while Debian 8 is about to End of Life, usually as an OS ages, more and more dependencies have to be updated from those that shipped with the original OS and not all of them can be predicted to be architecturally independent.

Hence closing this feature request for now. If you disagree or have other points or ideas about this which you want to make, please re-open.

[boltronics]

I just came here to report this, and noticed it has already been reported but was dismissed. Thanks @dmurphy18 for taking the time to share some of the logic behind why it was configured this way, but I do disagree with a number of points.

Debian and Ubuntu have their own specific repositories that have their own unique configuration. eg RedHat doesn't a GPG-signed Release file, but it does have a decided SRPMS directory with src.rpm files, etc. Lots of differences to the structure, and the generated structure just comes down to that which the tooling generates. Surely the Debian and Ubuntu-specific tooling could just be configured to operate differently, without impacting the remaining OS repositories?

I understand it was made as a compromise at the time, but there are legitimate hindrances for users opting to use these repositories. By having this ticket closed, you are giving the impression that Salt developers never intend to fix the situation. Personally, I find this a bit disappointing.

My company has our own repositories hosted on S3, generated using apt-ftparchive from the apt-utils package, and some basic wrapper shell scripts (which I'm happy to share if you are curious). Basically, all I have to do is throw the packages in a folder once they pass QA, run one script to take care of generating all the Release files, Contents files, the signing, etc. and then I'm done. It doesn't matter how many versions of Salt I add, they are all added to the repository. I then use apt-cacher-ng to cache everything.

So for example, if using our internal repository I might I see:

salt-minion:
  インストールされているバージョン: 2019.2.5+ds-sp1
  候補:               2019.2.5+ds-sp1
  バージョンテーブル:
 *** 2019.2.5+ds-sp1 500
        500 http://172.31.21.100:3142/mycompany stretch/main amd64 Packages
        100 /var/lib/dpkg/status
     2019.2.4+ds-sp1 500
        500 http://172.31.21.100:3142/mycompany stretch/main amd64 Packages
     2019.2.3+ds-sp1 500
        500 http://172.31.21.100:3142/mycompany stretch/main amd64 Packages
     2017.7.8+ds-sp3 500
        500 http://172.31.21.100:3142/mycompany stretch/main amd64 Packages
     2017.7.7+ds-sp3 500
        500 http://172.31.21.100:3142/mycompany stretch/main amd64 Packages
     2017.7.7+ds-sp2 500
        500 http://172.31.21.100:3142/mycompany stretch/main amd64 Packages
     2017.7.7+ds-sp1 500
        500 http://172.31.21.100:3142/mycompany stretch/main amd64 Packages
     2017.7.7+ds-sp1 500
        500 http://172.31.21.100:3142/mycompany stretch/main amd64 Packages
     2017.7.5+ds-sp1 500
        500 http://172.31.21.100:3142/mycompany stretch/main amd64 Packages
     2017.7.4+ds-sp1 500
        500 http://172.31.21.100:3142/mycompany stretch/main amd64 Packages
     2017.7.3-sp-1 500
        500 http://172.31.21.100:3142/mycompany stretch/main amd64 Packages
     2017.7.2-sp-2 500
        500 http://172.31.21.100:3142/mycompany stretch/main amd64 Packages
     2016.11.2+ds-1+deb9u5 500
        500 http://172.31.21.100:3142/debian-security stretch/updates/main amd64 Packages
     2016.11.2+ds-1+deb9u4 500
        500 http://172.31.21.100:3142/debian stretch/main amd64 Packages

I have Salt deploy a pinning file, and it's trivial to switch to different versions. eg.

Package: salt-cloud salt-common salt-master salt-minion salt-syndic
Pin: version 2019.2.5+ds-sp1
Pin-Priority: 1000
Explanation: Don't upgrade Salt packages until they have passed internal QA.

So if I encounter a critical issue in the last release that was deployed, I can easily roll back with a single command, and (importantly) I still get system notifications of new versions released using cron-apt with apt-listchanges. This official Debian tooling breaks down when a separate repository exists for every new version. From this perspective, the current structure does not serve the intended purpose of working like a Debian repository - which I have to assume is the point or we would all just be using pipy or tarballs.

I understand that it's a convenient option to have separate repositories for the different main releases. Having a 3000 repository include 3000.0, 3000.1, 3000.2, etc. and have a 3001 repository include 3001.0, 3000.1, 3000.2, 3000.3, etc. instead of just the latest for that series would be good. That way, people don't have to pin packages if they always want to trust the latest stable version in a series, but they can still roll back in a pinch without having to mess with sources.list files first. Likewise, would also make sense to have the latest repository continue to have the latest stable version, but also keep all releases from the previous supported series as well, so anyone could pin to whatever they needed.

There are multiple ways to accomplish this, but I believe the cleanest way is to utilise separate _components_. So in https://repo.saltstack.com/py3/debian/10/amd64/latest/dists/buster/ (for example) you have the main component exclusively. You could just add a 3000 component, and a 3001 component, and then use the Packages files therein to reference the correct version. All releases can still co-exist in the same directory.

To illustrate a practical problem I have with the current Salt's Debian repository, here is the current package list I see on one of my hosts (as well as the old Debian-supported Salt packages):

salt-minion:
  インストールされているバージョン: 3001.1+ds-1
  候補:               3001.1+ds-1
  バージョンテーブル:
 *** 3001.1+ds-1 500
        500 http://172.31.21.100:3142/saltstack-3001-stretch stretch/main amd64 Packages
        100 /var/lib/dpkg/status
     2016.11.2+ds-1+deb9u5 500
        500 http://172.31.21.100:3142/debian-security stretch/updates/main amd64 Packages
     2016.11.2+ds-1+deb9u4 500
        500 http://172.31.21.100:3142/debian stretch/main amd64 Packages

From /etc/aptsources.list.d/saltstack.list:
deb http://repo.saltstack.com/py3/debian/9/amd64/3001 stretch main

From our apt-cacher-ng config:
Remap-saltstack-3001-stretch: /saltstack-3001-stretch ; http://repo.saltstack.com/py3/debian/9/amd64/3001

We do not deploy new packages to production without prior testing in our staging environment, so we pin a Salt version when bringing up a new host, before Salt is ever installed. In our recent testing with Salt's repository, since 3001.1 just came out today, this left us freshly launched machines launched to production in a broken state, with errors like:

          ID: salt-minion
    Function: pkg.installed
      Result: False
     Comment: Problem encountered installing package(s). Additional info follows:

              errors:
                  - Running scope as unit: run-rd35eca5cafd64e8a85f531fe5b36281e.scope
                    E: Version '3001+ds-1' for 'salt-minion' was not found
     Started: 14:18:59.813620
    Duration: 346.894 ms
     Changes:   

We could instead have apt-cacher-ng point to https://repo.saltstack.com/py3/debian/10/amd64/archive/3001.1/ but then obviously our system monitoring software will not inform us of new software releases. It's also a bigger system change to deploy a new apt sources file every time we want to update a release. Needing to change that and re-run apt-get update every time a new release comes out seems way too invasive! This assumes that people generally try to be up to date, which I think is the common use case.

A couple of side notes here:

Having the latest stable point-release in "archive" is a misleading place to store it, as Debian repository conventions indicate that archived contents are old EOL software that is no longer supported. I get that you could argue that you won't "maintain" that version any more because going forward you would only release new versions, but that's only half of the story and I still found this unexpected - it wasn't until I looked closely at the Package Repo page URL from the docs that I discovered where it was hiding.

Another problem I see is that we have 3001 for one particular release, and then 3001.1 for the next. It is not inconsistent. Anyone who wants to write a script to regularly scrape and automatically detect when a new "archive" point-release is made needs to take this inconsistency into consideration. While you might suggest that it is not a big deal, it doesn't seem like it was well thought out, and could be improved going forward. Looking at the release history, 3000 is the first release in the history of Salt to have such inconsistency with the version both before and after it.

Regarding the package architecture comments from the prior post, in my example above, I've mostly just rebuilt Salt's packages as is, maybe with some patches where things are broken, hence the architecture is amd64 (which is the only platform I currently manage, but wouldn't be surprised if we get some ARM EC2 instances in the near future so I'll probably hit this problem eventually also).

It took me a while to figure out what you were trying to say with your example, as I didn't understand why it would matter if in the future a package became architecturally dependent or not - it doesn't prohibit you from packaging things correctly now, and then changing them in future later as required, so I couldn't see the logic behind your response.

But then I remembered, Salt not only has separate repositories for releases, but also for architectures! This is definitely not normal, and it doesn't need to be this way. Take for example, the Bash package in the Debian repository:

http://ftp.debian.org/debian/pool/main/b/bash/

You can see everything is all in the same directory. It doesn't matter what architecture it is. You could have some_deb_package-1.0-0+salt_all.deb or whatever in there, no problem. Advantages of this approach come from caching, as well as consistency through meeting user expectations. As it stands, having separate packages for every architecture cached or mirrored locally wastes space unnecessarily. It also wastes your time as it means you need to spend additional time building packages.

This applies for distribution-specific releases too. We don't need separate repositories, as it is what is in the Releases files (under the aptly named dists folder) that matter. You don't need Debian /py3/debian/10, you just need /py3/debian with the distribution specified as the suite on the sources.list line - which we already have to do anyway.

I would even go so far as to say that the whole py3 thing is questionable, as you could have made stretch-py3 (for example) an alternative suite for supported distributions during the transition phase. Then when someone wants to upgrade from Stretch to Buster, they can just search/replace "stretch" with "buster" in their sources.list files and be on their way, instead of having to modify repository URLs.

One final issue I have with the repositories as they are currently: The content-type header is set to application/octet-stream which means it cannot be inspected easily in a web browser. eg. try clicking on the Release file in both of these URLs:

http://ftp.au.debian.org/debian/dists/buster/
https://repo.saltstack.com/py3/debian/10/amd64/latest/dists/buster/

I trust there is enough here to justify re-opening this ticket, if not for performance, then for usability. Let me know if there are any questions, or if I can be of assistance in the restructure.

Edit: Fixed a typo and formatting.

[boltronics]

By the way @dmurphy18, is the code used to generate the existing repository structure published somewhere we can look at (and maybe submit PRs for)?

[heini]

@boltronics:

Regarding the package architecture comments from the prior post, in my example above, I've mostly just rebuilt Salt's packages as is, maybe with some patches where things are broken, hence the architecture is amd64 (which is the only platform I currently manage, but wouldn't be surprised if we get some ARM EC2 instances in the near future so I'll probably hit this problem eventually also).

In fact, it's just the repo that (wrongly) claims it was amd64 only. The packages themselves are of architecture "all". If new releases come out, I usually download the packages I need, upload them to my Nexus Server and use them as-is on i386 and arm64.

[dmurphy18]

@boltronics You can find the repo's to build Salt here https://github.com/saltstack/salt-pack-py3 and https://github.com/saltstack/salt-auto-pack. But note, experimenting with Tiamat https://gitlab.com/saltstack/pop/tiamat going forward

[boltronics]

Nice! Thanks @dmurphy18 for those links, that's great. I look forward to giving the salt-stack-py3 repository in particular a read.