Salt-bootstrap: Bootstrap script fails to find py-salt package on FreeBSD 10

[boyonwheels]

Description of Issue/Question

The boot-strap script is failing on my most recent install. It has worked without issue in the past.

Setup

New install of FreeBSD 10.3-RELEASE-p11 system.

Steps to Reproduce Issue

Run the bootstrap script as instructed.

fetch -o bootstrap-salt.sh https://bootstrap.saltstack.com
sudo sh bootstrap-salt.sh

The script errors out on the following bits of code because there are no sysutils/py-salt packages in any of the repositories.

install_freebsd_9_stable() {
    # shellcheck disable=SC2086
    /usr/local/sbin/pkg install ${FROM_SALTSTACK} -y sysutils/py-salt || return 1
    return 0
}

install_freebsd_10_stable() {
    install_freebsd_9_stable
}

pkg search salt reveals that there are two potential packages: "py27-salt-2016.3.3" and "py27-salt-2016.11.0". Changing the relevant bit of code to reference "py27-salt-2016.11.0" fixed the issue. Although, I'm assuming this is less than ideal and not by design.

This is what fixed the script:

/usr/local/sbin/pkg install ${FROM_SALTSTACK} -y py-salt-2016.11.0 || return 1

[rallytime]

@boyonwheels Thank you for reporting this error. I am able to reproduce it as well. While I agree that placing the package name directly in the script like that is not the correct solution here (but a good workaround for now!), I am not sure what is the right solution off the top of my head as I'm not a heavy freebsd user.

[boyonwheels]

My assumption is that at some point there was a FreeBSD package for py-salt that pointed at the latest stable release within Saltstack's public repo. But, at some point, that package or link was broken. So, maybe it's not so much a problem with the bootstrap script, but on Saltstack's release engineering side? However, I don't know who, or where, to follow-up on that idea.

[rallytime]

Yeah, I will need to look at this a little more closely. It is my understanding that the freebsd packages install from the freebsd port that @cedwards maintains, and the latest 2016.11.1 package is definitely there: https://www.freshports.org/sysutils/py-salt/

Unless we are doing something differently that is not documented here on repo.saltstack.com. Perhaps @UtahDave knows about that issue. But it might be something to do with the FROM_SALTSTACK variable. I will need to look there.

[rallytime]

I looked into this a little more closely, and I don't think this is an issue with the bootstrap script. Something seems to have changed in the way we're linking FreeBSD packages with repo.saltstack.com, or something along those lines. This was working during testing of the last stable release of the bootstrap script, but is no longer working with this same version of the bootstrap script.

If we change sysutils/py-salt to just py27-salt on this line as Salt's installation instructions suggest, we do get a successful Salt installation, but it is for version 2016.11.0. The most recent version is 2016.11.1. This is the version currently in the fresh-ports link as well, as I mentioned above. Since the sysutils/py-salt link has been working for quite some time, I don't think this is the right fix. I think something else is going on on the salt packaging release side.

@dmurphy18 I am not sure if you are familiar with how the FreeBSD packages work or not.

ping @UtahDave - do you know what might be different for locating FreeBSD packages?

[cro]

ping @cedwards also.

[dmurphy18]

@rallytime For FreeBSD we currently mirror the FreeBSD builds available, hence repo.saltstack.com has only what is currently available. The FreeBSD builds are pulled form whatever @cedwards has currently built and @UtahDave just pulls in the latest from the mirror when a new version becomes available.

Note, the 2016.11.1 release was only just before the Christmas break and @cedwards was on vacation before 2016.11.1 was released, hence the new Salt 2016.11.1 for FreeBSD maybe not yet have been built.

[rallytime]

@dmurphy18 Do you know if this is automated? Because the version in freshports is 2016.11.1, but installing with pkg install py27-salt as our installation instructions state, it only installs 2016.11.0. It would be great if we could get this versioning sorted out in a more automated way.

[cedwards]

I'll see if I can schedule some time with @dmurphy18 and we can get this
sorted. Right now it is a manual process.

On Wed, Jan 4, 2017 at 1:23 PM, Nicole Thomas [e-mail]
wrote:

@dmurphy18 https://github.com/dmurphy18 Do you know if this is
automated? Because the version in freshports
https://www.freshports.org/sysutils/py-salt/ is 2016.11.1, but
installing with pkg install py27-salt as our installation instructions
state, it only installs 2016.11.0. It would be great if we could get this
versioning sorted out in a more automated way.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt-bootstrap/issues/996#issuecomment-270476648,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgonfkaFmRjlPs-97v7D8fZVPgftIayks5rO__RgaJpZM4LMakG
.

[rallytime]

Awesome! Thanks guys!

[pmather]

I have a similar issue. On FreeBSD/amd64 11-STABLE, I can bootstrap the "stable" version of Salt (which installs from the FreeBSD package repository) but not bootstrap the "git" version, which fails similar to the way reported above.

Looking at the package, it appears that the port origin is registered incorrectly in the SaltStack repo:

py27-salt-2016.11.0
Name           : py27-salt
Version        : 2016.11.0
Installed on   : Sat Jan  7 00:44:00 2017 UTC
Origin         : sysutils/py-salt-2016.11.0
Architecture   : freebsd:11:*
Prefix         : /usr/local
Categories     : sysutils python
Licenses       : APACHE20
Maintainer     : [email protected]
WWW            : https://pypi.python.org/pypi/salt
Comment        : Distributed remote execution and configuration management system
Options        :
    RAET           : off
    ZEROMQ         : on
Annotations    :
    cpe            : cpe:2.3:a:saltstack:salt:2016.11.0:::::freebsd11:x64
    repo_type      : binary
    repository     : SaltStack
Flat size      : 47.3MiB
Description    :
Salt : Remote Execution and State Manager
======================================

Salt is a powerful remote execution and state manager that can be
used to administer servers in a fast and efficient way.

WWW: https://pypi.python.org/pypi/salt
WWW: http://saltstack.org/

Note the origin is set to sysutils/py-salt-2016.11.0, not sysutils/py-salt, which the bootstrap script is attempting to install. It is not normal to encode the version number in the origin metadata of a package.

I build FreeBSD packages for my 11-STABLE system locally using Poudriere. The package built for the current version in FreeBSD ports records the origin correctly:

py27-salt-2016.11.1
Name           : py27-salt
Version        : 2016.11.1
Installed on   : Thu Jan  5 19:48:15 2017 EST
Origin         : sysutils/py-salt
Architecture   : freebsd:11:*
Prefix         : /usr/local
Categories     : sysutils python
Licenses       : APACHE20
Maintainer     : [email protected]
WWW            : https://pypi.python.org/pypi/salt
Comment        : Distributed remote execution and configuration management system
Options        :
    RAET           : on
    ZEROMQ         : on
Annotations    :
    cpe            : cpe:2.3:a:saltstack:salt:2016.11.1:::::freebsd11:x64
    repo_type      : binary
    repository     : local
Flat size      : 47.4MiB
Description    :
Salt : Remote Execution and State Manager
======================================

Salt is a powerful remote execution and state manager that can be
used to administer servers in a fast and efficient way.

WWW: https://pypi.python.org/pypi/salt
WWW: http://saltstack.org/

I suspect this is why the version from the FreeBSD repository installs but not the one in the SaltStack repository. However the latter is being built, it differs from the normal FreeBSD packaging conventions, at least as I understand them.

[amontalban]

Hey guys,

I just stumbled with this too, a couple of things:

1. I'm trying to install a GIT version for testing something that was working in 2016.3.2 and using:

sh bootstrap-salt.sh git v2016.3.2

For me doesn't make sense that if I'm requesting a GIT install it installs the package from repository, why not install just the requirements and then build it from GIT?

In order to get a list of dependencies you can run something like this:

root@devserver-freebsd-103:~ # pkg search -r FreeBSD -R -d sysutils/py-salt | grep -i origin | sed -e 's/^[[:space:]]*//' | tail -n +2 | awk -F\" '{print $2}'
lang/python27
devel/py-setuptools27
www/py-requests
devel/py-enum34
devel/py-yaml
misc/py-progressbar
security/py-pycrypto
devel/py-Jinja2
devel/py-botocore
devel/py-msgpack-python
net/py-libcloud
devel/py-futures
textproc/py-MarkupSafe
www/py-tornado
net/py-pyzmq
lang/python2

And either iterate over the list or install all at once adding | tr '\n' ' ' to the end so you can get a list of the packages.

2. As @pmanther pointed out the problem is that the port is not named correctly in your repository:

root@devserver-freebsd-103:~ # pkg search -Q repository sysutils/py-salt
sysutils/py-salt
Repository     : FreeBSD [pkg+http://pkg.FreeBSD.org/FreeBSD:10:amd64/quarterly]
Comment        : Distributed remote execution and configuration management system
sysutils/py-salt-2016.11.0
Repository     : SaltStack [http://repo.saltstack.com/freebsd/FreeBSD:10:amd64/]
Comment        : Distributed remote execution and configuration management system

I think best would be to swap it back to sysutils/py-salt, but in case you want to keep it you should get the proper package name with this:

root@devserver-freebsd-103:~ # pkg search -r SaltStack sysutils/py-salt
sysutils/py-salt-2016.11.0     Distributed remote execution and configuration management system

And then install that instead.

3. I'm open to help in this ticket as we use FreeBSD and SaltStack a lot and also have experience contributing to ports

Thanks!

[cedwards]

I can fix the issue with the package naming--that one was my fault. I just
need to get a minute with @dmurphy18.

On Thu, Jan 19, 2017 at 9:43 AM, Andres Montalban [e-mail]
wrote:

Hey guys,

I just stumbled with this too, a couple of things:

1. I'm trying to install a GIT version for testing something that was
working in 2016.3.2 and using:

sh bootstrap-salt.sh git v2016.3.2

For me doesn't make sense that if I'm requesting a GIT install it installs
the package from repository, why not install just the requirements and then
build it from GIT?

In order to get a list of dependencies you can run something like this:

root@devserver-freebsd-103:~ # pkg search -r FreeBSD -R -d sysutils/py-salt | grep -i origin | sed -e 's/^[[:space:]]*//' | tail -n +2 | awk -F\" '{print $2}'
lang/python27
devel/py-setuptools27
www/py-requests
devel/py-enum34
devel/py-yaml
misc/py-progressbar
security/py-pycrypto
devel/py-Jinja2
devel/py-botocore
devel/py-msgpack-python
net/py-libcloud
devel/py-futures
textproc/py-MarkupSafe
www/py-tornado
net/py-pyzmq
lang/python2

And either iterate over the list or install all at once adding | tr '\n'
' ' to the end so you can get a list of the packages.

2. As @pmanther pointed out the problem is that the port is not named
correctly in your repository:

root@devserver-freebsd-103:~ # pkg search -Q repository sysutils/py-salt
sysutils/py-salt
Repository : FreeBSD [pkg+http://pkg.FreeBSD.org/FreeBSD:10:amd64/quarterly]
Comment : Distributed remote execution and configuration management system
sysutils/py-salt-2016.11.0
Repository : SaltStack [http://repo.saltstack.com/freebsd/FreeBSD:10:amd64/]
Comment : Distributed remote execution and configuration management system

I think best would be to swap it back to sysutils/py-salt, but in case you
want to keep it you should get the proper package name with this:

root@devserver-freebsd-103:~ # pkg search -r SaltStack sysutils/py-salt
sysutils/py-salt-2016.11.0 Distributed remote execution and configuration management system

And then install that instead.

3. I'm open to help in this ticket as we use FreeBSD and SaltStack a lot
and also have experience contributing to ports

Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt-bootstrap/issues/996#issuecomment-273828859,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgonTtnqPi3HxUWmNlk6hnAJVhUYZtAks5rT5K5gaJpZM4LMakG
.

[amontalban]

Hey guys,

Another thing I just noticed by using bootstrap on FreeBSD (I normally use FreeBSD packages, but need some patches in 2016.11 branch) is that it leaves file /usr/local/etc/pkg/repos/saltstack.conf behind and it's making conflict with others packages because SaltStack repository have older packages and I'm using pkg.latest.

For example with package libxslt, what I'm thinking is that if you are using script to bootstrap the GIT version it should remove the file in install_freebsd_git_post().

I can do a PR to address this issue and also the install/deinstall of package just to get the startup scripts [1], let me know if you need it.

[1] https://github.com/saltstack/salt-bootstrap/blob/develop/bootstrap-salt.sh#L4929-L4937

Thanks!

[rallytime]

@amontalban If you want to submit something to fix the two issues you brought up here, that would be great. (git installation and the satlstack.conf conflict) that would be great.

However, I don't want to change the name of the package. That was just an issue on the packaging end, rather than bootstrap, that @cedwards and @dmurphy18 should be working out.

[amontalban]

@rallytime great thanks, I have my PR almost ready but I just saw this commit (2025bc874bf9d6ec6e1bb08c252f9a9ac684b166) and seems that we are installing SaltStack package from FreeBSD repository for FreeBSD 11 as @pmather said above.

Should I change it to install from SaltStack repository like FreeBSD 9/10?

Thanks!

[rallytime]

Hi @amontalban - Thanks for submitting your fix and apologies for not responding sooner. Looks like you figured it out, but yeah, that would be the correct thing to do.

[cedwards]

Sorry for the delay, but as of this morning 2016.11.2 is available in ports / pkgng. I have submitted these updates to @dmurphy18 and @UtahDave to refresh repo.saltstack.com as well.

This update fixes the pkg name mentioned previously and adds pidfile support to the service scripts.

[rallytime]

Thank you for fixing this up and for updating this issue @cedwards!

[rallytime]

@boyonwheels This should now be resolved as the packages are correctly named and updated. Is this issue ok to close?

[rallytime]

I can no longer reproduce this issue. Since we haven't heard back here, I am going to close this. If this pops up again, let us know!

[boyonwheels]

Sorry I missed this notification!

It's now timing out when I try to update FreeBSD pkgs. I don't know if this is related to the above or a new issue.

(If I try to manually access the URL through a web browser I get a 503 error from the web server)

pkg -d update
DBG(1)[1293]> pkg initialized
Updating FreeBSD repository catalogue...
DBG(1)[1293]> PkgRepo: verifying update for FreeBSD
DBG(1)[1293]> Pkgrepo, begin update of '/var/db/pkg/repo-FreeBSD.sqlite'
DBG(1)[1293]> Fetch: fetching from: http://pkgmir.geo.freebsd.org/FreeBSD:10:amd64/quarterly/meta.txz with opts "i"
DBG(1)[1293]> Fetch: fetching from: http://pkgmir.geo.freebsd.org/FreeBSD:10:amd64/quarterly/packagesite.txz with opts "i"
FreeBSD repository is up to date.
Updating SaltStack repository catalogue...
DBG(1)[1293]> PkgRepo: verifying update for SaltStack
pkg: Repository SaltStack load error: access repo file(/var/db/pkg/repo-SaltStack.sqlite) failed: No such file or directory
DBG(1)[1293]> PkgRepo: need forced update of SaltStack
DBG(1)[1293]> Pkgrepo, begin update of '/var/db/pkg/repo-SaltStack.sqlite'
DBG(1)[1293]> Fetch: fetching from: https://repo.saltstack.com/freebsd/FreeBSD:10:amd64//meta.txz with opts "i"
Fetching meta.txz: 100% 560 B 0.6kB/s 00:01
DBG(1)[1293]> PkgRepo: extracting meta of repo SaltStack
DBG(1)[1294]> PkgRepo: extracting signature of repo in a sandbox
DBG(1)[1293]> Fetch: fetching from: https://repo.saltstack.com/freebsd/FreeBSD:10:amd64//packagesite.txz with opts "i"
DBG(1)[1293]> Fetch: fetching from: https://repo.saltstack.com/freebsd/FreeBSD:10:amd64//packagesite.txz with opts "i"
Fetching packagesite.txz: 100% 0 B 0.0kB/s - stalled -
pkg: https://repo.saltstack.com/freebsd/FreeBSD:10:amd64//packagesite.txz: Operation timed out
Unable to update repository SaltStack
Error updating repositories!

[boyonwheels]

I checked in on this again today and it appears the server is now returning data, however, the Saltstack repo is giving me 2016.11.2 instead of 2016.11.3. The FreeBSD project's repo has 2016.11.3. Not sure why the Saltstack repo is giving me the older version!

salt --version
salt 2016.11.2 (Carbon)

Apologies, again, I don't know if I should be opening this as a new bug.

[rallytime]

Hi @boyonwheels the first issue you were seeing has been fixed. Sounds like we need to update the freebsd packages on repo.saltstack.com. Thanks for bringing this to our attention. We'll get it fixed.

@Ch3LL, @dmurphy18, and @UtahDave FYI

[amontalban]

@rallytime to be honest I don't see the added value of having a SaltStack repository for FreeBSD packages, specially when FreeBSD repository always have more recent version than SaltStack repository.

Additionally FreeBSD do some changes from time to time like https://github.com/freebsd/freebsd-ports/commit/c224eda3d2439a12b0b01965a302a254562b3fc6 that might affect people using SaltStack.

So I would recommend to not have a custom repository for FreeBSD, or if you still want to have it I can help you to get an automated pipeline with Jenkins when a change is detected to SaltStack port or it's dependencies in FreeBSD repository. We have this in place to build custom versions of some ports.

Thanks!