Salt-bootstrap: Bootstrap doesn't handle a bad EPEL mirror very well

[gravyboat]

Over the past day or two one of my coworkers has been getting this error which I'm assuming is due to a crappy EPEL mirror near where they are located:

INFO: Running install_centos_stable_deps()
curl: (7) couldn't connect to host
error: skipping http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm - transfer failed
Retrieving http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
 * ERROR: Failed to run install_centos_stable_deps()!!!

@s0undt3ch What's your opinion on trying to work in some sort of function to handle epel in situations like this? I'm thinking we could just do a sed in /etc/yum.repos.d/epel.repo to replace mirrorlist with baseurl, then refresh the data and try again. There's still the chance it could fail and it will be slower, but at least it gets you around mirrors that aren't working.

[mooperd]

Im also seeing this a lot.

error: skipping http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-2.noarch.rpm - transfer failed
Retrieving http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-2.noarch.rpm
ERROR: Failed to run install_centos_stable_deps()!!!
Error: There was a profile error: Command "ssh -t -t -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oControlPath=none -oPasswordAuthentication=no -oChallengeResponseAuthentication=no -oPubkeyAuthentication=yes -oKbdInteractiveAuthentication=no -i /etc/salt/Andrew.pem -p 22 [email protected] 'sudo /tmp/.saltcloud-2f9b94e8-2c3e-4d72-a046-a2eb93eda802/deploy.sh -c /tmp/.saltcloud-2f9b94e8-2c3e-4d72-a046-a2eb93eda802 -U'" failed. Exit code: 1

[mooperd]

deploy.sh

   2540     if [ "$DISTRO_MAJOR_VERSION" -eq 5 ]; then
   2541         # Use dl.fedoraproject.org to avoid redirect breakage:
   2542         # https://lists.fedoraproject.org/pipermail/users/2012-February/414558.html
   2543         rpm -Uvh --force "http://dl.fedoraproject.org/pub/epel/5/${EPEL_ARCH}/epel-release-5-4.noarch.rpm" || return 1
   2544     elif [ "$DISTRO_MAJOR_VERSION" -eq 6 ]; then
   2545         rpm -Uvh --force "http://download.fedoraproject.org/pub/epel/6/${EPEL_ARCH}/epel-release-6-8.noarch.rpm" || return 1
   2546     elif [ "$DISTRO_MAJOR_VERSION" -eq 7 ]; then
   2547         rpm -Uvh --force "http://download.fedoraproject.org/pub/epel/7/${EPEL_ARCH}/e/epel-release-7-2.noarch.rpm" || return 1
   2548     else
   2549         echoerror "Failed add EPEL repository support."
   2550         return 1
   2551     fi

It appears that epel-release-7-5 rather than 7-2 is the flavour of the month.

[s0undt3ch]

Yes @mooperd you need to update your bootstrap script, that's a know and fixed issue, unrelated to @gravyboat's

[gravyboat]

Just found another thing that needs to be considered. If the 'fastest mirror' plugin is installed it can cause issues regardless of what the baseurl is set to, /etc/yum/pluginconf.d/fastestmirror.conf needs to have the value inside of it swapped to 0 so we ignore the fastest mirror.

[s0undt3ch]

Can these changes be passed on the CLI? Or do we have to change the config files?

[gravyboat]

Fastest mirror can be disabled with –disableplugin=fastestmirror on the call to yum. As far as I'm aware with the URL however it isn't possible (usually because baseurl is actually commented out in favor of mirrorlist).

[drawsmcgraw]

@s0undt3ch Salt Cloud uses bootstrap-salt.sh in order to bootstrap new minions. I ran into @mooperd's problem, which led me to this thread. Given a fresh VM and a yum install salt-cloud, is there an elegant way to get the updated (e.g. non-broken) version of bootstrap-salt.sh?

[mooperd]

@drawsmcgraw -

salt-cloud --update-bootstrap

[s0undt3ch]

Thanks @mooperd.

[drawsmcgraw]

@mooperd Was not aware of that! Thanks!

[jfindlay]

@gravyboat, since repo.saltstack.com, we've been moving away from depending on EPEL, this should not be a problem for saltstack repo installs. RHEL 5, 6, and 7 now do not depend on EPEL, from what I remember, and once the ${_DOWNSTREAM_PKG_REPO} flag is adopted by the RHEL family, EPEL support should be removed from them. This still leaves the problem for installing from EPEL, though.

[samson-jerome]

Hi all,
It seems that the repo has changed again from 7.5 to 7.6 and it needs to be reflected in the bootstrap...
http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-6.noarch.rpm

[mooperd]

Isn't there a way of getting a computer to do this or something?

[jfindlay]

@samson-jerome, this has been updated in the latest bootstrap version, https://github.com/saltstack/salt-bootstrap/releases/tag/v2016.04.18. @mooperd, the problem is that not all RHEL derivatives have the epel-release package available, so the hardcoded url logic needs to be a fallback for those cases. I have not yet submitted this for lack of time. If you would like to do it, that would be fine with me.

[samson-jerome]

@jfindlay: thanks for the update, I saw that there had been an update and our install scripts are working fine.

[vutny]

In the latest stable version (2016.10.25) we construct reliable URL for EPEL release package download, such as:

epel_repo_url="${HTTP_VAL}://dl.fedoraproject.org/pub/epel/epel-release-latest-${DISTRO_MAJOR_VERSION}.noarch.rpm"

It works almost 100% of time because the RPM file is downloaded directly from Fedora's server, not from a mirror.
I think this issue has been resolved and could be safely closed. FYI @rallytime

[gravyboat]

Almost 100% of the time isn't 100% of the time (in the event the Fedora servers are up). Users need to be able to always get the latest release. It was 100% for me until I ran into this issue almost 2 years ago. The mirrors should still be used and then fall back to the base fedora servers only after they fail. Downloading directly from Fedora can be really slow.

[vutny]

Thanks for the feedback! I agree, but I believe we really can't provide the guarantee of any level for external dependency, such as EPEL... Some workarounds may be applied though.

Despite that, nowadays this has been improved a lot, because we download only a small release package - 15KB! (which is always up-to-date, by the way) from Fedora's highly available upstream server.
yum will still use the nearest consistent mirror found, depending on plugins and local configuration of course.

Another good thing here is that we don't need EPEL at all for installing Salt except of one particular case, see issue #724. So, we even can improve the bootstrap script reliability by:

• skipping EPEL configuration where it is unnecessary
• adding retries for epel-release package downloading or a fall-back URL
• keep running if download was unsuccessful, where is a chance that all pre-conditions are already met

@gravyboat What do you think?

[gravyboat]

@vutny I'm fine with that. Proposed improvements are a good idea and should provide enough redundancy to make this a non-issue.

[drawsmcgraw]

@vutny +1 from me for what it's worth to you. I've been bitten by this behavior more times than I'd like to admit.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.