Salt-bootstrap: No dependencies installation function found. Exiting...

Created on 19 Jul 2016  路  27Comments  路  Source: saltstack/salt-bootstrap

Description of Issue/Question

bootstrap fails to install salt-minion on Red Hat Enterprise Linux Server 6.8 Santiago

Steps to Reproduce Issue

salt-cloud succesfully created a virtual machine on a vmware datacenter but failed to install salt-minion using bootstrap script. So I logged in to this newly created machine and run the deploy.sh script manually:
./deploy.sh -H "http://xxx.xxx.xxx:8080" -D

  • INFO: Command line: './deploy.sh -H http://xxx.xxx.xxx.xxx:8080 -D'

    • INFO: System Information:

    • INFO: CPU: GenuineIntel

    • INFO: CPU Arch: x86_64

    • INFO: OS Name: Linux

    • INFO: OS Version: 2.6.32-642.el6.x86_64

    • INFO: Distribution: # RedHatEnterpriseServer 6.8

    • DEBUG: Binaries will be searched using the following $PATH: /usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

    • INFO: Using http proxy http://xxx.xxx.xxx.xxx:8080

    • INFO: Installing minion

    • DEBUG: install__redhatenterpriseserver_6_stable_deps not found....

    • DEBUG: install__redhatenterpriseserver_6_8_stable_deps not found....

    • DEBUG: install__redhatenterpriseserver_6_deps not found....

    • DEBUG: install__redhatenterpriseserver_6_8_deps not found....

    • DEBUG: install__redhatenterpriseserver_stable_deps not found....

    • DEBUG: install__redhatenterpriseserver_deps not found....

    • DEBUG: DEPS_INSTALL_FUNC=null

    • DEBUG: CONFIG_SALT_FUNC=null

    • DEBUG: PRESEED_MASTER_FUNC=null

    • DEBUG: install__redhatenterpriseserver_6_stable not found....

    • DEBUG: install__redhatenterpriseserver_6_8_stable not found....

    • DEBUG: install__redhatenterpriseserver_stable not found....

    • DEBUG: INSTALL_FUNC=null

    • DEBUG: install__redhatenterpriseserver_6_stable_post not found....

    • DEBUG: install__redhatenterpriseserver_6_8_stable_post not found....

    • DEBUG: install__redhatenterpriseserver_6_post not found....

    • DEBUG: install__redhatenterpriseserver_6_8_post not found....

    • DEBUG: install__redhatenterpriseserver_stable_post not found....

    • DEBUG: install__redhatenterpriseserver_post not found....

    • DEBUG: POST_INSTALL_FUNC=null

    • DEBUG: install__redhatenterpriseserver_6_stable_restart_daemons not found....

    • DEBUG: install__redhatenterpriseserver_6_8_stable_restart_daemons not found....

    • DEBUG: install__redhatenterpriseserver_6_restart_daemons not found....

    • DEBUG: install__redhatenterpriseserver_6_8_restart_daemons not found....

    • DEBUG: install__redhatenterpriseserver_stable_restart_daemons not found....

    • DEBUG: install__redhatenterpriseserver_restart_daemons not found....

    • DEBUG: STARTDAEMONS_INSTALL_FUNC=null

    • DEBUG: daemons_running__redhatenterpriseserver_6_stable not found....

    • DEBUG: daemons_running__redhatenterpriseserver_6_8_stable not found....

    • DEBUG: daemons_running__redhatenterpriseserver_6 not found....

    • DEBUG: daemons_running__redhatenterpriseserver_6_8 not found....

    • DEBUG: daemons_running__redhatenterpriseserver_stable not found....

    • DEBUG: daemons_running__redhatenterpriseserver not found....

    • INFO: Found function daemons_running

    • DEBUG: DAEMONS_RUNNING_FUNC=daemons_running

    • DEBUG: install__redhatenterpriseserver_6_stable_check_services not found....

    • DEBUG: install__redhatenterpriseserver_6_8_stable_check_services not found....

    • DEBUG: install__redhatenterpriseserver_6_check_services not found....

    • DEBUG: install__redhatenterpriseserver_6_8_check_services not found....

    • DEBUG: install__redhatenterpriseserver_stable_check_services not found....

    • DEBUG: install__redhatenterpriseserver_check_services not found....

    • DEBUG: CHECK_SERVICES_FUNC=null

    • ERROR: No dependencies installation function found. Exiting...

    • DEBUG: Removing the logging pipe /tmp/bootstrap-salt.logpipe

Versions and Systems

./deploy.sh -- Version 2016.07.07

Could you please help me resolve this issue.

Regards,
-Jarek

Bug Fixed Pending Verification
Source
picture gerronimo

Most helpful comment

I haven't tried your workaround, but I think I finally nailed it.
The answer is:
echo $LANG
pl_PL.utf8
and this is why this sed:
echo "CamelCase" | sed -r -e 's/([^A-Z-])([A-Z])/\1 \2/g'
gives me:
just CamelCase.
But if I do:
export LANG=C
then it prints out correctly
"Camel Case"
So the problem was localisation.
I think that this should be set somewhere in the beginning of the bootstrap script.
I haven't actually tested it on the script itself because of lack of time. I'll report when I do.

picture gerronimo on 16 Aug 2016
🎉1 😄1 👍1

All 27 comments

Hi @gerronimo

I think you don't have lsb-release installed on your system, but you _do have_ /etc/lsb-release file present. And I guess there are some local modifications in it, that's why Salt Bootstrap failed to select appropriate installation functions.

Could you please post output of the following commands?

lsb_release -si

cat /etc/lsb-release
vutny picture vutny on 1 Aug 2016

I am cloning from corporate images. I did't modify it myself and I am not aware of any modifications to it. However this might be the case.
These are the results:

lsb_release -si

RedHatEnterpriseServer

cat /etc/lsb-release

LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch

gerronimo picture gerronimo on 1 Aug 2016

This looks good, bootstrap script should be able to parse it correctly. I've also tried to bootstrap fresh RHEL 6.8 instance in the cloud and it worked for me.

I clearly see the problem here:

* INFO: Distribution: # RedHatEnterpriseServer 6.8

At least, there it should be no hash # sign before the distro name. I wonder where it came from...

Let's try another thing. Please post (or better create a GitHub gist) with shell debugging information after running command like this: (will give a lot of output)

sudo sh -xv ./deploy.sh -D
vutny picture vutny on 1 Aug 2016

I'm currently on a sick leave. I'll post results when I get to the office :)
Thank you for help.

gerronimo picture gerronimo on 2 Aug 2016

Sure, no problem. Take care and get better!

vutny picture vutny on 2 Aug 2016

Hello again.
I'm all good now. I played around and it turned out that I have misconfigured my profile for creating new vms. I've put 'dns_servers' in the wrong section and so new machines couldn't access our internal base repository. Now the problem is only when installing through salt-cloud,

  • INFO: Using http proxy https://XXX.XXX.XXX.XXX:8080

    • INFO: Installing minion

    • INFO: Found function config_salt

    • INFO: Found function daemons_running

    • ERROR: No dependencies installation function found. Exiting...

And when run the same script deploy.sh -H 'https://XXX.XXX.XXX.XXX:8080' on this machine manually then it works fine. I event tried installing salt-minion and salt-master on the image and then pass -b to script_args. but the effect looks the same. Is there some tutorial on how to create the cloning image, what should be installed and so on.. ?

gerronimo picture gerronimo on 8 Aug 2016

Regarding '#' sign, I never seen it since :) Don't know where it came from in the first place but it got resolved somehow.

gerronimo picture gerronimo on 8 Aug 2016

Hi,

As for me, it looks like an issue with Salt Cloud (or maybe with specific provider), if you say that running the script manually just works.

Still, need some debugging information to figure out what's the case here. Try fresh instance for clear result.

  1. Enable debug mode for the bootstrap script by editing your profile like this:

yaml my-profile: provider: my-provider image: my-image ... script_args: -D -H 'https://XXX.XXX.XXX.XXX:8080'

  1. Show debug output from the Salt Cloud, for example:

sh salt-cloud -p my-profile my-new-vm -l debug

  1. You can find full debug log from bootstrap script in /tmp/ directory called bootstrap-salt.log on your newly created VM.

This line:

* ERROR: No dependencies installation function found. Exiting...

means that script still was unable to guess the OS distribution correctly, but the reason for that is not obvious.

vutny picture vutny on 8 Aug 2016

@gerronimo If this is failing only with salt-cloud, and not when running the bootstrap script by hand, we might want to move this issue over to the salt repo so it can be fixed in salt-cloud. In the mean time, the answers to the questions @vutny posted above will be helpful in tracking this issue down so we can fix it. The output of salt-cloud --versions-report will also be needed when this is moved over.

rallytime picture rallytime on 9 Aug 2016

I've put all logs here:
XXX

salt-cloud --versions-report
/usr/lib64/python2.6/site-packages/Crypto/Util/randpool.py:40: RandomPool_DeprecationWarning: This application uses RandomPool, which is BROKEN in older releases.  See http://www.pycrypto.org/randpool-broken
  RandomPool_DeprecationWarning)
Salt Version:
            Salt: 2016.3.1

Dependency Versions:
 Apache Libcloud: 0.20.1
            cffi: Not Installed
        cherrypy: Not Installed
        dateutil: 1.4.1
           gitdb: Not Installed
       gitpython: Not Installed
           ioflo: Not Installed
          Jinja2: 2.2.1
         libgit2: Not Installed
         libnacl: Not Installed
        M2Crypto: 0.20.2
            Mako: 0.3.4
    msgpack-pure: Not Installed
  msgpack-python: 0.4.6
    mysql-python: Not Installed
       pycparser: Not Installed
        pycrypto: 2.6.1
          pygit2: Not Installed
          Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
    python-gnupg: Not Installed
          PyYAML: 3.10
           PyZMQ: 14.3.1
            RAET: Not Installed
           smmap: Not Installed
         timelib: Not Installed
         Tornado: 4.2.1
             ZMQ: 3.2.5

System Versions:
            dist: redhat 6.8 Santiago
         machine: x86_64
         release: 2.6.32-573.12.1.el6.centos.plus.x86_64
          system: Linux
         version: Red Hat Enterprise Linux Server 6.8 Santiago
gerronimo picture gerronimo on 9 Aug 2016

Just as a sidenote. I've also cloned centos 6.8 image from that same instance of salt-cloud and it worked also perfectly fine, meaning the vm has been correctly cloned, salt-minion installed and I could ping it withsalt 'testing' 'test.ping'

gerronimo picture gerronimo on 9 Aug 2016

I see, everything looks good except for the bootstrap script is unable to parse distribution string RedHatEnterpriseServer for some reason under salt-cloud supervision.

With another round of our debugging session, let's figure that out.
I've prepared a modified version of the script that should shed the light on the issue.

Please install it on your machine where you run salt-cloud with such command:

curl 'https://raw.githubusercontent.com/vutny/salt-bootstrap/rhel-detection-debug/bootstrap-salt.sh' > /etc/salt/cloud.deploy.d/bootstrap-salt.sh

And repeat your RHEL 6.8 VM provisioning from scratch one more time.
I hope we'll find out what's happening there :)

vutny picture vutny on 9 Aug 2016

no problem Let's see if it helps.

gerronimo picture gerronimo on 9 Aug 2016

@gerronimo I understood. __camelcase_split function returns wrong result. The fun thing is that I was unable to reproduce this behavior... In your case sed expression just doesn't work for unknown reason when running the bootstrap script with salt-cloud.

Let's try to guard our back and slightly change the function call and body.
Please repeat all the steps including the script download and post a result.

Also, try run the script non-interactively, like this:

ssh -t -t [email protected] 'sh /tmp/.saltcloud-*/deploy.sh -D'

We are blind here, seems like there are some specific adjustments in your VM image.

vutny picture vutny on 10 Aug 2016

https://gist.github.com/jbober/6342a3700db142458b7a2a8efcb9c35f

Again when I run deploy.sh directly on the machine ( bootstrap-salt_manual.log) it works as expected but fails over ssh :)
I'm confused.

jbober picture jbober on 10 Aug 2016

Hello,
I've tinkered a little with this line:
echo "$*" | sed -r -e 's/([^A-Z-])([A-Z])/\1 \2/g'
Theres something strange about this line.I believe It captures whole word and not splitting it. At least on my CentOS 6.8 and redhat. Whenever I made this sed fail then salt-cloud was able to provision on my redhats. I just deleted -r. For centos I believe this is not used.
For now that will get me going. Thank you for your support. You've been very helpful.

gerronimo picture gerronimo on 11 Aug 2016

No problem! Good that it worked for you.

But... This is very strange. sed would not be able to process such extended regex without -r key.
It should be rewritten like this:

sed -e 's/\([^A-Z-]\)\([A-Z]\)/\1 \2/g'

Please could you try this and share a result with us?
My guess is that you have some specific SSH server configuration on your VM image. If it's possible, could you also post /etc/ssh/sshd_config file content from your RHEL 6.8 instance?

It would be really cool to find why such things happen.

vutny picture vutny on 12 Aug 2016

Or maybe it's more related to the shell (non-)interactive or (non-)login mode. Looking at /root/.bash_profile and /root/.bashrc files could possibly shed some light on the issue.

vutny picture vutny on 12 Aug 2016

Hello again.
I intentionally made this sed fail. Only then it when further with the installation.
Anyway, . When I took dockerized centos 6 and 7 and run this:
echo "CamelCase" | sed -r -e 's/([^A-Z-])([A-Z])/\1 \2/g'
Then indeed it gives
Camel Case as expected. I may have broken bash on my developers machines.
I am using devtoolset-4 and software collections. That may interfere somehow.
I really need to check that.

gerronimo picture gerronimo on 12 Aug 2016

Oh, now I got it!
Because the pipe command in __camelcase_split function failed when you deleted -r option for sed, the $DISTRO_NAME variable become empty and distribution was correctly detected later during release files parsing. Nice! :)

So, I think I have an idea how to workaround this...

vutny picture vutny on 15 Aug 2016

@gerronimo Please try the script from my PR above (it's a new branch):
https://github.com/vutny/salt-bootstrap/blob/rhel-detection-workaround/bootstrap-salt.sh

I hope it should solve your issue.
But still, it would be nice to know the root cause of this if you will find anything...

vutny picture vutny on 15 Aug 2016

I haven't tried your workaround, but I think I finally nailed it.
The answer is:
echo $LANG
pl_PL.utf8
and this is why this sed:
echo "CamelCase" | sed -r -e 's/([^A-Z-])([A-Z])/\1 \2/g'
gives me:
just CamelCase.
But if I do:
export LANG=C
then it prints out correctly
"Camel Case"
So the problem was localisation.
I think that this should be set somewhere in the beginning of the bootstrap script.
I haven't actually tested it on the script itself because of lack of time. I'll report when I do.

gerronimo picture gerronimo on 16 Aug 2016
🎉1 😄1 👍1

Nice catch! I'm also using UTF-8 locale, but didn't observed such effect, so I've excluded the possibility of localization issue. Well done!

In my PR I've replaced character ranges with POSIX classes, so now sed expression should be fully locale aware. Please test when you'll have some time. Thanks!

vutny picture vutny on 17 Aug 2016

Hello again I've tested your workaround and also setting LANG variable.
The only ghing I changed was adding export LANG=C in a second line.

Both methods work fine.
Here are the logs:
https://gist.github.com/jbober/0f35f129b5aaaccad4add01dfc127936

Thank you very much for your help I really appreciate it.

gerronimo picture gerronimo on 17 Aug 2016
👍2

Sounds great! I'm glad that you've found a root cause and the solution.
When my PR will get merged, this issue should be resolved permanently.

P.S. Generally I'm against overriding locale settings in the shell scripts... Do it only if you have no other choice. There is always a possibility to receive some localized input from file names or contents, so this is a double-edge sword. It's always better to write localization-friendly code :)

vutny picture vutny on 17 Aug 2016

Sure, I'll keep that in mind. Thanks for helping me out.

gerronimo picture gerronimo on 18 Aug 2016

Ping @rallytime
Since this issue is resolved now, can we get PR #932 merged please? Thanks in advance!

vutny picture vutny on 18 Aug 2016
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ggmartins picture ggmartins  路  8Comments
ninja- picture ninja-  路  15Comments
jworl picture jworl  路  4Comments
m87carlson picture m87carlson  路  9Comments
chrisnew picture chrisnew  路  8Comments