Salt: Minion did not return. [Not connected] when execute: salt 'minion1' test.ping

[AIROBOTAI]

Description of Issue/Question

The question for the similar issue may have been asked for many times. However after searching, I found no answer solve my problem. So I post it here to seek for your help.

My system is Ubuntu 14.04. After setup master and minion (named "minion1"), I test salt on master using the command

salt 'minion1' test.ping

I got the following error:

minion1:
    Minion did not return. [Not connected]

Following others' advices, I have tested the following commands:

• On minion, I firstly remove the previous master key by:

rm /etc/salt/pki/minion/minion_master.pub

and restart minion, but the previous error remained. So I eliminated the case of unmatched master key.

• On minion, I execute

salt-call test.ping

to get:

local:
    True

so I was sure that the communication between master and minion is good.

• On minion, I firstly stop salt-minion using command:

service salt-minion stop

and run it in debug mode using command:

salt-minion -l debug

to get the following return:

[DEBUG   ] Reading configuration from /etc/salt/minion
[INFO    ] Using cached minion ID from /etc/salt/minion_id: minion1
[DEBUG   ] loading log_handlers in ['/var/cache/salt/minion/extmods/log_handlers', '/usr/lib/python2.7/dist-packages/salt/log/handlers']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/log_handlers, it is not a directory
[DEBUG   ] Configuration file path: /etc/salt/minion
[INFO    ] Setting up the Salt Minion "minion1"
[DEBUG   ] Created pidfile: /var/run/salt-minion.pid
[DEBUG   ] Chowned pidfile: /var/run/salt-minion.pid to user: root
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] loading grain in ['/var/cache/salt/minion/extmods/grains', '/usr/lib/python2.7/dist-packages/salt/grains']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/grains, it is not a directory
[DEBUG   ] Attempting to authenticate with the Salt Master at 192.168.0.101
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Authentication with master successful!
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] loading grain in ['/var/cache/salt/minion/extmods/grains', '/usr/lib/python2.7/dist-packages/salt/grains']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/grains, it is not a directory
[DEBUG   ] loading module in ['/var/cache/salt/minion/extmods/modules', '/usr/lib/python2.7/dist-packages/salt/modules']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/modules, it is not a directory
[DEBUG   ] Loaded localemod as virtual locale
[DEBUG   ] Loaded groupadd as virtual group
[DEBUG   ] Loaded linux_sysctl as virtual sysctl
[DEBUG   ] Loaded parted as virtual partition
[DEBUG   ] Loaded linux_acl as virtual acl
[DEBUG   ] Loaded apt as virtual pkg
[DEBUG   ] Loaded sysmod as virtual sys
[DEBUG   ] Loaded djangomod as virtual django
[DEBUG   ] Loaded upstart as virtual service
[DEBUG   ] Loaded useradd as virtual user
[DEBUG   ] Loaded dpkg as virtual lowpkg
[DEBUG   ] Loaded debconfmod as virtual debconf
[DEBUG   ] Loaded virtualenv_mod as virtual virtualenv
[DEBUG   ] Loaded cmdmod as virtual cmd
[DEBUG   ] loading returner in ['/var/cache/salt/minion/extmods/returners', '/usr/lib/python2.7/dist-packages/salt/returners']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/returners, it is not a directory
[DEBUG   ] Loaded syslog_return as virtual syslog
[DEBUG   ] Loaded couchdb_return as virtual couchdb
[DEBUG   ] Loaded carbon_return as virtual carbon
[DEBUG   ] Loaded sqlite3_return as virtual sqlite3
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnpackValueError: 
Traceback (most recent call last):
  File "/usr/bin/salt-minion", line 14, in <module>
    salt_minion()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 30, in salt_minion
    minion.start()
  File "/usr/lib/python2.7/dist-packages/salt/__init__.py", line 219, in start
    self.prepare()
  File "/usr/lib/python2.7/dist-packages/salt/__init__.py", line 207, in prepare
    self.minion = salt.minion.Minion(self.config)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 518, in __init__
self.returners)
  File "/usr/lib/python2.7/dist-packages/salt/utils/schedule.py", line 71, in __init__
    clean_proc_dir(opts)
  File "/usr/lib/python2.7/dist-packages/salt/utils/schedule.py", line 267, in clean_proc_dir
    job = salt.payload.Serial(opts).load(fp_)
  File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 108, in load
    return self.loads(data)
  File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 95, in loads
    return msgpack.loads(msg, use_list=True)
  File "_unpacker.pyx", line 119, in msgpack._unpacker.unpackb (msgpack/_unpacker.cpp:119)
UnpackValueError
Traceback (most recent call last):
  File "/usr/bin/salt-minion", line 14, in <module>
    salt_minion()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 30, in salt_minion
    minion.start()
  File "/usr/lib/python2.7/dist-packages/salt/__init__.py", line 219, in start
    self.prepare()
  File "/usr/lib/python2.7/dist-packages/salt/__init__.py", line 207, in prepare
    self.minion = salt.minion.Minion(self.config)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 518, in __init__
self.returners)
  File "/usr/lib/python2.7/dist-packages/salt/utils/schedule.py", line 71, in __init__
    clean_proc_dir(opts)
  File "/usr/lib/python2.7/dist-packages/salt/utils/schedule.py", line 267, in clean_proc_dir
    job = salt.payload.Serial(opts).load(fp_)
  File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 108, in load
    return self.loads(data)
  File "/usr/lib/python2.7/dist-packages/salt/payload.py", line 95, in loads
    return msgpack.loads(msg, use_list=True)
  File "_unpacker.pyx", line 119, in msgpack._unpacker.unpackb (msgpack/_unpacker.cpp:119)
msgpack.exceptions.UnpackValueError

This problem has being hanging there for some time, could anyone help me on that? Thanks in advance!

Edit.

I found a strange thing after the before post. When I run the command

service salt-minion status

Oddly enough it returned:

salt-minion stop/waiting

if I further check using: ps aux| grep salt-minion, I found no process of salt-minion is running. I wonder that's why salt minion1 test.ping cannot return correct value. However, no matter what I did (service salt-minion start), I could never make the return of service salt-minion status shows salt-minion is start/running. What should I do now?

Versions Report

Master:

Salt: 2015.5.3
Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
Jinja2: 2.7.2
M2Crypto: 0.21.1
msgpack-python: 0.3.0
msgpack-pure: Not Installed
pycrypto: 2.6.1
libnacl: Not Installed
PyYAML: 3.10
ioflo: Not Installed
PyZMQ: 14.0.1
RAET: Not Installed
ZMQ: 4.0.4
Mako: 0.9.1
Tornado: 3.1.1
Debian source package: 2015.5.3+ds-1trusty1

Minion:

Salt: 0.17.5
Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
Jinja2: 2.7.2
M2Crypto: 0.21.1
msgpack-python: 0.3.0
msgpack-pure: Not Installed
pycrypto: 2.6.1
PyYAML: 3.10
PyZMQ: 14.0.1
ZMQ: 4.0.4

[Ch3LL]

@AIROBOTAI your minion version is pretty old. If you upgrade your minion to the same version as your master do you have any luck?

[AIROBOTAI]

@Ch3LL Yes! That's where the problem in. Since I use the default source of ubuntu 14.04 without updating the repository, thus the version is quite out-of-date. Thanks for your help!

For other people having the same problem as me, please do the following first:
add-apt-repository ppa:saltstack/salt
apt-get update
apt-get install salt-minion
Then it works fine!