Flower: Workers always show as offline

[gtaylor]

http://i.imgur.com/6HI03k8.png

Cheetah==2.4.4
GnuPGInterface==0.3.2
Landscape-Client==13.07.3
M2Crypto==0.21.1
PAM==0.4.2
PyYAML==3.10
Twisted-Core==11.1.0
Twisted-Names==11.1.0
Twisted-Web==11.1.0
amqp==1.4.5
anyjson==0.3.3
apt-xapian-index==0.44
argparse==1.2.1
backports.ssl-match-hostname==3.4.0.2
billiard==3.3.0.18
boto==2.2.2
celery==3.1.12
chardet==2.0.1
cloud-init==0.6.3
command-not-found==0.2.44
configobj==4.7.2
## FIXME: could not find svn URL in dependency_links for this package:
distribute==0.6.24dev-r0
euca2ools==2.0.0
flower==0.8.0
httplib2==0.7.2
iotop==0.4.4
keyring==0.9.2
kombu==3.0.20
language-selector==0.1
launchpadlib==1.9.12
lazr.restfulclient==0.12.0
lazr.uri==1.0.3
oauth==1.0.1
paramiko==1.7.7.1
pyOpenSSL==0.12
pycrypto==2.4.1
pycurl==7.19.0
pyserial==2.5
python-apt==0.8.3ubuntu7.2
python-debian==0.1.21ubuntu1
pytz==2014.4
redis==2.9.1
simplejson==2.3.2
tornado==3.2.2
ufw==0.31.1-1
unattended-upgrades==0.1
wadllib==1.3.0
wsgiref==0.1.2
zope.interface==3.6.1

[mher]

This can happen if your workers are very busy.

[mher]

The next version will include more intelligent algorithm,

[gtaylor]

We average maybe 10 tasks a minute, almost all of which are very low quick to return.

I'm running current git master now, does that contain the new algorithm, or is this just a target for future development>

[mher]

The new algorithm is not merged into the master.

[ghost]

Hi gtaylor,
check kombu version.

In these days I had a similar problem solved upgrading kombu to version 3.0.21 (from 3.0.20)

pip install kombu==3.0.21

I'm still investigating at the moment but for now flower seems to work.
Let me know if it helps.

[gtaylor]

@dcellai Good call, this seems to have fixed us!

@mher I'm not sure of the best way to resolve this going forward. Perhaps >=3.0.21 should be a requirement in the latest git master?

[jclark360]

Fixed same issue for me too.

[maklaut]

As I understand at present time Flower can't adequately monitor system with busy workers because monitoring messages and task messages goes to the same queue (or concurrency limit).
@mher can you explain us the new algorithm please?

[mher]

@maklaut monitoring and control messages go to different queues. Currently flower periodically sends inspect commands to workers. If workers are busy the info can be out of date.

The upcoming version doesn't use inspect commands at all. The new Dashboard is implemented using celery events. You can checkout it in dashboard branch. Feel free to give a feedback :)

[mher]

The new dashboard has been merged into the master. Please try it.

[gtaylor]

It looks like the latest master has us back to showing offline for all workers:

http://i.imgur.com/JKkgTwF.png

Babel==1.3
Cheetah==2.4.4
Landscape-Client==14.01
PAM==0.4.2
PyYAML==3.10
Twisted-Core==13.2.0
Twisted-Names==13.2.0
Twisted-Web==13.2.0
amqp==1.4.6
anyjson==0.3.3
apt-xapian-index==0.45
argparse==1.2.1
backports.ssl-match-hostname==3.4.0.2
billiard==3.3.0.18
celery==3.1.13
certifi==14.05.14
chardet==2.0.1
cloud-init==0.7.5
colorama==0.2.5
configobj==4.7.2
-e git+https://github.com/mher/flower.git@3ba05a31fe4c69c57dd9abb1e368aceefe36635f#egg=flower-master
futures==2.1.6
html5lib==0.999
iotop==0.6
jsonpatch==1.3
jsonpointer==1.0
kombu==3.0.22
oauth==1.0.1
prettytable==0.7.2
pyOpenSSL==0.13
pycurl==7.19.3
pyserial==2.6
python-apt==0.9.3.5
python-debian==0.1.21-nmu2ubuntu2
pytz==2014.7
redis==2.10.3
requests==2.2.1
six==1.5.2
ssh-import-id==3.21
tornado==4.0.1
urllib3==1.7.1
wsgiref==0.1.2
zope.interface==4.0.5

[pinple]

Hello, I use flower 0.9.2 now, but there is also this bug.Do you merge the new algorithm into master in version 0.9.2 ?

thanks.

[alejandrohdo]

I have just running this version of: kombu == 4.2.2.post1, flower == 0.9.2 and celery == 4.2.1, I have the same problem that flower is out of line in some workers in an unpredictable way, some solution about?. Thank you