Kombu: Upgrading to kombu 4.6, celery workers no longer respond to pings

[djdduty]

We didn't have our Kombu version pinged and it appears Celery didn't have it pinned enough to prevent it auto updating.

Using python 3.7, and Celery 4.3.0.

With kombu 4.5.0 workers correctly respond to pings via "celery -A app inspect ping" while a worker is running.

With kombu 4.6.0 the ping command yields "Error: No nodes replied within time constraint." while a worker is running.

[auvipy]

did you find the root of this regression?

[olemoign]

I'm getting the same problem with inspect stats.

[dsalisbury]

See #1051

[youngquan]

I'm getting the same problem with inspect status.

[auvipy]

try to track down the root cause of regression

[browniebroke]

In case it helps, here is a tiny project with the issue: https://github.com/browniebroke/celery-ping-issue

Can see the problem in Travis: https://travis-ci.com/browniebroke/celery-ping-issue/builds/114915716

[browniebroke]

I've installed Kombu from source with a specific SHA to try to track down the commit that introduced the bug, and it looks like it's coming from this one https://github.com/celery/kombu/commit/73d2219887dfc5469c0c1ea382baf4749049aef2

The previous code change (https://github.com/celery/kombu/commit/c1b1010daa6c8cee35230ce2f73b3f2c3bb7b10b) works fine, build the issue breaks from 73d2219887dfc5469c0c1ea382baf4749049aef2

All builds are visible here: https://travis-ci.com/browniebroke/celery-ping-issue/builds

[auvipy]

reverted the root cause of regression https://github.com/celery/kombu/pull/1058

[auvipy]

can you try the master again after your confirmation I will push a new release

[browniebroke]

https://travis-ci.com/browniebroke/celery-ping-issue/builds/114935966

Collecting fastuuid (from kombu==4.6.1->-r requirements.txt (line 2))
  ERROR: Could not find a version that satisfies the requirement fastuuid (from kombu==4.6.1->-r requirements.txt (line 2)) (from versions: none)
ERROR: No matching distribution found for fastuuid (from kombu==4.6.1->-r requirements.txt (line 2))
ERROR: Service 'celery' failed to build: The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1

[browniebroke]

Looks like this was a temporary failure, works when installing from latest commit on master:
https://travis-ci.com/browniebroke/celery-ping-issue/builds/115081177

[thedrow]

I've installed Kombu from source with a specific SHA to try to track down the commit that introduced the bug, and it looks like it's coming from this one 73d2219

The previous code change (c1b1010) works fine, build the issue breaks from 73d2219

All builds are visible here: https://travis-ci.com/browniebroke/celery-ping-issue/builds

How is it the source of the issue?
It should be an optimization.
Simply reverting #1058 is not a solution.
We should investigate to find what's wrong with that approach and fix that.

[browniebroke]

I agree reverting is not the proper solution, but until one is found, it is usually a safe and simple in the short term.

The alternative is to leave the bug out in the latest version on PyPI while you have the proper fix, but that's not helping maintainers or users: you would receive duplicate bug reports, spend time triaging them and users might spend time trying to upgrade only to find out that their Celery monitoring is broken.

I don't know _how_ this is the source of the issue, I was simply trying to be helpful by trying to narrow down when the ping behaviour started to break.

I appreciate your time and efforts on this project. Sorry if I made your work more difficult here, that was not my intend.