Kombu: Kombu 4.2.2: py.test seems to fail/hang

[fxfitz]

Hi guys,

My CI detected that the new release of Kombu is not working in my project anymore, specifically when trying to run py.test with my application.

So far I don't know what the root cause of the problem is with Kombu, but pinning to 4.2.1 worked for us.

This is what I was able to pull out of pytest with PYTEST_DEBUG enabled, which pointed me down the road of "did they release a new version of Kombu":

early skip of rewriting module: hiredis.hiredis [assertion]
    early skip of rewriting module: hiredis.version [assertion]
    early skip of rewriting module: redis.lock [assertion]
    early skip of rewriting module: redis.sentinel [assertion]
    early skip of rewriting module: django_celery_results.backends [assertion]
    early skip of rewriting module: django_celery_results.backends.cache [assertion]
    matched marked file 'celery.backends' (from 'celery') [assertion]
    find_module called for: celery.backends [assertion]
    rewriting '/usr/local/lib/python3.6/site-packages/celery/backends.py' [assertion]
    matched marked file 'celery.backends.base' (from 'celery') [assertion]
    find_module called for: celery.backends.base [assertion]
    rewriting '/usr/local/lib/python3.6/site-packages/celery/backends/base.py' [assertion]
    early skip of rewriting module: django_celery_results.backends.database [assertion]
    matched marked file 'celery.app.events' (from 'celery') [assertion]
    find_module called for: celery.app.events [assertion]
    rewriting '/usr/local/lib/python3.6/site-packages/celery/app/events.py' [assertion]
    matched marked file 'celery.events' (from 'celery') [assertion]
    find_module called for: celery.events [assertion]
    rewriting '/usr/local/lib/python3.6/site-packages/celery/events.py' [assertion]
    matched marked file 'celery.events.dispatcher' (from 'celery') [assertion]
    find_module called for: celery.events.dispatcher [assertion]
    rewriting '/usr/local/lib/python3.6/site-packages/celery/events/dispatcher.py' [assertion]
    matched marked file 'celery.events.event' (from 'celery') [assertion]
    find_module called for: celery.events.event [assertion]
    rewriting '/usr/local/lib/python3.6/site-packages/celery/events/event.py' [assertion]
    matched marked file 'celery.events.receiver' (from 'celery') [assertion]
    find_module called for: celery.events.receiver [assertion]
    rewriting '/usr/local/lib/python3.6/site-packages/celery/events/receiver.py' [assertion]
    early skip of rewriting module: kombu.mixins [assertion]
    early skip of rewriting module: kombu.utils.limits [assertion]

And it just hangs from there.

[russoz]

I am also seeing problems, from a Django+Celery application, in the very first call to a celery task, after apparently calling the task successfully (it is a ignore_result=True task), it looks like the cleanup code messes something up

  File "/Users/aznamensky/xxxxx/virtualenv/lib/python3.6/site-packages/kombu/utils/eventio.py", line 209, in unregister
self._quick_unregister(fd)
ValueError: file descriptor cannot be a negative integer (-1)

Also worth mentioning we're using celery[redis], and the error happens in the context of a local unit test, for which we run the redislite package to fake a redis server.

[thedrow]

We haven't done anything in 4.2.2 except adding code that allows both redis 2.x and redis 3.x python clients.
Seems like the two problems you are describing are separate.
@russoz please open another issue and provide more details.

[007]

kombu fails to connect to a redis sentinel when redis==2.10.6 and kombu==4.2.2, but works fine with kombu==4.2.1 and the same redis version and config.

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/functional.py", line 335, in retry_over_time
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 275, in connect
    return self.connection
  File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 824, in connection
    self._connection = self._establish_connection()
  File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 779, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 941, in establish_connection
    self._avail_channels.append(self.create_channel(self))
  File "/usr/local/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 923, in create_channel
    channel = self.Channel(connection)
  File "/usr/local/lib/python3.6/site-packages/kombu/transport/redis.py", line 512, in __init__
    self.client.ping()
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 777, in ping
    return self.execute_command('PING')
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 673, in execute_command
    connection.send_command(*args)
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 610, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 585, in send_packed_command
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/redis/sentinel.py", line 45, in connect
    self.connect_to(self.connection_pool.get_master_address())
  File "/usr/local/lib/python3.6/site-packages/redis/sentinel.py", line 101, in get_master_address
    self.service_name)
  File "/usr/local/lib/python3.6/site-packages/redis/sentinel.py", line 223, in discover_master
    raise MasterNotFoundError("No master found for %r" % (service_name,))

[thedrow]

I'm very much confused here.
@fxfitz Which error are you seeing? @007's or @russoz's?

[fxfitz]

I believe I am seeing the error that @007 is describing; I first noticed this when we were adding Sentinel support and 4.2.2 came out.

[groves]

It appears that 4.2.2 is cut from master instead of the 4.2 branch: https://github.com/celery/kombu/issues/966

Perhaps that explains why there's more in it than allowing new redis clients?

[thedrow]

Yes, that's my bad. I re-released 4.2.2.
Please upgrade.

[russoz]

For the record I can confirm my problem disappeared with the re-release of 4.2.2