Kombu: Kombu 4.3.0: Unrecoverable error: ValueError('file descriptor cannot be a negative integer (-1)

[petersanchez]

4.3.0 seems to have broken our Celery install. Downgrading to 4.2.2-post1 resolves it.

Redis backend

redis==2.10.6
Django==2.1.5
celery==4.2.1

OS: FreeBSD 12 p3.

Strange thing is, on my Ubuntu 18.04 laptop, the same requirements, and it works as expected.

I also completely flushed the redis db to ensure it wasn't some stuck task or odd issue causing it. Same thing. Only downgrading resolved it.

[2019-02-11 15:28:19,386: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2019-02-11 15:28:19,389: DEBUG/MainProcess] | Worker: Building graph...
[2019-02-11 15:28:19,390: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Pool, Autoscaler, StateDB, Beat, Consumer}
[2019-02-11 15:28:19,411: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2019-02-11 15:28:19,411: DEBUG/MainProcess] | Consumer: Building graph...
[2019-02-11 15:28:19,429: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Tasks, Control, Heart, Agent, Gossip, event loop}
[2019-02-11 15:28:19,459: DEBUG/MainProcess] | Worker: Starting Hub
[2019-02-11 15:28:19,459: DEBUG/MainProcess] ^-- substep ok
[2019-02-11 15:28:19,459: DEBUG/MainProcess] | Worker: Starting Pool
[2019-02-11 15:28:19,619: DEBUG/MainProcess] ^-- substep ok
[2019-02-11 15:28:19,619: DEBUG/MainProcess] | Worker: Starting Beat
[2019-02-11 15:28:19,623: DEBUG/MainProcess] ^-- substep ok
[2019-02-11 15:28:19,624: DEBUG/MainProcess] | Worker: Starting Consumer
[2019-02-11 15:28:19,625: DEBUG/MainProcess] | Consumer: Starting Connection
[2019-02-11 15:28:19,642: INFO/MainProcess] Connected to redis://localhost:6379/0
[2019-02-11 15:28:19,643: DEBUG/MainProcess] ^-- substep ok
[2019-02-11 15:28:19,643: DEBUG/MainProcess] | Consumer: Starting Events
[2019-02-11 15:28:19,659: DEBUG/MainProcess] ^-- substep ok
[2019-02-11 15:28:19,659: DEBUG/MainProcess] | Consumer: Starting Mingle
[2019-02-11 15:28:19,659: INFO/MainProcess] mingle: searching for neighbors
[2019-02-11 15:28:19,677: CRITICAL/MainProcess] Unrecoverable error: ValueError('file descriptor cannot be a negative integer (-1)')
Traceback (most recent call last):
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 317, in start
    blueprint.start(self)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/mingle.py", line 40, in start
    self.sync(c)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/mingle.py", line 44, in sync
    replies = self.send_hello(c)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/mingle.py", line 57, in send_hello
    replies = inspect.hello(c.hostname, our_revoked._data) or {}
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/app/control.py", line 143, in hello
    return self._request('hello', from_node=from_node, revoked=revoked)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/app/control.py", line 95, in _request
    timeout=self.timeout, reply=True,
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/app/control.py", line 454, in broadcast
    limit, callback, channel=channel,
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pidbox.py", line 315, in _broadcast
    serializer=serializer)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pidbox.py", line 290, in _publish
    serializer=serializer, retry=True,
  File "/usr/local/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pidbox.py", line 248, in producer_or_acquire
    yield producer
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/messaging.py", line 239, in __exit__
    self.release()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/resource.py", line 105, in release
    self.release(R)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pools.py", line 74, in release
    resource.__connection__.release()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/resource.py", line 105, in release
    self.release(R)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/resource.py", line 134, in release
    self.close_resource(resource)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 950, in close_resource
    resource._close()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 338, in _close
    self._do_close_self()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 328, in _do_close_self
    self.maybe_close_channel(self._default_channel)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 320, in maybe_close_channel
    channel.close()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 852, in close
    self._disconnect_pools()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 538, in _disconnect_pools
    async_pool.disconnect()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/redis/connection.py", line 998, in disconnect
    connection.disconnect()
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 941, in disconnect
    channel._on_connection_disconnect(self)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 546, in _on_connection_disconnect
    self.connection.cycle._on_connection_disconnect(connection)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 282, in _on_connection_disconnect
    self.poller.unregister(connection._sock)
  File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/utils/eventio.py", line 209, in unregister
    self._quick_unregister(fd)
ValueError: file descriptor cannot be a negative integer (-1)
[2019-02-11 15:28:19,690: DEBUG/MainProcess] | Worker: Closing Hub...
[2019-02-11 15:28:19,691: DEBUG/MainProcess] | Worker: Closing Pool...
[2019-02-11 15:28:19,691: DEBUG/MainProcess] | Worker: Closing Beat...
[2019-02-11 15:28:19,691: DEBUG/MainProcess] | Worker: Closing Consumer...
[2019-02-11 15:28:19,691: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2019-02-11 15:28:19,691: DEBUG/MainProcess] | Worker: Stopping Beat...
[2019-02-11 15:28:19,692: INFO/MainProcess] beat: Shutting down...
[2019-02-11 15:28:19,692: DEBUG/MainProcess] | Worker: Stopping Pool...
[2019-02-11 15:28:20,726: DEBUG/MainProcess] | Worker: Stopping Hub...
[2019-02-11 15:28:20,726: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2019-02-11 15:28:20,727: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2019-02-11 15:28:20,727: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2019-02-11 15:28:20,727: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2019-02-11 15:28:20,727: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2019-02-11 15:28:20,728: WARNING/MainProcess] Traceback (most recent call last):
[2019-02-11 15:28:20,728: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
[2019-02-11 15:28:20,728: WARNING/MainProcess] self.blueprint.start(self)
[2019-02-11 15:28:20,729: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
[2019-02-11 15:28:20,729: WARNING/MainProcess] step.start(parent)
[2019-02-11 15:28:20,729: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
[2019-02-11 15:28:20,730: WARNING/MainProcess] return self.obj.start()
[2019-02-11 15:28:20,730: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 317, in start
[2019-02-11 15:28:20,730: WARNING/MainProcess] blueprint.start(self)
[2019-02-11 15:28:20,730: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
[2019-02-11 15:28:20,731: WARNING/MainProcess] step.start(parent)
[2019-02-11 15:28:20,731: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/mingle.py", line 40, in start
[2019-02-11 15:28:20,731: WARNING/MainProcess] self.sync(c)
[2019-02-11 15:28:20,731: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/mingle.py", line 44, in sync
[2019-02-11 15:28:20,731: WARNING/MainProcess] replies = self.send_hello(c)
[2019-02-11 15:28:20,732: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/mingle.py", line 57, in send_hello
[2019-02-11 15:28:20,732: WARNING/MainProcess] replies = inspect.hello(c.hostname, our_revoked._data) or {}
[2019-02-11 15:28:20,732: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/app/control.py", line 143, in hello
[2019-02-11 15:28:20,732: WARNING/MainProcess] return self._request('hello', from_node=from_node, revoked=revoked)
[2019-02-11 15:28:20,733: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/app/control.py", line 95, in _request
[2019-02-11 15:28:20,733: WARNING/MainProcess] timeout=self.timeout, reply=True,
[2019-02-11 15:28:20,733: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/app/control.py", line 454, in broadcast
[2019-02-11 15:28:20,733: WARNING/MainProcess] limit, callback, channel=channel,
[2019-02-11 15:28:20,734: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pidbox.py", line 315, in _broadcast
[2019-02-11 15:28:20,734: WARNING/MainProcess] serializer=serializer)
[2019-02-11 15:28:20,734: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pidbox.py", line 290, in _publish
[2019-02-11 15:28:20,734: WARNING/MainProcess] serializer=serializer, retry=True,
[2019-02-11 15:28:20,735: WARNING/MainProcess] File "/usr/local/lib/python3.7/contextlib.py", line 119, in __exit__
[2019-02-11 15:28:20,735: WARNING/MainProcess] next(self.gen)
[2019-02-11 15:28:20,735: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pidbox.py", line 248, in producer_or_acquire
[2019-02-11 15:28:20,735: WARNING/MainProcess] yield producer
[2019-02-11 15:28:20,736: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/messaging.py", line 239, in __exit__
[2019-02-11 15:28:20,736: WARNING/MainProcess] self.release()
[2019-02-11 15:28:20,736: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/resource.py", line 105, in release
[2019-02-11 15:28:20,736: WARNING/MainProcess] self.release(R)
[2019-02-11 15:28:20,736: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/pools.py", line 74, in release
[2019-02-11 15:28:20,737: WARNING/MainProcess] resource.__connection__.release()
[2019-02-11 15:28:20,737: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/resource.py", line 105, in release
[2019-02-11 15:28:20,737: WARNING/MainProcess] self.release(R)
[2019-02-11 15:28:20,737: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/resource.py", line 134, in release
[2019-02-11 15:28:20,738: WARNING/MainProcess] self.close_resource(resource)
[2019-02-11 15:28:20,738: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 950, in close_resource
[2019-02-11 15:28:20,738: WARNING/MainProcess] resource._close()
[2019-02-11 15:28:20,738: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 338, in _close
[2019-02-11 15:28:20,739: WARNING/MainProcess] self._do_close_self()
[2019-02-11 15:28:20,739: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 328, in _do_close_self
[2019-02-11 15:28:20,739: WARNING/MainProcess] self.maybe_close_channel(self._default_channel)
[2019-02-11 15:28:20,739: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 320, in maybe_close_channel
[2019-02-11 15:28:20,739: WARNING/MainProcess] channel.close()
[2019-02-11 15:28:20,739: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 852, in close
[2019-02-11 15:28:20,740: WARNING/MainProcess] self._disconnect_pools()
[2019-02-11 15:28:20,740: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 538, in _disconnect_pools
[2019-02-11 15:28:20,740: WARNING/MainProcess] async_pool.disconnect()
[2019-02-11 15:28:20,740: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/redis/connection.py", line 998, in disconnect
[2019-02-11 15:28:20,741: WARNING/MainProcess] connection.disconnect()
[2019-02-11 15:28:20,741: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 941, in disconnect
[2019-02-11 15:28:20,741: WARNING/MainProcess] channel._on_connection_disconnect(self)
[2019-02-11 15:28:20,741: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 546, in _on_connection_disconnect
[2019-02-11 15:28:20,742: WARNING/MainProcess] self.connection.cycle._on_connection_disconnect(connection)
[2019-02-11 15:28:20,742: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 282, in _on_connection_disconnect
[2019-02-11 15:28:20,742: WARNING/MainProcess] self.poller.unregister(connection._sock)
[2019-02-11 15:28:20,742: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/utils/eventio.py", line 209, in unregister
[2019-02-11 15:28:20,743: WARNING/MainProcess] self._quick_unregister(fd)
[2019-02-11 15:28:20,743: WARNING/MainProcess] ValueError
[2019-02-11 15:28:20,743: WARNING/MainProcess] :
[2019-02-11 15:28:20,743: WARNING/MainProcess] file descriptor cannot be a negative integer (-1)
[2019-02-11 15:28:20,743: WARNING/MainProcess] During handling of the above exception, another exception occurred:
[2019-02-11 15:28:20,743: WARNING/MainProcess] Traceback (most recent call last):
[2019-02-11 15:28:20,743: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/bin/celery", line 11, in <module>
[2019-02-11 15:28:20,744: WARNING/MainProcess] sys.exit(main())
[2019-02-11 15:28:20,744: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/__main__.py", line 16, in main
[2019-02-11 15:28:20,745: WARNING/MainProcess] _main()
[2019-02-11 15:28:20,745: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/celery.py", line 322, in main
[2019-02-11 15:28:20,757: WARNING/MainProcess] cmd.execute_from_commandline(argv)
[2019-02-11 15:28:20,757: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
[2019-02-11 15:28:20,758: WARNING/MainProcess] super(CeleryCommand, self).execute_from_commandline(argv)))
[2019-02-11 15:28:20,758: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/base.py", line 275, in execute_from_commandline
[2019-02-11 15:28:20,766: WARNING/MainProcess] return self.handle_argv(self.prog_name, argv[1:])
[2019-02-11 15:28:20,766: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/celery.py", line 488, in handle_argv
[2019-02-11 15:28:20,767: WARNING/MainProcess] return self.execute(command, argv)
[2019-02-11 15:28:20,767: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/celery.py", line 420, in execute
[2019-02-11 15:28:20,767: WARNING/MainProcess] ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
[2019-02-11 15:28:20,767: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/worker.py", line 223, in run_from_argv
[2019-02-11 15:28:20,769: WARNING/MainProcess] return self(*args, **options)
[2019-02-11 15:28:20,769: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/base.py", line 238, in __call__
[2019-02-11 15:28:20,769: WARNING/MainProcess] ret = self.run(*args, **kwargs)
[2019-02-11 15:28:20,769: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bin/worker.py", line 258, in run
[2019-02-11 15:28:20,770: WARNING/MainProcess] worker.start()
[2019-02-11 15:28:20,770: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/worker.py", line 210, in start
[2019-02-11 15:28:20,770: WARNING/MainProcess] self.stop(exitcode=EX_FAILURE)
[2019-02-11 15:28:20,771: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/worker.py", line 253, in stop
[2019-02-11 15:28:20,771: WARNING/MainProcess] self._shutdown(warm=True)
[2019-02-11 15:28:20,771: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/worker.py", line 268, in _shutdown
[2019-02-11 15:28:20,771: WARNING/MainProcess] self.blueprint.stop(self, terminate=not warm)
[2019-02-11 15:28:20,772: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 177, in stop
[2019-02-11 15:28:20,772: WARNING/MainProcess] self.on_stopped()
[2019-02-11 15:28:20,772: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/worker.py", line 164, in on_stopped
[2019-02-11 15:28:20,773: WARNING/MainProcess] self.consumer.shutdown()
[2019-02-11 15:28:20,773: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 352, in shutdown
[2019-02-11 15:28:20,773: WARNING/MainProcess] self.blueprint.shutdown(self)
[2019-02-11 15:28:20,773: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 159, in shutdown
[2019-02-11 15:28:20,774: WARNING/MainProcess] self.send_all(parent, 'shutdown')
[2019-02-11 15:28:20,774: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/bootsteps.py", line 151, in send_all
[2019-02-11 15:28:20,774: WARNING/MainProcess] fun(parent, *args)
[2019-02-11 15:28:20,774: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/events.py", line 69, in shutdown
[2019-02-11 15:28:20,794: WARNING/MainProcess] self._close(c)
[2019-02-11 15:28:20,794: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/celery/worker/consumer/events.py", line 63, in _close
[2019-02-11 15:28:20,794: WARNING/MainProcess] ignore_errors(c, dispatcher.connection.close)
[2019-02-11 15:28:20,794: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/common.py", line 295, in ignore_errors
[2019-02-11 15:28:20,796: WARNING/MainProcess] return fun(*args, **kwargs)
[2019-02-11 15:28:20,796: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 372, in release
[2019-02-11 15:28:20,796: WARNING/MainProcess] self._close()
[2019-02-11 15:28:20,796: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 338, in _close
[2019-02-11 15:28:20,797: WARNING/MainProcess] self._do_close_self()
[2019-02-11 15:28:20,797: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/connection.py", line 331, in _do_close_self
[2019-02-11 15:28:20,797: WARNING/MainProcess] self.transport.close_connection(self._connection)
[2019-02-11 15:28:20,797: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 953, in close_connection
[2019-02-11 15:28:20,799: WARNING/MainProcess] channel.close()
[2019-02-11 15:28:20,799: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 852, in close
[2019-02-11 15:28:20,799: WARNING/MainProcess] self._disconnect_pools()
[2019-02-11 15:28:20,800: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 538, in _disconnect_pools
[2019-02-11 15:28:20,800: WARNING/MainProcess] async_pool.disconnect()
[2019-02-11 15:28:20,800: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/redis/connection.py", line 998, in disconnect
[2019-02-11 15:28:20,801: WARNING/MainProcess] connection.disconnect()
[2019-02-11 15:28:20,801: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 941, in disconnect
[2019-02-11 15:28:20,802: WARNING/MainProcess] channel._on_connection_disconnect(self)
[2019-02-11 15:28:20,802: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 546, in _on_connection_disconnect
[2019-02-11 15:28:20,803: WARNING/MainProcess] self.connection.cycle._on_connection_disconnect(connection)
[2019-02-11 15:28:20,803: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/transport/redis.py", line 282, in _on_connection_disconnect
[2019-02-11 15:28:20,803: WARNING/MainProcess] self.poller.unregister(connection._sock)
[2019-02-11 15:28:20,803: WARNING/MainProcess] File "/usr/home/grinch/envs/grinch/releases/20190210185743/grinch/env/lib/python3.7/site-packages/kombu/utils/eventio.py", line 209, in unregister
[2019-02-11 15:28:20,804: WARNING/MainProcess] self._quick_unregister(fd)
[2019-02-11 15:28:20,804: WARNING/MainProcess] ValueError
[2019-02-11 15:28:20,804: WARNING/MainProcess] :
[2019-02-11 15:28:20,804: WARNING/MainProcess] file descriptor cannot be a negative integer (-1)
[2019-02-11 15:28:20,804: DEBUG/MainProcess] removing tasks from inqueue until task handler finished

 -------------- celery@dev01 v4.2.1 (windowlicker)
---- **** ----- 
--- * ***  * -- FreeBSD-12.0-RELEASE-p3-amd64-64bit-ELF 2019-02-11 15:28:19
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         grinch:0x802663c18
- ** ---------- .> transport:   redis://localhost:6379/0
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery


[tasks]
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . comments.tasks.process_comment_mentions
  . grinch.tasks.log_invoice_action
  . grinch.tasks.run_maintenance_tasks
  . grinch.tasks.run_orgs_tasks
  . grinch.tasks.send_activation_reminders
  . grinch.tasks.send_email_broadcast
  . grinch.tasks.send_invoice_client_submit_email
  . grinch.tasks.send_simple_email
  . grinch.tasks.send_status_update_csv_email
  . grinch.tasks.update_files_permissions
  . payments.tasks.process_payments
  . payments.tasks.send_client_payment_invoice_email
  . payments.tasks.send_invoice_payment_refund_email
  . payments.tasks.send_org_payment_invoice_email
  . web.core.celery.debug_task

[georgepsarakis]

@petersanchez could you please try upgrading the redis client library? Could this error be related to the issue described here https://github.com/andymccurdy/redis-py/commit/4e1e74809235edc19e03edb79c97c80a3e4e9eca ?

[petersanchez]

Same issue with redis==3.1.0 and Kombu==4.3.0.

Downgraded Kombu==4.2.2-post1, keeping the updated redis, and it works as expected.

[georgepsarakis]

I found this change that possibly explains the issue.

For reference, all changes between 4.2.2-post1 and 4.3.0: https://github.com/celery/kombu/compare/v4.2.2-post1...v4.3.0

[lithammer]

I come bearing two solution proposals for this, #1011 and #1012.

[trunet]

is there any ETA for 4.3.1 with this fix?

[trunet]

replying to myself: 4.4.0 is released and it fixed this issue for me. Thanks!

[petersanchez]

Confirmed 4.4.0, redis==3.2.0 fixes this for me as well.