Kombu: Prefetch with SQS enters infinite loop

[maedoc]

I'm using kombu (4.0.2 or git master) with the SQS transport, starting Celery with -c 2 and pretty quickly (within seconds of starting Celery, after processing a handful of items), I see the master process at 100% CPU. GDB attach, py-bt always shows

Traceback (most recent call first):
  File "/var/virtualenvs/dbg/lib/python3.5/site-packages/kombu-4.0.2-py3.5.egg/kombu/async/hub.py", line 193, in call_soon
    if not isinstance(callback, Thenable):
  File "/var/virtualenvs/dbg/lib/python3.5/site-packages/kombu-4.0.2-py3.5.egg/kombu/transport/SQS.py", line 284, in _loop1
    self.hub.call_soon(self._schedule_queue, queue)
  File "/var/virtualenvs/dbg/lib/python3.5/site-packages/kombu-4.0.2-py3.5.egg/kombu/transport/SQS.py", line 293, in _schedule_queue
    self._loop1(queue)
  File "/var/virtualenvs/dbg/lib/python3.5/site-packages/vine/promises.py", line 136, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/var/virtualenvs/dbg/lib/python3.5/site-packages/kombu-4.0.2-py3.5.egg/kombu/async/hub.py", line 282, in create_loop
    item()
....

In the SQS channel, I see that such a loop can occur if qos.can_consume() always returns False, but I don't understand enough of the virtual channel implementation (yet) to see why it would get stuck in that state, when the SQS queue has messages available and celery workers are idling (i.e. waiting on _recv somewhere in billiard).

Any ideas or debugging tips? Are there some parameters to be tuned to avoid this?

FWIW This doesn't occur with Celery 3.1.25 & kombu 3.0.37, but we'd like to upgrade.

_edit_ the Celery config is really simple, just pointing at a valid SQS queue and a few routes & queues defined.

_edit 2_ To wit, starting Celery with -c 1 -P solo, items are processed normally. I've also noticed that exactly 8 tasks are received each time before the loop, which suggests (to me) that the channel is working, but the workers aren't consuming the messages?

[maedoc]

Looks like an known issue in Celery; I didn't find this in my prior searches, but fix described in celery/issues/3712.

[maedoc]

Reopening for specific issue that prefetching enters infinite loop. Despite the workaround, in our case, with thousands of tasks, we'd prefer our workers don't try to fetch them all at once.

[maedoc]

I set up same environment on a single macOS machine and can't seem to reproduce. Could this be a Linux specific or multi-host specific problem?

[maedoc]

We switched to the Redis broker since we already had an instance running (in Elasticache or locally) and haven't had a single issue since.