Cocotb: Stack overflow issue with lots of coroutines waiting on queues

[alexforencich]

I'm running in to an issue where I have a bunch of coroutines that are blocked on queue.get() and for some reason this is causing a stack overflow in the scheduler. Minimal example:

async def test_cr(queue):
    item = await queue.get()


async def run_test(dut):
    k = 1024

    queue_list = [Queue() for k in range(k)]
    cr_list = [cocotb.scheduler.start_soon(test_cr(q)) for q in queue_list]

    await Timer(10, 'ns')

If I run that, I get something like this:

     0.00ns ERROR    ..coroutine.Task 491.0x7fd8bc463ee0        scheduler.py:523  in _unschedule                     Exception raised by this forked coroutine
Fatal Python error: _Py_CheckRecursiveCall: Cannot recover from stack overflow.
Python runtime state: initialized

Current thread 0x00007fd8bea76d00 (most recent call first):
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/triggers.py", line 396 in __init__
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/triggers.py", line 449 in wait
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/queue.py", line 126 in get
  File "/home/alex/Projects/cocotb/stack_overflow_test/test.py", line 39 in test_cr
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/outcomes.py", line 36 in send
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/decorators.py", line 201 in _advance
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 857 in _schedule
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 719 in add
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 909 in _schedule
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 719 in add
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 909 in _schedule
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 719 in add
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 909 in _schedule
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 719 in add
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 909 in _schedule
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 719 in add
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 909 in _schedule
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 719 in add
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 909 in _schedule
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 719 in add
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 909 in _schedule
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 719 in add
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 909 in _schedule
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 719 in add
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 909 in _schedule
  File "/home/alex/.local/lib/python3.9/site-packages/cocotb/scheduler.py", line 719 in add

If I set k = 512, I get a different stack trace:

 0.00ns ERROR    ..coroutine.Task 491.0x7feba324e7f0        scheduler.py:523  in _unschedule                     Exception raised by this forked coroutine
 0.00ns ERROR    cocotb.regression                         regression.py:408  in _score_test                     Test Failed: run_test_001 (result was RecursionError)
                                                                                                                 Traceback (most recent call last):
                                                                                                                   File "/home/alex/Projects/cocotb/stack_overflow_test/test.py", line 39, in test_cr
                                                                                                                     item = await queue.get()
                                                                                                                   File "/home/alex/.local/lib/python3.9/site-packages/cocotb/queue.py", line 126, in get
                                                                                                                     await event.wait()
                                                                                                                   File "/home/alex/.local/lib/python3.9/site-packages/cocotb/triggers.py", line 449, in wait
                                                                                                                     return _Event(self)
                                                                                                                   File "/home/alex/.local/lib/python3.9/site-packages/cocotb/triggers.py", line 396, in __init__
                                                                                                                     PythonTrigger.__init__(self)
                                                                                                                 RecursionError: maximum recursion depth exceeded

If I set k = 256, it does not crash.

I'm running the latest version of cocotb from git. I ran this with icarus verilog. I think the max recursion depth is set to 1000.

[marlonjames]

This is a consequence of how the scheduler handles queued coroutines, causing recursion instead of correctly using a loop.

_schedule() pops the first pending coroutine and calls itself, recursively, blowing the stack if there are enough in the queue.

[ktbarrett]

Is there a workaround for now?

[marlonjames]

We could move handling the pending coroutines to the event loop, after schedule and before handling pending events. I think that would be equivalent scheduling order.

[ktbarrett]

I mean for @alexforencich who is using 1.5 right now. Would fork avoid this issue?

EDIT: Switching to cocotb.fork works. This is likely because it runs the coroutine until the first await moving the pending coroutine off the queue and onto waiting for some event. Another solution would be to run await NullTrigger() after every start_soon. This would be similar to the await cocotb.scheduler.start(coro) function we had talked about introducing to accompany start_soon.

[alexforencich]

I moved from fork to start_soon as you guys told me that fork is going to be deprecated. Anyway, it's not a major show stopper right now, I'll just hold off these changes until this is fixed and back-ported to 1.5.x.