For a different and interesting approach to asynchronous processing, see @bboser's https://github.com/bboser/eventio for a highly constrained way of using async / await, especially the README and https://github.com/bboser/eventio/tree/master/doc. Perhaps some combination of these makes sense.

dhalbert on 6 Dec 2018

I'm unqualified at this point to talk about implementation, but from an _end user_ perspective I like the idea of this abstraction quite a bit. It feels both like a way to shortcut some ad hoc polling loop logic that I suspect people duplicate a lot (and often badly), and also something that could be relatively friendly to people who came up on high-level languages in other contexts.

People aren't going to stop wanting interrupts / parallelism, but this answers a lot of practical use cases.

brennen on 6 Dec 2018

I like event queues and I agree they are quite easy to understand and use, however, I'd like to point out a couple of down sides for them, so that we have more to discuss.

Queues need memory to store the events. Depending on how large the event objects are, how often they get added and how often you check them, this can be a lot of memory. Since events are getting created and added from a callback, that has to be pre-allocated memory. And I don't know of a good way of signalling and handling overflows in this case — depending on use case, you might want to have an error, drop old events, drop new events, etc. To save memory you might want to have an elaborate filtering scheme, and that gets complex really fast.
Queues encourage a way of writing code that introduces unpredictable latency. The way your code usually would flow, you would do your work for the given frame, then you would go through the content of all the event queues and act on them, then you would wait for the next frame. In many cases that is perfectly fine, but in some you would rather want to react to the event as soon as possible.
Every new kind of queue will need a new class and its own C code for the callback and handling of the data. So if you have a sensor that signals availability of new data with an interrupt pin, you will need custom C code that will get called, read the sensor readings and put them on the queue. That means that all async drivers would need to be built-in.
Sometimes a decision needs to be done while the event is being created, and can't wait. For example, in the case of the I2C slave, you need to ACK or NACK the data, and you have to hold the bus in a clock-stretch until you do.

That's all I can think of at the moment.

deshipu on 6 Dec 2018

Hm, I do not fully understand, why such a MessageQueue model should be easier to understand than callbacks. Maybe it's, because I am used to callbacks ;-)
What is so special with your target customers, that you think, they do not understand callbacks?

I think, you have to invest much more brain in managing a couple of MessageQueues for different types of events (ethernet, i2c, timer, exceptions, spi, .....) or one MessageQueue, where you have to distinguish between different types of events, than in implementing one callback for each type of event ant pass it to a built in callback-handler.

def byte_reader(byte):
      deal_with_the(byte)

uart = busio.UART(board.TX, board.RX, baudrate=115200, byte_reader)

framlin on 19 Dec 2018

I like the idea of message queues but I'm not convinced that they're any easier to understand than interrupt handers. Rather I think that conceptually interrupt handlers/callbacks are relatively easy to understand but understanding how to work with their constraints is where it gets a bit more challenging. Message queues are a good way of implementing the "get the operable data out of the hander and work on it in the main loop" solution to the constraints of interrupt handlers but as @deshipu pointed out, there are still good reasons to need to put some logic in the handler. Maybe both?

Similarly I like how eventio works but I think it's even more confusing than understanding and learning to work with the constraints of interrupt handlers. That in mind, it's tackling concurrency in a way that I think might be more relatable to someone who came to concurrency from the "why can't I blink two leds at once" angle.

One thing I was wondering about is what a bouncy button would do to a message queue. Ironically I think overflow might actually be somewhat useful in this case as if the queue was short enough you'd possibly lose the events for a number of bounces (but not all of them unless your queue was len=1. I'll have to ponder this one further). With a longer queue you could easily write a debouncer by looking for a time delta between events above a threshold.

No matter how you slice it, concurrency is a step beyond the basics of programming and I don't think any particular approach is going to allow us to avoid that. It seems to me that we're being a bit focused choosing a solution to a set of requirements that we don't have a firm grasp on yet. I think it's worth taking the time to understand who the users of this solution are and what their requirements are.

siddacious on 19 Dec 2018

See #1415 for an async/await example.

notro on 20 Dec 2018

What is so special with your target customers, that you think, they do not understand callbacks?

The problem is not with the callback mechanism itself, but in the constraint that MicroPython has that you can't allocate memory inside a callback. This is made much more complex than necessary by the fact that Python is a high level language with automatic memory management, that lets you forget about memory allocation most of the time, so it's not really obvious what operations can be used in a callback, and how to work around the ones that can't.

deshipu on 22 Dec 2018

One solution would be to enable MICROPY_ENABLE_SCHEDULER and _only_ allow soft IRQ's, running the callback inline with the VM. This would prevent people from shooting themselves in the foot.

Refs:

notro on 22 Dec 2018

In my implementation of interupts I’ve added a boolean “fast” that defaults
to false and controls running the handler via the scheduler (no constraints
on allocation) or directly in cases where latency is critical.

I am also considering running the gc automatically in the eventio loop but
have not yet considered all potential side effects. Ditto permitting
interrupt handlers is straightforward in eventio, for cases when they are
needed.

Bernhard

On Sat, Dec 22, 2018 at 02:27 Noralf Trønnes notifications@github.com
wrote:

One solution would be to enable MICROPY_ENABLE_SCHEDULER and only allow
soft IRQ's, running the callback inline with the VM. This would prevent
people from shooting themselves in the foot.

Refs:

stm32:Handle_EXTI_Irq
http://elixir.tronnes.org/circuitpython/4.0.0-alpha.4/source/ports/stm32/extint.c#L520

esp32:machine_timer_isr
http://elixir.tronnes.org/circuitpython/4.0.0-alpha.4/source/ports/esp32/machine_timer.c#L99

py/vm.c
http://elixir.tronnes.org/circuitpython/4.0.0-alpha.4/source/py/vm.c#L1297

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/adafruit/circuitpython/issues/1380#issuecomment-449534913,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC3bpFgD_0IZDU1HT4_V5XkegYEUsY5zks5u7YqOgaJpZM4ZFtp-
.

bboser on 22 Dec 2018

Thank you all for your thoughts and trials on this. I'll follow up in the near future but am deep in Bluetooth at the moment. The soft interrupts idea and the simplified event loop / async / await stuff is very interesting. I think we can make some progress on this.

dhalbert on 26 Dec 2018

From my experience with what I think is the CircuitPython audience (I teach technology to designers), I don't think message queues are easier to understand than other approaches, and are probably harder in many cases. As @siddacious says, concurrency takes a while for newcomers to wrap their heads around no matter what the method.

I also think it's important to distinguish between event driven needs and parallelism. In my experience, the most common need amongst my students is doing multiple things at once, e.g. fading two LEDs at different rates, and perhaps doing this while polling a distance sensor. This requirement is different from the straw man example above.

Some possible directions:

MicroPython supports _threads. To me, this would be a fine solution, especially since it is already documented and there are examples out there. Mine for example: https://github.com/pvanallen/esp32-getstarted/blob/master/docs/multitasking.md
Implement something along the lines of the TaskScheduler Arduino library, which is great: https://github.com/arkhipenko/TaskScheduler

pvanallen on 3 Apr 2019

I have done some more thinking and studying on this topic. In particular, I've read (well, read the front part; skimmed a lot more) _Using Asyncio in Python 3_, and I've read about curio and trio (github), which is even simpler than curio, started by @njsmith. Trio emphasizes "structured concurrency". I also re-reviewed @bboser 's https://github.com/bboser/eventio, and @notro 's example of a simple async/await system in #1415. This is also reminiscent of MakeCode's multiple top-level loops.

Also I had some thoughts about some very simple syntax for an event-loop system I thought might be called when. Here are some strawman doodlings, which are a lot like eventio or @notro's proposal. I am _not_ thinking about doing asynchronous stream or network I/O here, but about handling timed events in a clean way, and about handling interrupts or other async events. Not shown below could be some kind of event queue handler, which would be similar to the interrupt handler.

Maybe the functions below need to be async? Not sure; I need to understand things further. I'm more interested in the style than the details right now.

Note that that when.interval() subsumes even doing await time.sleep(1.0) or similar: it's built in to the kind of when.

I am pretty excited about this when/eventio/notro-event-loop/trio/MakeCode model, as opposed to asyncio, which is very complex. asyncio started from a goal of handling tons of network I/O, and is also partly a toolkit for writing concurrency packages, as Caleb Hattingh (author of the asyncio book above) points out.

# A cute name
import when

import board, digitalio

d1 = digitalio.DigitalInOut(board.D1)
d1.switch_to_output()

d2 = digitalio.DigitalInOut(board.D2)
d2.switch_to_output()

d3_interrupt = digitalio.Interrupt(board.D3, change=digitalio.Interrupt.RISIING)

#################################
# Decorator style of using `when`

#Starts at 0.0 seconds, runs every 1.0 seconds
@when.interval(d1, interval=1.0)
def blink1(pin):
     pin.value = not pin.value

# Starts at 0.5 seconds, runs every 1.0 seconds
@when.interval(d2, interval=1.0, start_at=0.5)
def blink2(pin):
     pin.value = not pin.value

# This is a soft interrupt. The actual interrupt will set a flag or queue an event.
@when.interrupt(d3_interupt)
def d3_interrupt_handler(interrupt):
    print("interrupted")

# Start an event loop with all the decorated functions above.
when.run()

####################################
# Programmatic style of using `when`

def toggle_d1():
     d1.value = not d1.value

def toggle_d1():
     d2.value = not d2.value

when.interval(toggle_d1, interval=1.0)
when.interval(toggle_d2, interval=1.0, start_at=0.5)

def d3_interrupt_handler():
    print("interrupted")

when.interrupt(d3_interrupt_handler, d3_interupt)

when.run()

dhalbert on 8 Apr 2019

I started a thread in the trio forum: https://trio.discourse.group/t/python-for-microcontrollers-and-structured-concurrency/154

dhalbert on 8 Apr 2019

For comparison, here is how you would do it with some kind of async framework (let's call it "suddenly"):

import suddenly
import digitalio

async def blink1(pin):
    pin.switch_to_output()
    while True:
        pin.value = not pin.value
        await suddenly.sleep(1)

async def blink2(pin):
    await suddenly.sleep(0.5)
    await blink1(pin)

async def interrupt(pin):
    while True:
        await pin.change(digitalio.Interrupt.RISING)
        print("interrupted")

suddenly.start(blink1(digitalio.DigitalInOut(board.D1)))
suddenly.start(blink2(digitalio.DigitalInOut(board.D2)))
suddenly.start(interrupt(digitalio.DigitalInOut(board.D3)))
suddenly.run()

or a shorter:

suddenly.run(
    blink1(digitalio.DigitalInOut(board.D1)),
    blink2(digitalio.DigitalInOut(board.D2)),
    interrupt(digitalio.DigitalInOut(board.D3)),
)

deshipu on 8 Apr 2019

@deshipu Right, right, yes, we're talking about the same thing! I left out the asyncs and awaits, or propose they might be hidden by the when mechanism. I'm trying to come up with strawman pseudocode before getting into the details. I freely admit there may be mistakes in my thinking here, but I'm trying not to get sucked into the details of an existing paradigm yet.

trio and curio use async with as a style. I'll doodle with the same style:

import when

# similar defs as in previous comment ...
# ...

# I am deliberately leaving out the `async`s because I want to understand we actually need them and when we don't. How much can we hide in the library?
with when.loop() as loop:
    loop.interval(blink1, 1.0)
    loop.interval(blink2, 1.0, start_at=0.5)
    loop.interrupt(some_function)
    loop.event(event_handler, queue=some_event_queue)
    loop.done_after(60.0)         # stop loop after 60 seconds
    loop.done(some_predicate_function)

# ^^ Runs the loop until done.

    # in general:
    # loop.something(function_name_or_lambda, args=(), more args if necessary)

dhalbert on 8 Apr 2019

It's not possible to "hide" the async keyword in the library, because then you create a function that is being invoked when you "call" it. With async, "call" will simply produce an iterator object, which the library can then exhaust in its main loop, handling any Futures it gets from it along the way.

I think that syntax makes a very big difference for beginners, and that the "callback" style that you propose is very difficult to grasp for people not used to it. With the async style syntax, you basically write each function as if it was the only function in your program (you can test it as the only function), and then add the async to it and await to all parts that block, and it just works.

deshipu on 8 Apr 2019

@deshipu Thank you for the enlightenment. I'll rework the examples with async. I think we still might be able to avoid explicit awaits in some cases. I like the interval() style, which pushes the timing into when instead of making it part of the function. But maybe that is too much of a toy example.

dhalbert on 8 Apr 2019

@deshipu But I am seeing trio use what you call the "callback" style:
https://trio.readthedocs.io/en/latest/tutorial.html#okay-let-s-see-something-cool-already
Notice child1, not child1(), below, etc.
There are other examples where the args are separated from the function, e.g. start(fn, arg).

Do you have an example of an await/async library that uses your style?

Trimmed example from link above:

```python3
async def child1():
# ...

async def child2():
# ...

async def parent():
print("parent: started!")
async with trio.open_nursery() as nursery:
nursery.start_soon(child1)
nursery.start_soon(child2)

trio.run(parent)

dhalbert on 8 Apr 2019

Here is a very simple implementatin of such an async framwork, that can only await on a sleep function:

import time


TASKS = []


class Task:
    def __init__(self, when, coro):
        self.coro = coro
        self.when = when


def sleep(seconds):
    return [seconds]


def start(*awaitables, delay=0):
    now = time.monotonic()
    for awaitable in awaitables:
        TASKS.append(Task(now + delay, awaitable))


def run(*awaitables):
    start(*awaitables)
    while TASKS:
        now = time.monotonic()
        for task in TASKS:
            if now >= task.when:
                try:
                    seconds = next(task.coro)
                except StopIteration:
                    TASKS.remove(task)
                else:
                    task.when = now + seconds


# async def test():
def test1():
    for i in range(10):
        print(i)
        # await sleep(1)
        yield from sleep(1)

def test2():
    yield from sleep(0.5)
    yield from test1()

run(test1(), test2())

deshipu on 8 Apr 2019

This presentation explains the trampoline trick that it uses:
https://www.youtube.com/watch?v=MCs5OvhV9S4

deshipu on 8 Apr 2019

As for examples, asyncio uses that style: https://docs.python.org/3/library/asyncio.html

deshipu on 8 Apr 2019

Of course in a proper implementation you would use a priority queue for tasks that are delayed, and a select() (with a timeout equal to the time for the next item in the priority queue) for tasks that are blocked on input/output, such as interrupts, reading, or writing.

deshipu on 8 Apr 2019

I mocked up a simple framework that lets you sleep and wait for a pin change (uses polling internally): https://github.com/deshipu/meanwhile

deshipu on 8 Apr 2019

@dhalbert I think your "when" proposal has promise. I like the simplicity of it. And compared to other approaches, I think moving the interval outside the def is better because it makes the method more reusable and cleaner. From my perspective, the use of "callbacks" is not that hard for people to understand, whereas the use of await/yield requires explaining cooperative multitasking etc.

I'm interested to hear how your approach would handle terminating an interval. And did you consider giving an interval an optional number of repeats? E.g.

when.interval(toggle_d2, interval=1.0, start_at=0.5, repeats=20)

pvanallen on 8 Apr 2019

Since with async the interval is just a loop with a delay, you can easily control the number of repeats:

async def blink1(pin, interval, start_at,  repeats):
    await meanwhile.sleep(start_at)
    for repeat in range(repeats):
        pin.value = not pin.value
        await meanwhile.sleep(interval)

meanwhile.run(blink1(pin_d2, 1.0, 0.5, 20))

deshipu on 9 Apr 2019

I added a mock of a hypothetical implementation of an async framework if we had the select call available (just for file operations for now, but for sercoms or interrupts it would be similar): https://github.com/deshipu/meanwhile/blob/master/meanwhile_select.py

deshipu on 9 Apr 2019

@deshipu

Since with async the interval is just a loop with a delay, you can easily control the number of repeats

Sure, but for beginning coders, I think keeping the common aspects of these intervals out of the function definitions makes them simpler to write and understand. E.g. this is boiled down to the essence of the task:

def toggle_d1():
     d1.value = not d1.value

and leaves all the bookkeeping for delay, start time, and repeats to the library. For my students, this simplification would be very helpful in their gaining confidence and trying out new things like cooperative multitasking. Over time, they'll then develop a greater understanding and be able to add more complex features.

I would like to have arguments, and I wonder if @dhalbert's when approach will permit it? E.g.:

def toggle(pin):
     pin.value = not pin.value

when.interval(toggle(pin), interval=1.0, start_at=0.5, repeats=20)

pvanallen on 15 Apr 2019

@pvanallen in my limited experience, it's really difficult to teach people this style of programming (where the inside of the loop is in a separate function from the rest of the code), and it results in code that is difficult to follow. You can of course do it with async if you really hate yourself:

def blink_inner(pin):
    pin.value = not pin.value

async def blink1(pin, interval, start_at,  repeats, func=blink_inner):
    await meanwhile.sleep(start_at)
    for repeat in range(repeats):
        func(pin)
        await meanwhile.sleep(interval)

meanwhile.run(blink1(pin_d2, 1.0, 0.5, 20, blink_inner))

deshipu on 15 Apr 2019

Come to think of it, it should be trivial to make a decorator function that would add what blink1 does in the above example to any function.

deshipu on 15 Apr 2019

Sorry to come to this late. As context I am a long time embedded control developer who has written a couple of low level preemptive tasker/schedulers (easier than you might guess) and I've least read several versions of UNIX/Linux schedulers. I used to be up on current theory, including studying how an OS scheduler problem (priority inversion) killed the Pathfinder Mars lander, but that was a while ago. I just wanted to add one thing. Usually concurrency is implemented with a small number of core tools (message queue, resource locks, semaphores ... choose one) then more complex abstractions are built on top of them. I suggest breaking the problem into two parts, 1) a basic core abstraction for internal implementation, and 2) User visible interfaces (API) which use the core abstraction to do the heavy lifting. A good choice of core abstraction gives you a simple reliable building block to implement different user interfaces. It doesn't have to be simple to understand. It may be that only developers see it. It just has to be reliable. Simplicity is the job of the user API.

zencuke on 1 May 2019

This issue came up in a discussion with @tannewt during PyCon19. Only skimmed the comments here, it looks like no one has mentioned the simple scheduler/event loop that is part of the CPython standard library and may be of interest for this:

https://docs.python.org/3/library/sched.html

skoslowski on 5 May 2019

👍1

I have ported the Trio core to MicroPython. No interrupt support yet, because lack of time, but adding that shouldn't be too difficult.

Interrupt handling would most likely look like this:

import trio
async def main():
    p = Pin(…)
    async for evt in p.interrupt():
        print("Hey, pin changed:", evt)
trio.run(main)

smurfix on 8 Jun 2019

I'm coming to this late too but I guess I don't understand the cons to asyncio; for me it's simple, well documented, well supported and used widely for Python. It requires few language additions and the implementation is available as a python module. Asyncio, like the message queue example, is also synchronous - it provides concurrency but not parallelism - so is relatively straightforward to reason through when issues arise.

I use it in many of my MicroPython applications and I'm yet to see a more straightforward way to achieve concurrency. The lack of asyncio on CircuitPython is one of the stumbling blocks to me using the platform.

It seems especially odd to avoid since one of CircuitPython's goals is CPython compatibility...

I do encourage folks to review Peter Hinch's extensive work on asyncio in MicroPython. There's a lot of useful information in there as to how asyncio can be used effectively on an embedded platform.

I'd also suggest supporting interrupts and threads (both are supported on MicroPython and are nicely orthogonal to asyncio) but they're actually less critical to me than asyncio.

Now, it's possible I'm a _little_ biased since I gave a talk on Asyncio on (Micro)Python at last years' PyCon AU. :)

mattytrentini on 19 Jun 2019

👍1

@mattytrentini When you say asyncio, do you mean asyncio, or uasyncio? My impression from talking to Paul Sokolovsky a bit is that they have substantially different APIs, and in fact he strongly disagreed with a lot of choices that asyncio made. Also, the main asyncio devs say that they think Trio generally did things better and their goal is to incrementally convert asyncio into trio... (of course there are a lot of complications here so no-one knows yet how it will all play out).

Regarding the bigger issues in this thread: the simple @when style works really well for describing simple behaviors – if the program you want to write is "once a second, blink a light", then you can't really beat a library that lets you write basically translate that sentence directly into Python. The downside of this kind of callback scheduling is that if you want to express more complex behavior, then you're stuck writing state machines by hand, and there's really no mechanism for composing together simple behaviors to make more complex ones. This makes it incredibly difficult to write more complex programs. That's the motivation for async/await and libraries like trio/asyncio – to let you use Python's normal mechanisms like functions, loops, etc. to describe complex, composable asynchronous behaviors. (If anyone's seen my pycon talk, this is why the twisted happy eyeballs code is so hard to understand – it's just a raw state machine with no abstraction mechanism.)

Or a simple, concrete example would be, in Nina's keynote at PyCon this year, she had a demo of making the lights flash through a cycle of several colors. To do that, she had to track the program state by hand, using code like color_pos = (color_pos + 1) % len(colors). When I say "raw state machine", that's the "state" I mean. OTOH if you can use regular Python tools, you can write something like:

async def cycle_lights(colors):
    for color in itertools.cycle(colors):
        await wait_for_button_press()
        cpx.pixels.fill(color)

Now that color_pos = (color_pos + 1) % len(colors) is still happening, but it's hidden away inside higher-level abstractions – for and itertools.cycle.

So I actually don't know what approach is better for CircuitPython. Is it more important to make it as easy as possible to get started? or is it more important to give kids a toolset that can grow with them?

njsmith on 19 Jun 2019

From my perspective: yes you have that nice little @when.interval which works well enough – except when it doesn't: after you learn how to do a simple blinkenlight, the next idea is to vary the blink frequency. Boom you're back at the start line and need to figure out how to do it manually anyway.

The point about async-anything is that you can open-code your state machines, which is much easier to understand than random global (or, if you're good, instance) state variables. Plus, you don't need to deal with threads and locks and concurrency bugs related thereto.

The point where trio gets a leg up on asyncio is where you need to stop doing things / do things differently. For instance, I need one thing to happen when the device that controls my door has MQTT connectivity – but it must do something entirely different when it doesn't, otherwise I'm locked out.

With trio this is dead easy, I just tell some simple keepalive code to cancel the MQTT task and I can be reasonably sure that the Trio runtime takes all my sub-tasks and handlers down cleanly. Problem solved. With asyncio? I do not want to go there ever again.

smurfix on 19 Jun 2019

Paul is quite opinionated ;) - but in any case the uasyncio API follows asyncio _reasonably_ closely. Certainly to most _users_ of the API it will feel similar if not the same.

I also agree that you Trio folks have made some _excellent_ decisions and some significant improvements. But I'd rather wait for it to be accepted through a PEP process before having people build on it (and the interface possibly changing as it goes through review). Asyncio is what we have, and it's still reasonably good.

While the @when syntax is neat for those simple cases I feel it's not such an improvement to warrant the machinery it's hiding. Further, the asyncio equivalent isn't significantly worse IMO and, as you've alluded to, it provides tools that a user can grow into.

As for cancelling, yes, Trio gets this right. But we have workarounds in asyncio that, while not great, are pragmatic.

mattytrentini on 20 Jun 2019

Interesting article about asyncio: https://www.roguelynn.com/words/asyncio-we-did-it-wrong/

tannewt on 29 Jul 2019

Right. They even complain that they don't have Trio's "nursery" concept. (Well, they could use the "anyio" wrapper, which does provide the equivalent.)

It'd be interesting to rewrite the same code in Trio and compare the pitfalls.

smurfix on 30 Jul 2019

When I read that article I found it quite _pro_-asyncio, despite the inflammatory title: "I do believe asyncio is quite user-friendly, but I did underestimate the inherit complexity concurrent programming brings."

In fact, I think it _supports_ the stance that you should _not_ try to build a simpler system - asynchronous code is difficult, as the author says: "whether you use asyncio, Twisted, Tornado, or Golang, Erlang, Haskell, whatever".

mattytrentini on 30 Jul 2019

In fact, I think it supports the stance that you should not try to build a simpler system - asynchronous code is difficult

Sure, but it doesn't have to be that difficult.

Writing something like the Happy Eyeballs algorithm takes 50 lines in Trio but 500 in "native" asyncio. That should tell us something. As should the fact that the people responsible for asyncio plan to evolve it into a Trio-ish direction as quickly as possible.

smurfix on 30 Jul 2019

👍1

Interesting pypi package related to our use of properties and async: https://async-property.readthedocs.io/en/latest/readme.html

tannewt on 2 Aug 2019

In Trio, we teach beginners that await blah() is a special kind of function call, that you use when calling special functions defined with async def. We don't teach about await obj syntax, bare coroutine objects, the awaitable protocol, or any of that. This means we can't use some clever tricks like async_property, but I think the simplification is worth it. (It also makes it pretty easy for linters to detect when you forgot an await, which is otherwise a really easy and confusing mistake to make.)

njsmith on 2 Aug 2019

Problem is, CircuitPython already uses properties for all the things that take time that you probably want to be asynchronous, like reading values from sensors or sending data to displays.

deshipu on 2 Aug 2019

You're going to have to break compatibility to make those async, though, one way or another.

I don't know what's best for your situation; I just wanted to point out some of the options and tradeoffs.

njsmith on 2 Aug 2019

asyncio is not hard to use, is standard and would help CircuitPython appeal even more to a particular set of users. It is hard to wrap your head around in an hour, and is not suitable for a person's "hello world" but anyone can learn the basic asyncio patterns; kids and adults alike. I would switch to MicroPython for uasyncio but for the CircuitPython device libraries I'm locked into.

Where I'm coming from: I have written asynchronous software in many languages and in coroutine, thread, coroutine+thread and ad-hoc state paradigms. Asynchronous code _is_ difficult, but not providing a higher order abstraction of some sort to achieve it just makes life more difficult for those who require it. asyncio has been the most pleasant experience I have had to date writing asynchronous code. It's powerful and its primitives are straightforward. That said I don't care particularly care how, but I'm eagerly watching for CircuitPython to offer a path to support syntax like sensor_value = await sensor.read() so I can delete hundreds of lines of code. Please do consider prioritizing this issue.

WarriorOfWire on 18 Jan 2020

Well, from my PoW trio's structured-programming abstractions (bounded scopes for tasks, deterministic cancellation and error propagation, and its avoidance of callbacks) avoid many pitfalls you tend to run into when writing async code "in the wild" and, frankly, "forces" me to write better and more concise programs.

smurfix on 18 Jan 2020

A few random thoughts:

Some programmers will be coming from MakeCode. Something similar or with easy-to-explain differences would be cool.

Some beginning programmers (usually those with entrepreneurial spirit) use LiveCode, a HyperCard-like language. LiveCode uses event/message handlers. Events and messages are handled only during waits and the like.

New programmers and programmers new to a way to do this often get confused about when events and callbacks are handled. Or about the changing of shared variables.

Perhaps the implementation should move into asynchronous processing in a uniform way, setting the groundwork for such in CircuitPython.

Dar-Scott on 18 Jan 2020

thanks for all the good thoughts. we def want to add some support for this. what would be extremely helpful is if folks could post up usage cases where they need concurrancy/async/sleep/interrupts. pseudocode or descriptions - there's a lot of cases and we want to make sure we cover them :)

ladyada on 18 Jan 2020

Concepts can build on those learned in beginning CP examples.

Events/callbacks
An example is digitalio, used to create an abstraction around a pin. One can change whether it is an input or output and so on. Input can have a pull-up. Perhaps an input can also have functions to call on change events. The function is passed the object. The added functionality can be a model for making other classes that have events. It is what to do when. Such objects might be called event objects. They add the concept of event.

The concept of change can be generalized so some sort of happening. This would include receiving a line from a UART.

A timer event class would be very handy and expected. Building on above, a timer event is based on a happening. The details can be changed even in its event function. A simple timer class can be used to build other timers.

When are event functions allowed to be called?

A concept learned is that things happen when you tell the board to do them. So a function handle_events() might be handy. It can have optional parameters of how many to do, or whether to sleep for a while. Recursion can be limited; a simple concept might be that an event function cannot be called if it is busy. An alternative is to prevent handle_events() from actually handling _any_ events when an event is active; that is, only one event can be executing. One possibility is to use time.sleep() with review as to whether that breaks some things. This means that all code sequences 'tween those are essentially critical sections, that is, synchronization is a cinch. You don't worry about it.

Alternatively, they can be called at any time. This requires added concepts and functions.

Events act like a simple function call, a concept learned. They are not executed in parallel, with lines interleaved or anything like that, a common point of confusion.

Events do not occur if the object no longer lives, however, if its event function is running, destruction is after it completes. No zombie events occur.

Little baby programs
This is an alternative to events. (One might ponder allowing both.)

The concept of a program can be built-upon for running little threadies or sub-programs. A couple concepts borrowed are load and control-C ("keyboard interrupt"). The words "run" and "stop" can be readily applied. Good words might be found in beginning examples.

Perhaps any function or an object with a run method can be run. For the function it is much like the program is the the body of the function and everything in scope is like a built-in capability.

Times to allow switching can be explicitly provided or implied in time.sleep(). Taking turns or passing the speaking baton might be concepts already understood. An alternative is to allow switching at returns and loop ends. The running function stops when it returns.

This requires some synchronization concepts and functions. It might be a memory hog.

Dar-Scott on 18 Jan 2020

Use case...

LED control. The state of device is reflected in one or more LEDs. (I recently used a light-pipe to the an on-board LED in a quick build.) Besides color, the blink is controlled. That might be off, fast blink, slow blink, or warble. The blink adds a distinction that is color-blind friendly, red might be only fast blink. The rate can be off a little and a little jitter is OK.

Dar-Scott on 18 Jan 2020

Another use case: MQTT. I control an LED strip via MQTT messages and the latency in handling the MQTT loop severally lowers the rate at which I can update the LEDS.

rdlaner on 18 Jan 2020

Use case...

Keypad handler. Keypad concepts include short tap, double tap, long press and shift. Sequences of taps and presses enabled by shifts (including the trivial single) trigger events that might be dynamically changed. Timing of button presses must be human. Debounce is handled. Events are allowed to change attributes.

Dar-Scott on 18 Jan 2020

Thanks for chiming in ladyada! Here is what I'm pining for every time I open my current project:

I'd like the hardware interrupts to be handled by the library and only used to wake my code in a normal execution context (i.e., I don't want to write interrupt handlers where I'm not allowed to allocate or call other functions. At least I don't want to _need_ to normally).
Every listener of an event (like an on_fall()) receives notification of that event when it occurs.

Here's a program that should swap 2 LED's back and forth lit/unlit:

button = adafruit_debouncer.Debouncer(digitalio.DigitalInOut(D1))
button.direction = digitalio.Direction.INPUT
led1 = digitalio.DigitalInOut(D2)
led2 = digitalio.DigitalInOut(D3)
led1.direction = digitalio.Direction.OUTPUT
led2.direction = digitalio.Direction.OUTPUT
led1.value = True
led2.value = False

# This is what we're going to do 2 different ways below:
def uncalled_equivalent_loop():
  while True:
    button.update()  # debouncer
    if button.fell:
      led1.value = not led1.value
      led2.value = not led2.value


# First way, with a program loop per coroutine.
def explicit_manual_coroutines():
  # inner function just to keep the global namespace clean
  # toggle_pin essentially becomes a top_level program loop.
  async def toggle_pin( pin: digitalio.DigitalInOut, button: digitalio.DigitalInOut ) -> :
    while True:
      value = await button.on_change()  # like rise/fall on Debounce
      if value:  # swap pin on fall.  Could have just used on_fall() instead
        pin.value = not pin.value
  # Register the coroutines (program loop _facets_) with whatever coroutine impl
  run_this_coroutine(toggle_pin(led1, button))
  run_this_coroutine(toggle_pin(led2, button))


def quality_of_life_style():
  # on_* functions _should_ absolutely expect async and non-async functions!
  button.on_fall(lambda: led1.set_value(not led1.value))
  button.on_fall(lambda: led2.set_value(not led2.value))


if __name__ == '__main__':
  # start up with button pressed to use explicit manual coroutines
  if button.value:
    explicit_manual_coroutines()
  else:
    quality_of_life_style()

  # Yields control to the coroutine runner, leaving it to sleep and wake async contexts as they are eligible and processor time is available.
   # you do this now instead of `while True:` when you are building an async project.
  async_coroutine_feature.run_forever()

I've not tested it but the intent should be clear. Both of the user async styles above should ideally be supported, though the 2nd one is much more convenient for typical programs (obvs you don't have to use a lambda, I'm just showing ideal compactness). This toy does not illustrate the high value of async/await coroutines but it captures the 2 ways I want to use them.

Today I hack around the lack of coroutines via ad-hoc state and loop()'s on all of my objects that pulse to the beat of the while True. This is pretty wasteful and costs hundreds of microseconds per loop, as almost every time around all of my objects' loop()s do nothing; but baking that knowledge into my root run loop lies on the way of madness =). This is what select() was _made_ for!

One other thing that would help flesh out the possible implementations is a yield() like for when you actually want a program loop _and_ coroutines. You can make your program loop a coroutine like:

async def run():
  while await yield():
    # you need to yield or await to make room for other coroutines to run.
    # if you don't await in this loop, you want to yield() each time around to let pending events resolve (or let other root loops get a turn on the processor or let your animation widget move forward a frame if it's currently registered to do it's thing because you pushed a button or[...])
    do_your_program_loop_things()

[...]
if __name__ == '__main__':
  run_this_coroutine(run())
  # register other root coroutines to run
  async_coroutine_feature.run_forever()

and you can have several of them to keep your features more cohesive (when it makes sense).

WarriorOfWire on 18 Jan 2020

I'd like to comment, and hopefully improve on, this example:

  async def toggle_pin( pin: digitalio.DigitalInOut, button: digitalio.DigitalInOut ) -> :
    while True:
      value = await button.on_change()  # like rise/fall on a Debounce
      if value:  # swap pin on fall.  Could have just used on_fall() instead
        pin.value = not pin.value
  run_this_coroutine(toggle_pin(led1, button))
  async_coroutine_feature.run_forever()

I don't understand what you mean by "swap pin on fail".

Well … asyncio is going to deprecate doing this sort of thing and Trio never supported it. Better:

  async def toggle_pin( pin: digitalio.DigitalInOut, button: digitalio.DigitalInOut ) -> :
    while True:
      value = await button.on_change()
      if value:
        pin.value = not pin.value
  async def main():
    await run_coroutine(toggle_pin(led1, button))
    await run_coroutine(toggle_pin(led2, button2))
    ## what should happen if the main program falls off its end?
  async_runner.run(main())

Presumably run_coroutine returns an object we can then use to cancel/kill the coroutine.

Another improvement I'd make is that toggle_pin(led1, button)) looks like a function call but isn't, as it does not actually execute the coroutine – you need to pass it to a coroutine runner for anything to happen. It's more consistent to explicitly pass the procedure and its arguments, if any. Further advantage: If the procedure is wrapped, the wrapper will run when the runner actually enters the task, not when you call run_coroutine. Thus,

  async def main():
    await run_coroutine(toggle_pin, led1, button)
    await run_coroutine(toggle_pin, led2, button2)
    ## what should happen if the main program falls off its end?
  async_runner.run(main)

Next, there's a neat "async for …" idiom we can use for polling the pin:

  async def toggle_pin( pin: digitalio.DigitalInOut, button: digitalio.DigitalInOut ) -> :
    async for value in button.on_change():
      if value:
        pin.value = not pin.value

though I wonder iwhether that loop shouldn't return a sequence of events instead, with common attributes ike timestamps or the event source, so that people can build more complex systems without adding all that by hand in mutiple places.

Further, fixing the "what should happen if the main program falls off its end?" problem requires some sort of task group. Trio calls them "nursery". The idea is that a task group ends when all the tasks that were started in it end (plus, if any task causes an exception, all the others are cancelled before the exception can propagate, thus you don't need to do your own task housekeeping):

  async def main():
    async with TaskGroup() as tg:
      tg.start(toggle_pin, led1, button)
      tg.start(toggle_pin, led2, button2)
    # The 'async with' waits for the taks to end

I do need to get my (rudimentary) port of Trio to Micro/CircuitPython updated … while it's not too old to serve as a basic starting point, I didn't yet find the time to add actual interrupt code etc..

smurfix on 18 Jan 2020

My workaround in CircuitPython is to create state machines. Objects are checked often to see if their states need to change or they need to call back.

My classes have functions called spin. (In my mind is an image of the magician adding spinning plates to his act while keeping the other plates spinning.) Each time spin is called it checks I/O and time.monotonic_ns() and changes state of the object accordingly and calls callbacks as needed. I use time.monotonic_ns() a lot. (I feel like Greg in _Over the Garden Wall_ throwing candy everywhere, only it's mononotic_ns.) I try to make spin functions get in and out fast if they have nothing to do.

At the main level, I have a function called spin that calls each of the spin methods (often hard coded in). My wait function (like time.sleep()) calls spin and is used only at that level.

At the top is also my primary loop and either it or sub loops call spin a lot. Sometimes that is all my code does, build everything and then go into a loop around spin, though sometimes I put in a sleep or a gc.collect(). If I have several major states that are quite different, the code in the big loop would move among states, perhaps using wait in transitions, and then have a spin loop in the state.

My callbacks are not allowed to call spin. I can probably come up with a better rule, but that is a simple one.

If I need to, I sometimes tweak spin to balance attention to spin in subsystems.

Dar-Scott on 18 Jan 2020

Use case...

Motion control. Like the LED use case, but with motion.

(Back in October I made a hat for my costume. I was in a hurry and took some shortcuts in motion control software and in a few places made some sudden changes, fast but not too bad. As I was going out the door something broke and I quickly changed a servo motor to something else available, bigger but it could fit. We got to the event and I put my hat on, a gesture that enables it. Fortunately, nobody got hurt.)

Dar-Scott on 18 Jan 2020

I'm using a hacked kindle to make a picture frame that has a phone number,
and it will display whatever you text to it... but only if it rhymes. A way
to bring more doggerel into my life. An embedded-ish context that's maybe
more like Raspberry Pi than CircuitPython, but it seems relevant to me?

Since folks have compiled vanilla Python 3.7 for it, I took it as a chance
to use Trio for the first time, and I LOVED it. I'm a full time programmer,
so I was getting all ready to use my usual stuff, state flags and callbacks
and event objects and stuff, to handle polling for new texts and reading
keypresses and having a few interactive modes. But despite handling several
things at the same time, the code turned out mostly as while loops, where
you just ask for input and then do something. It felt a lot more like
writing my first interactive console number-guessing games and stuff, back
in the day. It's just that there were two or three of those while loops
happening at the same time sometimes. It was great! No flags or callbacks
at all, basically the only "state variable" is the page number.

So I think Trio-style concurrency will be a much smaller leap for people
who are starting out: you can keep the shape of your while loop mostly the
same, you just have to adjust some of the lines. And then something else
can happen at the same time. Super cool.

tgs on 18 Jan 2020

👍1

I'd like to comment, and hopefully improve on, this example:

Hi @smurfix! I super disagree with a lot of your take on my proposal - but please don't mistake my below frankness for anything other than direct, explicit communication for clear understanding ❤️
tl;dr: I believe it is almost uniformly a set of regressions in flexibility, conformity and cognitive burden.

I don't understand what you mean by "swap pin on fail".

You misread. fall not fail. In Debounce it's called fell though, so I also misremembered. I typed this out without looking at the api to remember its name 😅

  async def toggle_pin( pin: digitalio.DigitalInOut, button: digitalio.DigitalInOut ) -> :
    while True:
      value = await button.on_change()
      if value:
        pin.value = not pin.value
  async def main():
    await run_coroutine(toggle_pin(led1, button))
    await run_coroutine(toggle_pin(led2, button2))
    ## what should happen if the main program falls off its end?
  async_runner.run(main())

I don't want to prescribe which event handler gets invoked first. You've changed the semantic of the application in your main. Also, toggle_pin is already an async method: An author should not need additional boilerplate to get its return value beyond await. run_coroutine should _not_ exist as in this example unless you maybe intend to have multiple event loops? I think that would be overkill though...
This is a very limiting form that has syntactic overlap with my proposal but I would hope that this is not the way it is implemented at a language level as it adds burden to the writer without enabling meaningful alternative patterns.

Presumably run_coroutine returns an object we can then use to cancel/kill the coroutine.

run_coroutine should instead be called async and return an object (we'll call it an awaitable) that you can await to get its value. We can also simply affix it to our method definitions as you've continued to do and just drop run_coroutine altogether.
Seems fine to instead let the code in the coroutine decide if it's done. Cancellation is not hard to achieve without forcing syntactic bloat at every call site; I would claim that external cancellation is a comparatively rare need. Ctrl+c is not conceptually problematic either: If you're running python, it's just however it currently does it but it also sets a flag to notify the underlying runner. If it's awaiting, you probably would want the implementation's framework to also be awaiting the interrupt signal and do the needful just the same...

If you're hung up on external cancellation, it seems fine to propose that an awaitable should be "cancellable". I do disagree but not that strongly.

Another improvement I'd make is that toggle_pin(led1, button)) looks like a function call but isn't, as it does not actually execute the coroutine

This is not fully correct and may be a reason for some of the misunderstanding: It _absolutely is_ a function call and defines exactly how to call the named async function - in python's implementation this invocation's return value is called a coroutine. An async function returns something that is awaitable (like a coroutine). The language way to get the value out of an awaitable is via the await keyword that signals a resume pointcut for the scheduler. If you're bothered by the form of the invocation, remember that a keyword like await sticking out after an = looks like invalid syntax until you are exposed to it too.

– you need to pass it to a coroutine runner for anything to happen. It's more consistent to explicitly pass the procedure and its arguments, if any. Further advantage: If the procedure is wrapped, the wrapper will run when the runner actually enters the task, not when you call run_coroutine. Thus,
  async def main():
    await run_coroutine(toggle_pin, led1, button)
    await run_coroutine(toggle_pin, led2, button2)
    ## what should happen if the main program falls off its end?
  async_runner.run(main)

Your run_coroutine is a little confusing to me - why isn't is just a decorator you'd apply to toggle_pin so you can just literally invoke the function you're trying to invoke by _invoking it_ directly by name? You want to wrap the coroutine for some reason - usually in Python you'd use a decorator for that.
And If it's a decorator that requires a special kind of calling convention, why not instead go the standard python route and name that decorator async?
And if it's named async and has special calling convention requirements why not promote it to the function definition itself?
Wrapping asynchronous code under needless layers of indirection is a bad thing. In your example you still have the burdens of async and await, you've just added an extra hoop a person has to jump through. _And_ it still misses the original point in the example of decoupled event handlers registered to the button's fall (sorry, should have been fell) event.

Next, there's a neat "async for …" idiom we can use for polling the pin:

  async def toggle_pin( pin: digitalio.DigitalInOut, button: digitalio.DigitalInOut ) -> :
    async for value in button.on_change():
      if value:
        pin.value = not pin.value

This pattern always feels perverse to me, bleeding implementation detail when an implementation of async/await leans on generators. More seriously, this reverses the direction of event flow at the event registration site (the async for line) and kind of requires the event listener to store an event queue. Gonna go gang-of-four for a sec here; my proposed model allows the Subject (on_change) to control how it notifies the Observers (toggle_pin). This lets you write simple subjects that yield 1 event as it happens _or_ queue them up for delivery irrespective of Observers. Maybe I'm a closet functional nerd or something but this is just backwards imo. Also, it breaks the brain to try and reason about "what does StopIteration mean to button.on_change!?"

Further, fixing the "what should happen if the main program falls off its end?" problem requires some sort of task group.

There is no problem and no necessary solution beyond the elegant definition of today: If the program falls off the end it is done. Changing that semantic is not something I'd support. If async_coroutine_feature.run_forever() returns, it has run forever or run everything that it will ever run.
If you have a set of tasks that will complete, when they complete they'll drop out of the coroutine runner's list. When that's empty "forever" is done as nothing will ever add something to it. (Remember how I said I don't want to write hardware events? This means nothing will ever add python code to be run when the event loop has no more user coroutines in some state)

  async def main():
    async with TaskGroup() as tg:
      tg.start(toggle_pin, led1, button)
      tg.start(toggle_pin, led2, button2)
    # The 'async with' waits for the taks to end

This is again fine modulo the start function. It should take an awaitable instead and therefore probably be called add or associate but naming things is hard. I'm not a huge fan of using the async __exit__ to drive coroutine execution, but if my proposal is held, you can add TaskGroup like this as a library and people can use it if they want to.

Most of what you propose (maybe all of it) is possible to implement on top of what I've proposed as a library, and that's where I'd propose to keep it. 👍

WarriorOfWire on 19 Jan 2020

@WarriorOfWire Yes it's a reduction in flexibility. The point is that in the 60s, removing "goto" in favor of "for" and "while" loops was a reduction in flexibility too, and people complained for much the same reason you complain now. Surprise, Python doesn't have a "goto" and nobody misses it.

The concurrency equivalent of "goto" is to have a task/coroutine that you can fire off and forget. I want Python to not have that either, and for much the same reason. Sure it's more flexible, but the cost of that flexibility is that you need to remember to kill it off when you no longer need it, including when your code throws an exception. People are habitually bad at doing that consistently and correctly. The point of Trio-style structured concurrency is that the runtime does that for you, which only works if you tell it what should happen, which requires structured code. Yes, there is no way around it, but why would you need one in the first place?

(Actually, I lied, there is – a task group is an object you can pass around …)

I also don't quite understand your objection to "async for". The literal translation of async for value in source is

while True:
   value = await source.__anext__()

thus I can't see any reversal of control flow or anything.

Another improvement I'd make is that toggle_pin(led1, button)) looks like a function call but isn't, as it does not actually execute the coroutine

This is not fully correct and may be a reason for some of the misunderstanding: It absolutely is a function call

Well, of course it's a function call, but it doesn't actually execute any of the named code (with the exception of decorators). It just creates a coroutine object that you then need to either pass to your loop runtime (run it in parallel) or await (run it serially and wait for it). Now what should happen if you don't do either of these? Any decorators run, along with their side effects, but the main code doesn't, which seldom is what I'd want.

NB: what should happen if the coro object is passed to your run_coroutine but the code inside then raises an exception? How would you handle that situation if the coroutine in question is supposed to run indefinitely and thus won't be awaited for?

smurfix on 19 Jan 2020

@WarriorOfWire I like the elegance of your quality_of_life_style(). I would like to discuss that and, to that end, simplified your program and added detail with some assumptions.

import warriorofwire_async
import digitalio
import warriorofwire_debouncer

button = warriorofwire_debouncer.Debouncer(digitalio.DigitalInOut(D1))
button.direction = digitalio.Direction.INPUT  # ?
led1 = digitalio.DigitalInOut(D2)
led2 = digitalio.DigitalInOut(D3)
led1.direction = digitalio.Direction.OUTPUT
led2.direction = digitalio.Direction.OUTPUT
led1.value = True
led2.value = False

# This is what we're going to do below:
def uncalled_equivalent_loop():
  while True:
    button.update()  # debouncer
    if button.fell:
      led1.value = not led1.value
      led2.value = not led2.value

# on_* functions expect functions
button.on_fall(lambda: led1.set_value(not led1.value))
button.on_fall(lambda: led2.set_value(not led2.value))

# Do this instead of `while True:` when you are building a warriorofwire_async project.
warriorofwire_async.run_forever()

I apologize for any undue distortion. I realize that this is a simple example and yet, given that, I note that this looks like callbacks. Either of us could readily create the module I renamed warriorofwire_async to handle callbacks. (I considered calling it darz_async to minimize any projection of something with your name on it that might offend, read it that way if need be.)

Though much of the rest assumes callbacks, this might well apply to coroutines in general. A callback version of the warriorofwire_async library can be a way to get ready for built-in callbacks based on primitive events.

This is a possible direction assuming built-in callbacks.

on_ All classes with a value attribute should move toward having an on_value_change() method even if it is timer polled underneath. Other on_ are allowed for any class. If there are no functions for an on_* then overhead is minimized. Names should include present tense verbs of the right aspect (eg. "receive" but not "smell"). Though connecting multiple functions to an on_* can be handy, it might be hard for me to use with my style of modularity; I'm fine without it. One advantage of only a single function is that it is easy to clear or reconfigure on the fly. In my discussion, I'll assume only one is allowed, though most of it should apply to multiple callbacks. One can create a wrapper to handle multiple, if it is needed.

built-in An example of a class that is built-in and has an on_value_change() method would be a revised DigitalInOut. Calling on_value_change create an internal mechanism that will cause the provided function to be called upon change. This would apply even when the direction is out. The function is called without any dynamic context. One might set it up like this:
my_pin.on_value_change(enable_doomsday_device)

timer An important new built-in class is the timer. A very simple one might be one that has a ns threshold attribute and the callback occurs when the equivalent of time.monotonic_ns() passes that threshold, that is, when time.monotonic_ns() > threshold becomes True, a change. The threshold can be atomically changed at any time. Fancy timers can be built from this.

In Python The revised Debounce would be written using timers and (when available) the on_change() method of is input. It should have a value attribute and thus an on_change() method, and can also have on_fall() and on_rise() methods. It is used just as one might use the digital io directly. All callbacks originate from the primitive callbacks.

Pluses and Minuses This method allows effective encapsulation. It does not require multiple stacks. Surrogates for the built-in callbacks can be created to try this out. It does require state-machine thinking in the Python code.

That doesn't always fit well. If, while doing all the other stuff, the code must make some Internet db queries to find the right admins and techs and then send an individual text or email to each one, it is easiest to describe that procedurally.

Dar-Scott on 19 Jan 2020

@WarriorOfWire Yes it's a reduction in flexibility. The point is that in the 60s, removing "goto" in favor of "for" and "while" loops was a reduction in flexibility too, and people complained for much the same reason you complain now. Surprise, Python doesn't have a "goto" and nobody misses it.

This false equivalence argument does not convince me. The challenges of concurrent software do not approximate the problem form of goto, and a global task group instance would be essentially equivalent to what is CPython's global (well, threadlocal) "task group" - i.e., trio changes nothing at the language level and adds burden to developers which may be unwelcome.

I also don't quite understand your objection to "async for". The literal translation of async for value in source is
while True:
   value = await source.__anext__()
thus I can't see any reversal of control flow or anything.

I want on_<event> methods exposed by digitalio and friends. That directly implies the Observer design pattern. Remember, _Subjects update Observers_. By making the observer next() the subject, each observer of the button mutates the state of the button subject (or, again, requires the button subject to implement its notification channel as a replayable queue per-observer). It's very incorrect to do this way both logically and practically:

There's no user benefit (trick code is not a benefit where a plain await button.on_press() obviously means "next time the button is pressed await is done.")
It forces an implementation to use more memory (notifications are necessarily coerced into a collection by the button framework at some point, no? Remember, some observers will consume events at a slower rate!)
If it isn't providing an event stream, it's a shocking piece of code to read: for in iterates _each thing_. If you have slow consumers and button does not implement a notification queue per observer you will not execute your loop for each press and you will have limited recourse. Instead, if an Observer wants to catch each event it must quickly memoize every event and handle it when it's time to handle it. E.g., button.on_press(lambda: self.add_one_press()) in the case of a listener that wants to move a cursor up a menu and not irritate the user by habitually missing distinct clicks due to layout latency. (displayio is another lib that'll need some async love)

Another improvement I'd make is that toggle_pin(led1, button)) looks like a function call but isn't, as it does not actually execute the coroutine

This is not fully correct and may be a reason for some of the misunderstanding: It absolutely is a function call

Well, of course it's a function call, but it doesn't actually execute any of the named code (with the exception of decorators). It just creates a coroutine object that you then need to either pass to your loop runtime (run it in parallel) or await (run it serially and wait for it). Now what should happen if you don't do either of these? Any decorators run, along with their side effects, but the main code doesn't, which seldom is what I'd want.

Trio has literally exactly the same requirement. trio.plz_invoke_this(function, the, args, the: kwargs) is an alternative calling convention for when you want to synchronously execute an async method. library.run_coroutine(function(the, args, the: kwargs)) is no different - it's just standard form that other Python programmers already understand. Trio doesn't fix the "you didn't await that boi" problem either. Garbage in, garbage out is never more true than when writing slipshod concurrent software (as a slipshod developer, I do know what I'm asking for).

NB:

K, I'll answer both:

what should happen if the coro object is passed to your run_coroutine but the code inside then raises an exception?

Absolutely does not matter beyond defining a consistent semantic. Python has Exceptions, and exceptions bust upward through stack frames. If you don't catch it, they will reach the coroutine runner. It's reasonable for the runner to do one or more of these policies and other reasonable policies can be conceived of:

Provide a handler callback. It receives the exception and maybe other stuff that would be useful to someone who might want to make their handler re-drive tasks or something. It would be reasonable for the default handler to do one of the other strategies.
End the runtime as though __main__ threw off the top (allowing displayio to print it and all that beautiful stuff CircuitPython does for us).
print() the exception info and drop the coroutine. Not very friendly; I'd rather end the program as above by default, and have a pluggable handler facility for when I just have to get crazy.

How would you handle that situation if the coroutine in question is supposed to run indefinitely and thus won't be awaited for?

It is handled by either of the above policies. If it's the last coroutine and you are using the unfriendly print() scheme, your program ends because you've waited "forever" and the coroutines are done. If you use the developer-friendly option (throw as from __main__ and halt), what happens when __main__ throws? It stops. If you use a custom exception handler, I'd hope that you could re-raise and get the developer-friendly version if your logic demanded it or equally print and continue or requeue a new coroutine or _whatever_ you wanna do.

I apologize for any undue distortion. I realize that this is a simple example and yet, given that, I note that this looks like callbacks.

@Dar-Scott Sounds like you have a handle on what I'm after. In fact I have implemented callbacks for the Adafruit rotary encoder (link if you're curious). It's _fine_ but only has the resolution of loop() on account of missing coroutines in CircuitPython. Event sources like this ought to be able to optimistically barge ahead of a main loop, and if you had coroutines you could make entire applications reactively sourced from events. Which, again, opens doors for automatic deep sleep modes (imho a real sleeper of a feature 🥁).

Callbacks are just better with coroutines, though you obvs don't _need_ them. I really just want to write event-oriented code for event-oriented applications and have my microcontroller take every advantage of applications structured that way!

WarriorOfWire on 19 Jan 2020

Hello again everyone. I'm happy to see this discussion continuing in a spirited manner with a lot of back and forth, ideas proposed and assessed, and folks expressing their needs and wants.

As far as I can tell everyone is doing a good job of communicating well and even handedly, however I will none the less take this opportunity to remind everyone of the code of conduct for CircuitPython and its libraries:

https://github.com/adafruit/circuitpython/blob/master/CODE_OF_CONDUCT.md

This is not meant to suggest that anyone has violated the CoC; were that the case I would say so. It is merely a reminder that it exists and that all parties should hold themselves accountable to it in order to help keep discussions such as these welcoming to all.

Thanks everyone! With any luck 2020 will be the year of (among other things) concurrency for CircuitPython!

siddacious on 19 Jan 2020

🎉3

@WarriorOfWire I'm glad we seem to be on the same page.

A callback is one task. A coroutine is effectively a sequence of tasks. Yes, a callback can be implemented as a trivial coroutine, but not all of the coroutine overhead is needed.

Of the methods for handling event programming, these come to my mind for this discussion:

looping calling update() for all modules requesting it
built-in primitive callbacks
async/await and functions built with those

My gut feel (not to be trusted) says that the built-in primitive callbacks are the sweet spot for optimum use of a microcontroller.

Dar-Scott on 20 Jan 2020

This false equivalence argument does not convince me.

Well, it does convince me. I have written a lot of asyncio code which uniformly became shorter, more correct, and easier to understand when I rewrote it with Trio's/anyio's semantics.

A single global corotine runner / task group does nothing to keep exceptions local because the call stack that led to its invocation gets lost. As an example. take a producer/consumer pattern: if one dies, you want the other to be cancelled too, otherwise you get a deadlock. Yes you can do this manually, but it's much easier to write your code in a way that makes all of this happen automatically. This includes restarting a failing part of your code cleanly, without affecting all the others and without writing a single extra line of exception-handling code. You simply cannot do that with a single global taskgroup.

it's a shocking piece of code to read

It's exactly equivalent to the while True: value = await thing.on_change() code promoted by you, simply adding a bit of syntactic sugar / a more structured way of handling a stream of events (take your pick) by removing one line of code. Any technical problem of the one is exposed equally by the other. There is no stack frame; this is an object with an __anext__ method, not a full async generator.

I strongly recommend reading https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/ and actually experimenting with some nontrivial Trio code vs. its native asyncio equivalent before making any opinionated statements about the uselessness of it all, let alone calling other people's code "shocking".

smurfix on 20 Jan 2020

@WarriorOfWire:

Of the methods for handling event programming, these come to my mind for this discussion:
looping calling update() for all modules requesting it
built-in primitive callbacks
async/await and functions built with those
My gut feel (not to be trusted) says that the built-in primitive callbacks are the sweet spot for optimum use of a microcontroller.

The main problem I have with the first two is that you need to decompose long-running subtasks into state machines. People (myself included) are, ingeneral, pretty good with linear code that has a couple of awaited calls sprinkled in (signalling points where the flow of control can be suspended/resumed, implying that places without an await cannot be interrupted thusly) and in general pretty bad with keeping track of callback chains and state machines. (The assumption is that that interrupting your code between any two statements in order to run some callback is not an option. People are equally bad at getting their locking correct; deadlocks in a microcontroller are worse than on the desktop.)

One can easily import "primitive callbacks" into an async/await system; just teach the callback to trigger an event you're awaiting. Thus, if the default callback triggers an event (and signals an overrun if the previous one hasn't been awaited yet) you get both – just override it if necessary.

smurfix on 20 Jan 2020

Thank you everyone for your continuing comments. Just FYI, async/await are available since we have MicroPython as our base code, but they are not currently turned on. So we could build something on top of those.

dhalbert on 20 Jan 2020

@smurfix (That was my comment.) I recognize that async/await allows for easier and better expression of some asynchronous tasks. However...

I keep having to address running out of memory. I worry that a full async/await would make things a lot worse. (I have not tried it in MicroPython, so that is just my gut feel.)

Yes, the await can be a marker reminding the programmer that things can change.

The use of async/await might improve overall performance but it will hurt latency.

A problem with async/await for me is that it is a shock to beginning programmers, both in syntax and in concepts. Getting things to run can be frustrating.

Dar-Scott on 20 Jan 2020

I have pondered a little on what might be a simple approach to doing several things at once, easy concepts for beginners. I just made this up, so it might be quite flawed.

run One way to do to several things at once (in my proposed scheme) is to "run" several functions, as one might run several programs. This builds upon common concepts and borrows the term "run". The functions share the processor and might even communicate amongst themselves. They coexist, cooperate and collaborate; they might be called co-functions or coroutines. In this scheme, they are co- simply because they were run at the same time.

side-effects and extended side-effects Though, in general, we favor functions with no side-effects in programming, such as sin(), we know that often that is what we want, functions with effect, such as write(). If there are more than one co-function, some functions might have other effects than the ones in its purpose. That is, there might be effects that take place during the call, such as variables changed or output pins changed, that are done by another co-function that has been run. They might take longer than expected.

Just as a programmer takes care in using functions with side-effects, one takes care when using functions with extended effects. (Such functions are called "suspendable" in Kotlin, I think.) One can expect that there are no changes in variables shared among co-functions between calls to functions with extended effects.

A function has extended effects if it is a built-in function with extended effects or it can possibly call a function with extended effects. (It is not defined as such and a calling of it in a special way does not make it so, just as a function with side-effects does not have any special defining method or calling method.)

program The program is itself a co-function. Built-in functions that might take a while are now functions with extended effects, that is, they might let other co-functions do some work. If there are any. The behavior for the calling co-function is the same. Some new built-in functions might be needed. (The program co-function might even be implemented the same as other co-functions, or built-in functions know the context, it is not visible to the user.)

only run With this scheme, the only command introduced to the beginning programmer is "run". No new syntax is introduced. The concept of when extended effects might occur must eventually be understood, too, but that is not needed immediately.

Advanced programmers can build any of their favorite async functions from these.

Dar-Scott on 21 Jan 2020

This is going to sound egotistical, but I really do urge everyone coming to this issue from an asyncio or callbacks-oriented background to learn a bit about Trio. You may or may not end up liking it – when has there ever been a programming concept that everyone liked :-) – but it's genuinely a paradigm-shift compared to other approaches, so you kind of have to spend some time with it to "get" how it fits together, and without that there's a lot of talking past each other.

To convince you that it's worth your time, I'll say that we do frequently get responses like @tgs's post up-thread, and the asyncio maintainers and Java core team have both said that they think the Trio-style "structured concurrency" approach is the way of the future.

Some good starting points would be this talk/live demo, the tutorial, or the Notes on structured concurrency for a more theoretical take.

njsmith on 21 Jan 2020

👍3

@njsmith I read your blog on Trio. You refer to the clear fact that "nurseries" have the same expressive power as "go statements." That is because Trio does not limit what goes inside of them. It neither prevents a user from stashing the result of a nursery expression's __aenter__(), e.g., on a package level variable nor does it eliminate the "go statement" (the apparent point of the library) even from the library's own feature set. The "nursery" construct is just one of many useful opt-in aids for writing clear, expressive software; callbacks and await statements are 2 other such expressive and useful aids.

The structure of "async with nurseryfactory() as nursery" that the blog post is set on is straightforward to implement in standard Python asyncio. Let's test that assertion with a 30 minute timeboxed starting point shall we... https://gist.github.com/WarriorOfWire/a5d15350c55cb3b2b61b74431e7cb484 You can draw that same picture around the School. I could have used gather() instead but 🤷‍♀ it seems like the issue at hand is more around the scoped lifecycle of the tasks and it's nearly bedtime.

I don't dispute the utility of such a structure. I've used this type of construct professionally for years. It's a good tool for rapidly giving easy-to-scan guarantees on batched parallel work like a web crawler, a directory scanner or a database ETL application. It's a square peg and there are many square holes. Quite a few are round though and Trio simply gets in the way of those.

Pretty sure by now I've made my stance clear while it has materialized over the past couple of days as it relates to the state of CircuitPython but let me summarize as I'll be unable to comment for several days going forward:

async/await are highly desirable in CircuitPython regardless of the hoops people are compelled to jump through. I prefer the standard hoops, not the standard hoops hidden behind a nascent idea's additional hoops.
Nobody is forcing anybody to write concurrent software; but novices who want to learn about big kid simultaneity can do so in a really nice environment with CircuitPython: No threads, no distractions, just pure cooperative multitasking. EZ mode 😄.
MicroPython's async/await facility is fine. _We should use that implementation_ and avoid needless divergence from both industry and community.

Here's a good video from pycon Australia to get you ramped up on coroutines on Python microcontrollers: https://www.youtube.com/watch?v=tIgu7q38bUw

Good luck CircuitPython maintainers! I can't wait to write tidy little coroutines on all of my Feathers!

WarriorOfWire on 21 Jan 2020

It neither prevents a user from stashing the result of a nursery expression's __aenter__(), e.g., on a package level variable nor does it eliminate the "go statement" (the apparent point of the library) even from the library's own feature set.

You're missing the point. The point isn't that you may or may not save nurseries to some variable and pass them along to some other code; that doesn't violate any invariants. The point is that the nursery's __aexit__() will block new tasks from being created in the nursery, wait for that nursery's existing tasks to finish, allows you to stop them all with one function call, and auto-cancels all other tasks when one of them raises an exception (and propagate that exception), without you having to write a single additional line of code.

Yes you can implement a toy version of that on top of asyncio in ten minutes, others have done so with a lot more effort (cf. anyio). So? That's not the point. You can implement for and while loops with goto easily, too, but if the programmer is still allowed to freely use "goto" that doesn't buy you very much.

Frankly, I haven't found any of your round holes yet. After all, there's no functional difference between registering a callback and starting a subtask that loops on await object.next_change(), and if you need a long-running non-scoped background task then nobody prevents you from creating a global taskgroup and starting your task there.

However, there is a conceptual difference in that any non-trivial program requires you to remember to un-register the callback or stop that non-scoped task yourself at some point, any exception the callback or global task raises must reach your program's top level and/or requires additional code to notify your "main" code, and its stack trace will not tell you how it got there (i.e. where it was registered / started). All three of those are not desireable from my POV, let alone from that of a beginner.

While I know that fire-and-forget is "easier" when all you want to write is a Blinkenlight equivalent, guess what happens when you later connect the Blinkenlight to MQTT and the network connection breaks? I'd rather spend ten minutes more up front to explain how taskgroups work ("This is how you do it." "That's complicated." "A bit, but it's just one extra line (the async with that creates the taskgroup) and it works the same whenever you need it, so it's actually quite simple." "OK."), than, somewhat later, waste half a day teaching people how to correctly clean up after themselves – they've gotten used to doing everything the asyncio.create_task way, and their code reflects that. Been there, done that. "Why does this crash?" "You didn't tell it to stop." "But why doesn't it just know that I don't need that any more?" "Because [computers are stupid, but instead I say] you didn't tell it to." "Computers are stupid." "…"

Registering a callback or starting a task with asyncio.create_task may seem more natural for you, but that's because that's what you're used to. That's understandable, but it doesn't help beginners (or in fact experts) to write better code. Been there, done that …

Asyncio isn't "standard". It's a library that forces you to jump through quite a few hoops when you want to write correct programs. Trio isn't "standard" either, and it doesn't have additional hoops, just different ones – and when you're done your program actually requires fewer hoop jumps than with anything else. (I'm not just saying that – I rewrote a bunch of libraries with Trio. Guess what happened to the line count.)

One of the staple examples of Trio is the Happy Eyeballs algorithm, which requires 40 lines when you think in Trio's terms, but 400 when you stay with the asyncio mindset. That alone should tell us something about which way is "better".

smurfix on 21 Jan 2020

😕1

I have to admit that I am impressed with the explicit cancellation scopes, but I wonder how hard it would be to actually implement this in CircuitPython. We don't have all the usual exception handling machinery that grown-up Python has, you can't even inspect an exception object inside Python code. The talk mentions that before 3.7 Trio had to do some magic with ctypes to rearrange the stack for the cancellation to work properly — obviously we can't do that in CircuitPython, but of course something could be added in C to handle it. What makes me nervous is the thought about how big it would be and how much extra code we would need to run for this. You know, CircuitPython doesn't even have await and async enabled yet, and in MicroPython await is just an alias for yield from. There is no async while, async with or async for or any of that. Would we need to add all this, or could it work with just plain versions of those?

deshipu on 21 Jan 2020

What do you mean, no async with? The code below worked since February 2019. (async with/for by itself was added in January 2016.)

… and no, there is no async while. CPython doesn't have that either.

$ cat test-trio.py
#!/usr/bin/env micropython

import trio
async def bar(s,n=None):
    try: await trio.sleep(s)
    except: return
    print("Slept",s)
    if n: n.cancel_scope.cancel()
async def foo():
    async with trio.open_nursery() as n:
        n.start_soon(bar,0.1)
        n.start_soon(bar,1.2,n)
        n.start_soon(bar,2.3)
trio.run(foo)
$ upython test-trio.py
Slept 0.1
Slept 1.2
$

Works. Micropython master branch, thus I can only assume that it works with CircuitPython too, once you merge up to the current µPy -- version 5.0 is in beta, so that shouldn't take too long.
Archives: https://github.com/smurfix/trio/tree/micro, https://github.com/smurfix/micropython-lib/tree/trio.

Cancellation isn't correct right now (the nursery doesn't catch the Cancelled exception; cancelling by itself does work, as the above code demonstrates; if you remove the except:-return handler you get a traceback). But that looks like a minor problem. I need to port the code to the current Trio master anyway, this experiment is half a year old, and check what the problem is.

smurfix on 21 Jan 2020

NB, I am not deeply enough into how micro/circuitpython is built to assess any code size questions. There's probably a fair bit of trimming that can be removed from Trio, and if we decide to do this then at some point we'll have to think about feature (im)parity and related matters.

Call stack hacking is/was necessary for Trio IIRC in order to collect the frame information so that a MultiError (the exception that's raised when more than one subtask of a nursery raises an exception at the same time) carries a reasonable stack dump. Micropython doesn't do things that way, so in the first version I just dropped this part.

smurfix on 21 Jan 2020

I wasn't aware that you are already working on it. And looks like I wasn't up to date with the async status in MicroPython — I stopped following closely before it was implemented, and missed it, thanks for the correction.

deshipu on 22 Jan 2020

No problem. I wasn't "working on it", strictly speaking; my branch is more like a proof-of-concept hack to discover whether µPy might be up to the task. Much would need to be done, preferably by people who actually know their way around Micro/CircuitPython (I don't – not yet anyway), to transform it into a useable ecosystem.

smurfix on 22 Jan 2020

I think at the moment the most important question is what kind of functions need to be exposed to the Python side of things so that async libraries could be implemented in the first place. Right now I'm mostly thinking about a select-like call, that could wait on multiple events at once (files, uart/spi/i2c, gpio changes, timers, etc.). Without this, we are reduced to a busy loop with polling, which can be sufficient for a proof-of-concept for the API, but not really useful.

deshipu on 22 Jan 2020

A Unix-style "select" actually isn't a good match for an async main loop. You don't want, or need, to shoehorn every feature that might conceivably wake up a task into a common select call. It mostly-works for Unix because almost-everything is a file descriptor there, but no sentence that starts with "everything is a …" makes sense on µPy.

Here's how I would handle things:

Whenever something happens, presumably that something triggers an interrupt.
The interrupt handler just hooks an object with a callback into a global list if it's not there already (this needs two atomic writes and requires no memory allocation) and disables the interrupt if that happens to be level-triggered. Replace this handler if you need ultra-fast reaction time but don't need memory allocation.
The callback by default just sets the trio.Event object associated with the interrupt so that the task handling it will wake up. Replace the callback if you need fast reaction time. There's no async context here and your code really shouldn't raise an excepition, but you can wake up tasks.
The Trio mainloop goes through this callback list, runs them all, and then continues with the next runnable task(s). If there are neither active callbacks nor tasks, it atomically checks the callback list again and goes to sleep if that's empty, to be woken up by the CPU's next interrupt. Repeat.

So that's the bottom-up layer. The top-down view is pretty straightforward:

User code calls e.g. "async with port.interrupt(RISING) as handler", which creates the handler object, hooks it into µPy (raise an error if the interrupt is already in use), enables the interrupt, and/or checks whether the condition is already met (set the event if it is, enable the interrupt if not).
User code calls handler.__anext__(). This method calls self.event.wait(), clears it (currently: by replacing it), clears and re-enables the interrupt, and returns whatever result it should generate.
Leaving the "async with" block disables the interrupt and removes the handler from the system.
It's fairly trivial to wrap this method with code that does things like de-bouncing port interrupts, process I²C transactions, and whatnot. Trio has demonstrated that it's very composeable.

Disclaimer: yes I know that this is a bag of words and no code, much less code that proves the concept. We might want to discuss this further on Gitter or Discourse or somewhere else that's more suitable than a CircuitPython issue, before somebody actually starts coding.

smurfix on 22 Jan 2020

@smurfix I think I'm missing something. How does Trio create a continuation?

Dar-Scott on 22 Jan 2020

use case...

I have a class that handles UART communication with an audio Bluetooth module. It has two levels of flow control, RTS/CTS and "OK". Messages belong to a particular channel and those have to be split apart on receiving and either queued or callback'd. Currently, a polling loop calls update() often. (Well, actually, it is named spin(), but I noticed that Debounce has update().) The function update() is expected to return quickly if it has nothing to do. I have goofed and let code callback to a function that indirectly calls update() and I have had to create some rules and a uniform way of doing things.

Dar-Scott on 22 Jan 2020

@Dar-Scott It doesn't. CPython uses fancy syntax around generator functions and calls them coroutines. µPy does the same thing. A coroutine is not a continuation as that word is usually used, because it's not a copyable object.

To explain: Python only allows you to continue a coroutine (you call its send method) and you get a value and some state back (at the next point the coro uses yield (not yield from) which is where you continue with the next send). This is not a symmetric relationship; the coroutine always returns to the caller of that send, and you can't copy a coroutine state to, for instance, retry with sending B if it crashed when you continued it with sending A.

In contrast, "real" continuations are symmetric. Calling into a continuation typically involves creation of a new continuation which the code you just called can use to get back to you. The magic that does this is named "call-cc" in Lisp ("call-with-current-continuation"). Using that primitive makes for truly mind-bending[ly simple/crazy/???] code, plus you can (indeed must) have code that looks like procedure calls but which never ever returns, but Python can't do that, thus we tend not to call it "continuation".

All Python coroutines are created by simply calling an async function without "await". You can then use coro.send(x) to send some X into the coroutine which essentially becomes the return value of the yield which the coroutine last suspended itself with.

Asyncio allows you to create coroutines yourself, and it's OK with sending anything at all to and from them. With Trio you don't do that; you create a task by calling nursery.start(async_fn) so that the nursery can call async_fn() itself in order to manage the new task for you. Also, the messages it sends are strictly Trio-internal types so that there's no ambiguity about the task's state (the task runner must know exactly what the task is waiting for, and the nursery must wait for all its tasks to end before leaving its context).

smurfix on 22 Jan 2020

I recognize we all have different notions of what is simple. I like the idea that CircuitPython implements something simple and the big boys can create from that tools they like. These are my notions of simple.

Moving functions around as data is sometimes hard to grok, but it seems to be introduced early and is an important concept. I think we can assume that concept, at least to the extent of modifying examples. (Coroutines, as the term is commonly used, can be confusing, because they are not really functions, but wrappers around functions.)

I have seen programmers get confused as to what is synchronous and what is asynchronous. In discussion, I find some interesting and understandable sources of confusion. Also, the words tend to focus on the wrong scope. This is the only time I use those words.

Perhaps adding this functionality should be done in a way consistent with the nature of teamwork for CircuitPython implementation. Built-in I/O objects should be able to be modified in any order and CircuitPython works along the way. New built-in I/O objects should be easy to design to be consistent with this functionality.

I see a couple simple ways to implement this.

callbacks This is the simplest to implement, but it requires adding methods to I/O functions (built-in and in Python) to connect callbacks. These can use on_* names. These are called with only static contexts, but exceptions work as usual. Just as in other uses the parameter is either a function name (variable) or a lambda. Scheduling is undefined. Or not?

An important question is "when do these occur?"

Between every simple statement is a possibility. This reduces latency. However, there is no guaranteed maximum latency. It requires use of single statement idioms for cooperation, but those can have wrappers. (There might be some clever way these can occur between iterations.)

Another possibility is for them to occur only when a certain function is called. This is similar to, but faster than, calling a function that calls update(). The value and available attributes do not have to be polled.

co-functions This is an alternative expressive approach. I just made up the name "co-function" (which might conflict with some usage) to make a distinction against coroutines. Here, co-functions are simply functions that are used in a certain way. No special labeling or wrapping is needed or desired. All of the concepts that one learns about functions in variables and about lambdas apply.

A function called (say) run() will start a co-function running. The parameter is the function. The function is passed just as any lambda or function might be. Any function at any time can call run(). There are no rules about what can do what when. (There might be some define shoulds.)

Borrowing a term from Kotlin, some functions are suspendable. (Spelled with an "a".) This allows execution to be shared, that is, execution might stop some other co-function gets to use execution. Some variables might change. The length of time in a suspendable might be longer than that expected for the simple functionality of the function.

Built-in classes with functions that currently block can make them suspendable; an example is reading from a UART. Some I/O built-in classes can have these functions added as prep before the transaction; an example is digital io. Maybe time.sleep() can be made suspendable or a new built-in class that waits until (say) monotonic crosses a value, can be added in the prep stage. Any function that can potentially call a suspendable function is suspendable; a naming convention might be handy (or burdensome and unreliable). If no co-function can be run the system spins until one can be run.

This approach means that code between suspendables will not see any surprise variable changes. That is, it is a "critical section". Also, software-based timing should work as usual. (An alternative that improves latency would be to drop the notion of suspendables and allow switching between simple statements, requiring use of single statement idioms for cooperation.)

What about calling a suspendable from the top level? It works as expected. That is, the top level is virtually a co-function. (And might be implemented as such.)

There are no surprises in context. Exceptions that go up out of a co-function end up at the top.

This does require maintaining some sort of continuation, a stack or something. This might be tricky and take up RAM.

Implementation might be in stages. The first is the prep where blocking functions are added to built-in I/O classes. The second is the addition of run and a single class with a suspendable function (say, a timer). The third and fourth are the conversion of all blocking functions to suspendables.

which I am OK with both of these. I am growing fonder of the later but recognize that it might have a big step in switching.

Dar-Scott on 22 Jan 2020

We might want to discuss this further on Gitter or Discourse or somewhere else that's more suitable than a CircuitPython issue, before somebody actually starts coding.

We are on CircuitPython's Discord all the time, and there are weekly meetings there for discussing more fleshed out ideas in a bigger group as well.

deshipu on 22 Jan 2020

@ smurfix My ignorance is showing here. You seem to have stepped around what I was trying to ask and I suspect that is because it is obvious. How do you save where you are in Python? I can picture an ad hoc method that saves lambdas representing the rest of the work, but that means translating the code in a special way that does that, and that becomes interesting in compound statements. I think I am stuck on some 20th century concept.

Dar-Scott on 22 Jan 2020

@Dar-Scott This was done in Python initially only for the so-called "iterator generators", functions that instead of (or in addition to) return use yield. "Calling" such a function produces an iterator object, which executes until the first yield, and then you can call next() on that object to get it to execute to the next yield and so on. Then a yield from statement was added, which basically iterates over another iterator, and yields each value it gets from it.

This turned out to be powerful enough to implement a kind of co-routines. The async keyword was added to make a function a special kind of iterator generator, and the yield from got renamed to await. If you are curious about a very simple example of how this works, you can look at my "meanwhile" library: https://github.com/deshipu/meanwhile

deshipu on 22 Jan 2020

deshipu Thanks!

Dar-Scott on 22 Jan 2020

@Dar-Scott You don't. It's implicit when you create and use a generator. The "yield" which you use when executing the generator's code saves the call stack and returns a value to the caller, which then calls send to get your generator to continue where it suspended itself.

There is no other way to create that call stack. The only way to "make something suspendable" is to convert the whole thing to async functions by liberally sprinkling "async" and "await" keywords onto your code. The only way to actually suspend something is to call "yield" in there, or rather await runtime.some_function() which does it for you, and which manages the generator/coroutine side of the whole housekeeping.

You then need Trio or asyncio or @deshipu 's meanwhile to manage the other side, i.e. the list of generators/oroutines which are runnable. This typically involves passing some magic object through that tells the runtime whether / when to resume a coroutine.

The (only, in Python) other way to make multiple things happen is to take some function or method, and telling CircuitPython "when this [interrupt] happens, call [that]". This way works for small programs ("when you press the button you turn on a light") but as things become more complex you want a normal call stack: the command interpreter reads lines, the line reader/editor reads characters, the character input needs to wait for the UART's next-character input – so, rather than forcing you to invert all that logic manually, which is a major source of hard-to-track bugs, I'd like Circuit/MicroPython to use Trio's abstractions, because they work very well IMHO, and hide all that complexity in a way that affords building complex projects without shooting yourself in the foot.

smurfix on 22 Jan 2020

👍1

@smurfix Reading your approach with interrupts I can't help but note that this is exactly how the select function needs to be implemented internally (keeping a list of references to objects that have something to show, and then returning it as soon as it's non-empty or there is a timeout), except it wouldn't use any library-specific objects internally, so that multiple different libraries could be implemented using it on the Python side. Sure, those references would need to be more than just file handles, as in Unix. I would really hate to lock users into just one "correct" way of doing things.

deshipu on 22 Jan 2020

@deshipu The generator function is what I was missing.

Your meanwhile is clean and simple. As is, it is reduced to polling, but allows a certain expressiveness is describing processes. It can be expanded to add priorities, or deadlines. Some sort of interrupt or callback might speed that up. The use of select() might work but it requires some notion of selectables.

I guess I was trying to add to CircuitPython rather than using Python (which includes user definable generator functions) in yielding.

Though I suppose there can be some magic in the compiler that translates a generator function to a bunch of lambdas, I suspect there has to be a continuation (of a sort) saved. That is, the yield has to get access to the Python stack frame and stuff it somewhere. So, implementing a generator is really hardly more complicated than adding primitive suspendables. The same core mechanism is required. Flipping that around, if the code is there to optionally implement generators, then it would be straightforward to implement primitive suspendables.

Dar-Scott on 22 Jan 2020

Please do not underestimate the level of experience that a "beginner" has when they start out with Circuitpython. Some of us have been programming with Python for several years, but have not yet ventured into some of the more advanced features. I would put myself in that category, having written quite a lot of robot control software in Python. I think I would be somewhere in the intermediate level, where I am just now starting to need some of the more advanced features Python offers. Some of us are no doubt true beginners, not having ever programmed before in any language. Also, clearly, there are those who are way more advanced than I am and who have used many of Python's more advanced features. We are all over the spectrum as far as our experience with programming, and may or may not have used Python.

I am currently working on software that will operate a small autonomous robot. It turns out that I am actually creating a sort of tool kit that can be used to construct code for many different robots. This is very much in the early stages and I am just now starting to look at breaking code into functions that can easily be used elsewhere. This current code can probably be called a type of polling, where a timer is checked and if it has reached a limit or overflowed, the task is run and then the timer is reset for the next interval. I do this all in Circuitpython.

For instance, I can already fire off tasks at a specified time interval that will interrupt the flow of my main loop to execute and then return control back to the main loop. I initialize timers for several tasks before the main loop starts and check for end count or overflow within my main loop for each task. So far, I have defined three different tasks that run at different intervals.

While this works well for what I am doing right now, I definitely see where I could make good use of some form of concurrency. Robots can easily have many different things that need to be going on at the same time. Distance sensors need to be checked to make sure the robot does not run into obstacles (moving or inanimate), wheel encoders need to be checked to sense whether a wheel has stalled and to check how far the robot has traveled, tilt sensors need to be checked to see if a robot is tilted more in a direction than what it can deal with, etc. You get the idea.

Right now, I am not sure how I am going to accomplish all of this with Circuitpython, but I am sure going to give it a good solid go. This may not belong in this thread, but I thought it would be good to have a context within which concurrency functions might be used. Perhaps this context might lead to a solid method of implementing concurrency within Circuitpython.

geekguy-wy on 19 Apr 2020

Yup, all of that is quite possible in vanilla CircuitPython, just a takes bunch of onerous state to maintain and careful registration of your sensors in loop() (of course there are near-infinite ways to do this).

If you had asyncio from the usual Python library or something like existing clever MicroPython implementations you could organize each sensor as its own mini almost-realtime program and contribute observations to a central robot state or events to subscribers (or both).

The need for asynchronous peripheral APIs is easily demonstrated. If you need to send an I2C message then wait 200 millis to receive the response you can't block loop() for that time in some projects. Existing peripheral API developers then need to choose between an ad-hoc polling API or just let their devices be unsuitable for realtime use cases. If we had a standard CircuitPython async/await approach, there is 1 async user experience to target, and it's the "right" one that users/developers expect. Peripheral developers then can make synchronous and asynchronous apis with confidence, as it makes sense for their devices while users benefit from uniformity and expanded applicability of devices in interactive projects (e.g., robots and screens) where milliseconds matter.

There are existing successful implementations to pattern off of both in Python and MicroPython. I'm still eagerly hoping 2020 is the year of async for CircuitPython!

WarriorOfWire on 21 Apr 2020

Yes, for robots, blocking would be very bad, even for a very short interval. It could cause important event(s) to be missed. I would not want my robot to collide with an obstacle because it could not catch the distance sensor event. I really hope the team can come up with a good and sensible implementation of concurrency for Circuitpython.

geekguy-wy on 26 Apr 2020

One use case I haven’t seen expressed is that of having real-time counter displays being asynchronously driven while other things are going on, and being able to stop/start/reset those counters. I was going to do a project that needed those about 6 months ago, and gave up on CircuitPython.

TonyLHansen on 23 May 2020

Personally, I would be happy with being compatible with how micro python does threads, and then add higher-level constructs afterwards. That way those who need asynchronous threads can do it, even if they are hairy monstrosities and not elegant beauties. Some of the proposals look really nice, but are useless if they are not available.

TonyLHansen on 24 May 2020

@TonyLHansen you can already use the "meanwhile" library (https://github.com/deshipu/meanwhile) — it implements a simple async reactor mostly compatible with how the big Python async functions work. The only downside of it is that in the absence of internal mechanisms it works by polling, but that shouldn't be a problem for things like counters.

Threads are rather hard to implement on small microcontrollers with very limited memory, and they are very counter-intuitive to program (it's very easy to write a program that has race conditions).

deshipu on 24 May 2020

What is the current status of this?

After skimming over the long discussion here, it seems like there was a sample implementation at #1415, but it won't be merged, and there's been a lengthy discussion comparing different approaches.

@bmeisels and I are working on an app framework for the AramCon Badge 2020, and some kind of async I/O will be super useful for us. Right now we're looking at the implementation from @deshipu, but we'd love to know what direction CircuitPython is going...

urish on 25 May 2020

@deshipu, thank you for the response.

@TonyLHansen you can already use the "meanwhile" library (https://github.com/deshipu/meanwhile) — it implements a simple async reactor mostly compatible with how the big Python async functions work. The only downside of it is that in the absence of internal mechanisms it works by polling, but that shouldn't be a problem for things like counters.

While "meanwhile" looks like it can handle simple counters, I don't see any way to control the threads once they've started. The primitives aren't there. From the sample program, it also looks like one timer needs to know details about the other timer? Or is that a mistake in the sample program?

Threads are rather hard to implement on small microcontrollers with very limited memory, and they are very counter-intuitive to program (it's very easy to write a program that has race conditions).

I totally agree that they're hard to implement. But I'm convinced that you can create race conditions with ANY multi-tasking/threading setup.

As for being counter-intuitive to program, that's true with ANY paradigm shift. (If you're used to apples, then passion fruit can be weird.) That's no reason to prevent their use.

"Striving for excellence motivates... striving for perfection is demoralizing" -Harriet B. Braiker
"Perfect is the enemy of good" -Voltaire

TonyLHansen on 25 May 2020

@TonyLHansen

There are no threads. This is cooperative multiprocessing — the tasks suspend their execution and let other tasks run explicitly, by yielding the control back to the main loop. The tasks don't have to know about each other's details, I'm not sure what you mean here.

I'm also not sure what kind of primitives you require. Maybe you could give me a simple example of the kind of a program you wanted to write with those counters, and I can show you how this can be done with that library?

But I'm convinced that you can create race conditions with ANY multi-tasking/threading setup.

No, there are setups that force your programs to be correct. I mean, obviously you can always create race conditions communicating with external systems, but that's unrelated to parallelization of your program — a completely single-threaded code can do that too.

deshipu on 26 May 2020

What is the current status of this?

After skimming over the long discussion here, it seems like there was a sample implementation at #1415, but it won't be merged, and there's been a lengthy discussion comparing different approaches.

@bmeisels and I are working on an app framework for the AramCon Badge 2020, and some kind of async I/O will be super useful for us. Right now we're looking at the implementation from @deshipu, but we'd love to know what direction CircuitPython is going...

We don't have any immediate plans to add async. @dhalbert is currently working on _bleio on Raspberry Pi with the Bleak library which uses Python asyncio and may inform our long term direction.

tannewt on 26 May 2020

Interesting related discussion here: https://forum.micropython.org/viewtopic.php?f=2&t=8429

tannewt on 27 May 2020

Thanks Scott!

urish on 27 May 2020

I'm also not sure what kind of primitives you require. Maybe you could give me a simple example of the kind of a program you wanted to write with those counters, and I can show you how this can be done with that library?

I've been thinking quite a bit on this topic about the primitives that I DO want.

When I write threads in normal python (NP) using the threading library, I essentially have two choices as how to create a thread:

1) create a threading.Thread() on an existing function, then invoke start() on that object
2) create an object derived from Thread and invoke its run() method

1 is fine when you have simple functions that can operate independently and might use a couple global variables. What you've provided sort of feels like #1.

However, if you want your thread to manipulate a lot of state variables, you really want them encapsulated into a class. And that's where #2 comes into play.

Then there's thread management. How do you get an individual thread to change what it's doing? Or pass data to a thread? How does a thread end? How does it get removed from the active thread list?

Using global variables (with #1) is fine when you want ALL of the threads using a particular function to change in the same way. However, finer grained management requires #2.

Then there's cooperative passing of data between threads: consumers and producers. Programming those requires some sort of semaphore or locking mechanism.

How do you encapsulate external events?

Those are the types of primitives that I would like to be able to work with.

So to summarize what I would want in a co-operative multi-processing library:

a) easy way to say "run this function"
b) easy way to say "run this method with this object"
c) easy way for those methods to give up control, either after performing some work and to say it needs to wait for
c) a way for the cooperating segments of code to say "I'm done", to say "wait for that cooperating segment to say its done", and for them to be reaped
d) some primitive that acts like a semaphore; higher-level constructs could then be built on top of that
e) some way to wrap external events

But I'm convinced that you can create race conditions with ANY multi-tasking/threading setup.

No, there are setups that force your programs to be correct. I mean, obviously you can always create race conditions communicating with external systems, but that's unrelated to parallelization of your program — a completely single-threaded code can do that too.

We'll have to agree to disagree. :)

TonyLHansen on 3 Jun 2020

a) easy way to say "run this function"

await your_function()

b) easy way to say "run this method with this object"

await your_object.your_method()

c) easy way for those methods to give up control, either after performing some work and to say it needs to wait for

await

c) a way for the cooperating segments of code to say "I'm done", to say "wait for that cooperating segment to say its done", and for them to be reaped

return

d) some primitive that acts like a semaphore; higher-level constructs could then be built on top of that

Just use any variable or attribute. You don't need special "safe" data structures, since control switches only happen in designated places, so all data structures are safe.

e) some way to wrap external events

Can you elaborate on that?

deshipu on 3 Jun 2020

Is there any progress on this behind the scenes? I would also really like to see this happen in CPY. I think especially with the ESP32-S2 coming along you will absolutely need this for networking.
In my experience working with a lot of art students at my day job over the years, events are the easiest concept to understand as it is just a function thats being "called for you" but the asyncio way seems also fine to me albeit it needing a bit more things to understand and take care of but especially if the overall Python community is adopting this style it is probably a good idea to stick with it in my opinion because many will come from Python and may have already learned it.

timonsku on 22 Aug 2020

@PTS93 @WarriorOfWire has discussed this with us extensively on discord, and created a simple library: https://github.com/WarriorOfWire/tasko. I intend to look at this in more detail fairly soon. I have been working on another BLE implementation and have not had time to work on this for the past few months.

dhalbert on 22 Aug 2020

👍1

That looks good! Though I assume it will support more than just intervals?
Hardware and generalized software interrupts are definitely something I'm looking forward to the most.
So trigger on input change or trigger on message received (e.g async network, async busio).

timonsku on 23 Aug 2020

I don't represent Adafruit. I'm just a big fan. But it is my understanding
that CircuitPython will never be targeted to the ESP-32 because it doesn't
support host mountable USB disk. It was originally targeted but was dropped
when USB disk became the standard development tool. Adafruit considers that
to be a technology requirement for CircuitPython.

Adafruit has often pointed out that MicroPython is supported on ESP-32 and
various multiprocessing techniques have been implemented there.
CircuitPython is after all a fork of MicroPython so that is a Python
solution. What it lacks is the complete and tested support for all the
hundreads of Adafruit sensors and add-ons

On Sat, Aug 22, 2020 at 2:22 PM Timon notifications@github.com wrote:

>
>

Is there any progress on this behind the scenes? I would also really like
to see this happen in CPY. I think especially with the ESP32-S2 coming
along you will absolutely need this for networking.

In my experience working with a lot of art students at my day job over the
years, events are the easiest concept to understand as it is just a
function thats being "called for you" but the asyncio way seems also fine
to me albeit it needing a bit more things to understand and take care of
but especially if the overall Python community is adopting this style it is
probably a good idea to stick with it in my opinion because many will come
from Python and may have already learned it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/adafruit/circuitpython/issues/1380#issuecomment-678674818,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJKESRXM6RRO7LIHV2XCG3SCAEHDANCNFSM4GIW3J7A
.

zencuke on 23 Aug 2020

@zencuke we are working on an ESP32S2 port. That does support USB.

dhalbert on 23 Aug 2020

Good news. Thanks. I didn't know about the ESP-32 USB disk.

zencuke on 23 Aug 2020

Hi, I am sorry but do not want to read all comments on this issue. I study and work on real-time systems. Circuit python is for microcontrollers. They are not PCs - real PC-multitasking on a single CPU is not needed at all.

Everything that CircuitPython programmer needs is scheduler :) If three tasks (A, B, C) with two sections (example: AA) runs this way:
ABACBC or BBAACC is the same. If you need cooperation between tasks, you can use shared variables, queues, etc. and split tasks into multiple interlaced tasks.

In real-time applications, what is needed are priorities, a way to set order or schedule, and a tool that tells you when tasks run.
1) The essential is scheduling. Users can plan everything, write it down into a timetable, and then implement it as the schedule for the scheduling mechanism. The user only needs to know when the task starts, its deadline, and if it is continuous and period of task repetition. There are required only two things: a way to create a schedule and inform the user if a task misses its deadline.
2) Priorities are harder to implement - you need a mechanism to stop a running task and run a task with the highest priority if required. Much more important is way to temporarily elevate the priority of low priority task if it access to resources needed by higher priority task to prevent deadlocks.

If someone implements scheduling framework into CircuitPython, someone else can implement its scheduler, and an inexperienced user can use it. There are multiple scheduling strategies. Every single one is good for a different problem, and no single strategy is right for everything.

Multitasking implemented with the scheduler is the way to go. It is predictive and straightforward. If every task informs CircuitPython about its start and end, tasks can easily be visualized for an inexperienced user to debug deadlines, schedules, or priorities easily.

I would be pleased if I found an EDF scheduler in CircuitPython or at least a way to set up static scheduling. I don't think it's going to be possible to implement advanced scheduler only in Python, and it's probably going to take help from inside of CircuitPython implementation.

eLEcTRiCZiTy on 7 Sep 2020

@eLEcTRiCZiTy that is exactly what async/await does.

deshipu on 7 Sep 2020

@deshipu Async&await is for PCs or Mobile phones when you do not need information on how asynchronous functions are executed in the background and only required is the result and or smooth GUI response. A small delay or more significant overhead is easily overlooked and usually does not bother anyone. ...and!.. usually good async scheduler needs another thread or some level of cooperative multitasking. When the async&await mechanism is implemented using some sort of synchronous polling it is not asynchronous, but it is a weird synchronous scheduler with wrong syntax sugar.

So, the short answer:
Only type async and await keyword is not enough and gives little control over things and can be dangerous in a limited microcontroller universe.

Long one:
Maybe you do not actually understand what microcontroller programming is. Yes, microcontrollers can be programmed using standard paradigms like on a PC, but with a relatively unsuccessful or unsatisfying outcome. Once you start using microcontrollers, you intentionally or unintentionally create a realtime system. They will be hard realtime systems or soft realtime systems. Hard RT systems are these when the tasks have deadlines, and these must be fulfilled at all costs (a nice example is automatic espresso machine). Soft RT systems have more laxity, and most people make them and some of them are crying out for something better. There are tasks without critical deadlines or with soft deadlines and the programmer has little control over the schedule or does not need it. Literally, there is sufficient that tasks are executed even with the delay or in different order.

On microcontrollers, you need to manage tasks in an entirely different way than multithreading or async calling (asynchronous functions besides interrupts exist even in realtime systems, but they are much less common and they are scheduled differently). Task have time when it can start, and time when it must end. You need to control when the task is executed by yourself (static scheduling) or need a predictive scheduling algorithm because everything has side effects. Side effects are there intended or hidden. Intended are relay switching, led blinking, etc. Hidden side effects like memory allocation can be dangerous. Tasks or async routines have memory usage, bus usage, and peripheries can be power-hungry, etc.

eLEcTRiCZiTy on 7 Sep 2020

Hi, I am sorry but do not want to read all comments on this issue.

Hi @eLEcTRiCZiTy, this issue has moved well past the theoretical and into practical territory. Please feel free to avail yourself of the concrete code listings in this issue and both the Circuitpython and upstream Micropython projects if you'd like to share insights from a position of context and understanding.

WarriorOfWire on 7 Sep 2020

In any case, async/await is the Pythonic way to talk about cooperative multitasking. Its overhead is implementation dependent and could be anything, including zero.
How to actually schedule these tasks when you need soft realtime (hint: you often don't) is an orthogonal problem. Same for the tasks' side effects, same for the method(s) to wake up a task that's waiting for an external event (= interrupt).

Please don't conflate these issues.

smurfix on 7 Sep 2020

@smurfix Hi, you are right. It is my fault. The scheduler is different problem and it is actually separable from an asynchronous calling implementation. Scheduler must support asynchronous tasks or they can be executed in a specific periodic task.

eLEcTRiCZiTy on 7 Sep 2020

async/await keyword support has been accepted into CircuitPython as of https://github.com/adafruit/circuitpython/pull/3540

There is currently no native scheduler implementation or interrupt-scheduled coroutines (yet!) but as @dhalbert mentioned back in August simple time-based pure-python async scheduling has been demonstrated on CircuitPython; also it's easy to reproduce from scratch (and completely understand) with not much more than following along with the good Mister Beazley https://www.youtube.com/watch?v=Y4Gt3Xjd7G8

I think it's wonderful that async/await keywords are not tied to a specific implementation of concurrency - you can use a global event loop if you like, or you can keep track of all your tasks and call send(None) on them in your own loop() method to move them along, or you can use a scoped / context manager event loop to do trio style concurrency... or you can literally do all 3 at once if you want to. The language is unopinionated about how you use coroutines and that's just dandy =)

WarriorOfWire on 13 Oct 2020

🎉1

There's one missing piece here: we need to be able to schedule a task when something happens, i.e. an interrupt routine must be able to wake up the core scheduler. This requires an atomic "check whether this attribute of that object is None; sleep until the next interrupt / for time X only if it is" library function as a building block.

MicroPython has machine.lightsleep(); I didn't find a CircuitPython equivalent. time.sleep() doesn't work for this because it's not terminated by an interrupt AFAIK.

smurfix on 17 Oct 2020

yes, i mentioned that interrupt support is not there.

But no, it is not necessary. If nobody can interrupt and there are no currently active tasks, then nobody can make a new task until the next scheduled task. See the code in the tasko library if you are unsure of how that works. Just like in CPython, you need to call sleep on the event loop if you want the event loop to do the sleeping.

WarriorOfWire on 17 Oct 2020

I'm not talking about the application, I'm talking about the event loop itself. In CPython the event loop doesn't call "sleep", it calls "select" or "epoll" or whatever, which is a sleep that can be stopped by what's essentially an interrupt.

What do you mean, it's not necessary? Of course it's necessary. People want to put the MCU into some sleep state when it has nothing to do, that saves a ton of battery power. There are lots of situations where you want the sleep to end when an interrupt arrives. Input pin change, serial character arrives, I2C slave gets selected … why should I wake up my MCU every 10ms to check for that? that' what interrupts are for. I want to use them.

An interruptible scheduler works by

while there are any runnable tasks: run them.
disable interrupts
if there are any runnable tasks: race condition averted! re-enable interrupts, start at the top
if there's a timeout (i.e. a task that's scheduled to run in the future), tell the hardware to trigger an interrupt when the time is up
PUT_CPU_TO_SLEEP hardware instruction (which implicitly re-enables interrupts)

Without steps 2 and 3 you'd get a race condition. You'd also get one if you re-enable interrupts before SLEEPing, the MCU must do that atomically as part of its SLEEP instruction.

If time.sleep() was guaranteed to terminate when an interrupt arrives, great, we could use it for the last two steps of the above algorithm – but I just looked at the code, and mp_hal_delay_ms() obviously doesn't do that. It also runs background tasks while sleeping, thus we can't even disable interrupts before calling it.

To build a "real" event loop, we'd need to be able to call port_interrupt_after_ticks() and port_sleep_until_interrupt() from Python, in addition to disabling and re-enabling interrupts (which is already possible). Simple enough to do IMHO.

smurfix on 17 Oct 2020

friend be my guest and implement it. I look forward to your pull request.

I'm not sure if I'm communicating badly but literally all I'm trying to say is that async/await keywords exist and they do sane things (sane as defined by their CPython behavior).

Scheduling is an utterly different problem altogether and yeah, if you are designing a scheduler of course you'd want interrupts but no you definitely do not _need_ them to schedule tasks in an ecosystem that does not have interrupts. I've provided an existence proof. If you _want_ them, or if you _need_ them for some reason, _please feel welcome to contribute_. I also _want_ them and will add support at some point when I have free time unless someone else has free time and motivation first.

In the meantime, barebones async/await noun/verb pair should work great for people to learn on circuitpython, library support notwithstanding.

WarriorOfWire on 17 Oct 2020

👍1

Will do.

smurfix on 17 Oct 2020

I'm closing this issue because async and await have been enabled thanks to @WarriorOfWire. For further work we can open new issues. Thanks all!

@smurfix See https://github.com/adafruit/circuitpython/issues/2796 for deep sleep and https://github.com/adafruit/circuitpython/issues/2795 for light sleep APIs.

tannewt on 18 Oct 2020

Circuitpython: Simpler mechanisms for asynchronous processing (thoughts)

Most helpful comment

All 122 comments

deshipu Thanks!

1 is fine when you have simple functions that can operate independently and might use a couple global variables. What you've provided sort of feels like #1.

Related issues