Docker-py: (Optionally) Return futures

[rgbkrk]

In order for me to use this in tornado and other asynchronous frameworks, I either have to make direct calls to the Docker API (using an async http client) or wrap a threadpool executor around this.

I'd like to see if there's a way for us to provide both a synchronous and asynchronous version of docker-py.

[rgbkrk]

I wrote a terrible way to do this when working with Tornado:

class AsyncDockerClient():
    '''Completely ridiculous wrapper for a Docker client that returns futures
    on every single docker method called on it, configured with an executor.
    If no executor is passed, it defaults to ThreadPoolExecutor(max_workers=2).
    '''
    def __init__(self, docker_client, executor=None):
        if executor is None:
            executor = ThreadPoolExecutor(max_workers=2)
        self._docker_client = docker_client
        self.executor = executor

    def __getattr__(self, name):
        '''Creates a function, based on docker_client.name that returns a
        Future. If name is not a callable, returns the attribute directly.
        '''
        fn = getattr(self._docker_client, name)

        # Make sure it really is a function first
        if not callable(fn):
            return fn

        def method(*args, **kwargs):
            return self.executor.submit(fn, *args, **kwargs)

        return method

The right way to do this would be to just extend your base class and change the underlying client.

/cc @minrk

[leonmax]

+1

[mastak]

+1

[jdavanne]

+1

[SakuraSound]

Is there any status for adding futures support to docker-py? Would like to see a solution within docker-py that isn't just utilizing asyncio.run_in_executor(INSERT DOCKER-PY COMMAND).

EDIT: perhaps this could be a separate project? The "simple" path would be duplication of code, but i could see how that would be messy...

[zbyte64]

Here is my approach using requests_futures:

from requests_futures.sessions import FuturesSession
from docker import Client

class AsyncIOClient(FuturesSession, Client):
    def request(self, *args, **kwargs):
        return FuturesSession.request(self, *args, **kwargs).result()

Returning a future at the request level mucks up everything but at least this way I get co-operative threading. Edit: My understanding is that this is still blocking at the IO level.

[cecton]

Hi everyone. I'm thinking of this too and wonder if it would be a good idea to have the Client class to use aiohttp's Session class instead of requests?

The idea is that requests doesn't handle async, so it would be nice to get async methods for every action. (Does that make make sense?)

[graingert]

https://pypi.python.org/pypi/yieldfrom.requests is one option

The solution for all these blocking libraries, is to use async/await coroutines, but they will not be usable until 2020-01-01 when all currently supported Python versions without async/await are both EOL.

Ideally requests, docker-py, etc etc will create an asyncio version of their libraries then import it into a legacy blocking version.

[graingert]

twisted now supports asyncio, and so you can create code that works with https://github.com/twisted/treq instead. This way there can be a blocking api that wraps the Twisted calls, and there can be a non-blocking api using Twisted or asyncio via Twisted on Python 3.

[graingert]

However it might be better to use the https://sans-io.readthedocs.io/ approach, then people can plug their own session in.

[SakuraSound]

I think a BYO-Netlib could be good way to deal with this, though I know that would be a much larger effort. My guess is that we would still need an asynchronous docker client class to enable async/await for all the functions

[graingert]

you could use pypi https://pypi.python.org/pypi/deferred just need to poke @hawkowl or @glyph to upload a fresh one

[graingert]

this way you can write a blocking request session wrapper that takes deferreds and the aiodocker-py library can replace it

[glyph]

If this would be useful we'd be quite happy to help out; just let us know!

However, Deferred is a pretty slow-moving abstraction, so you're unlikely to need any new features; you can just pip install it and try it out as-is.

[graingert]

@glyph well I'm really looking to have the async/await support here: https://github.com/twisted/twisted/pull/489

also wheels

[glyph]

@graingert Oh! Well in that case, yes, we should definitely do that :).

[graingert]

and maybe get it off the launchpad and into Git(Hu|La)b.

[glyph]

It's currently on GH as https://github.com/mikeal/deferred

[glyph]

(The plan is to move it into the Twisted org, of course.)

[graingert]

ah I found it on https://launchpad.net/deferred

[glyph]

Yeah that was the original location.

[oberstet]

FWIW, there is also txaio which would allow an implementation that runs on both Twisted and asyncio (user choice) ..

Disclosure: I am affiliated with txaio

[graingert]

@cecton have a look at this: https://github.com/aio-libs/aiodocker

[cecton]

@graingert Thx! I already saw it (too late) but I wanted something closer to docker-py and more official (because then Docker maintain it).

But my approach is bad too... I end up adapting all the code to put "async" and "await" keywords everywhere. This is definitely not what I wanted. I wanted some kind of wrapper on docker-py that would make the call asynchronous but now I kinda realize it's impossible or I totally missed something.

.... actually.... I just realize now that it is under aio-libs so it's maintain by the devs of aiohttp. 😕 great lol

[cecton]

To whom is interested: I managed to wrap docker-py: https://github.com/tenforce/docker-py-aiohttp

It can certainly be ported to other asyncio libraries using the same principles. The core of the principles are located here: https://github.com/cecton/requests2aiohttp

In brief: I made a class that is meant to override requests.sessions.Session. All the HTTP calls are redirected to aiohttp and I have a custom Response class that wraps aiohttp's Response class. Its main purpose is to have a common API with requests.sessions.Session so the docker-py code can use it the same way.

It's not state of the art but I added integration tests for the same Docker versions tested by docker-py and it seems very stable.