Aiohttp: Implementing auth handlers with `aiohttp.client`

[alexsdutton]

Is there a preferred way to hook in auth handlers to the HTTP client? If not, can we specify one, maybe with code changes to ClientSession to support it?

[asvetlov]

I don't follow. What do you want to do with auth?

[alexsdutton]

Implement e.g. OAuth, Negotiate, Digest auth on the client side, instead of just supporting Basic.

More generally, have some way of hooking in error handlers for 4xx responses that can perform additional requests before returning a response. Could also be used to transparently back off on 420 and 429 responses.

[alexsdutton]

My thinking was putting something near the bottom of the while True loop in ClientSession.request to allow something to be hooked in to handle 4xx (or any other) status by manipulating the request and performing it again. Nowhere near sure on the details yet though.

[alunduil]

Custom authentication mechanisms in requests would be awesome. Right now it looks like I have to subclass the ClientRequest and overload update_auth to perform my custom authentication logic.

If we simply didn't have the expectation that an auth was a BasicAuth but something that responded to an encode method (or something similar) that would be awesome. I'd also really like it if the request itself could be available for the encoding process for mechanisms that want to have a signature of the request itself.

[mattpepin]

Any progress on this one? Digest auth is often needed to interface with IoT devices/IP cameras.

[fafhrd91]

PR would be awesome. Personally, I don't have time for this

[fidiego]

Would this be a good place to mention I'd like to add support for AWS4Auth?

[dadocsis]

I have a use case for ntlm auth

[dadocsis]

Would it make sense to add event hooks similar to the requests package?
http://docs.python-requests.org/en/master/user/advanced/#event-hooks
This could allow devs to register custom authentication methods.

IE:

class MyAuth:
      async def do_custom_auth(self, response):
            if r.status_code == 401:                  
                  # do custom

      def __call__(self, request):
            request.register_hook('response', do_custom_auth)

[fidiego]

That would be very cool.

On Fri, Dec 1, 2017 at 8:25 AM, dadocsis [e-mail] wrote:

Would it make sense to add event hooks similar to the requests package?
http://docs.python-requests.org/en/master/user/advanced/#event-hooks
This could allow devs to register custom authentication methods.

IE:

`
class MyAuth:
async def do_custom_auth(self, response):
if r.status_code == 401:

do some custom auth

def __call__(self, request):
request.register_hook('response', do_custom_auth)

`

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aio-libs/aiohttp/issues/434#issuecomment-348493843,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AClF3K2xBosTHiayvQqEdRyGxdqOBnfaks5s7_4ygaJpZM4FVGp6
.

[sky-code]

maybe just add before_send hook as last point where request can be changed so by using one hook we can inject auth headers before send request ?

[asvetlov]

Public client request object is not exist (yet).

[RobertoPrevato]

I was just digging into a similar topic in the last days: OpenID Connect, OAuth2, validation of JWT issued by Azure B2C in Python, I made a working setup for servers (will share my code when I have time to clean it up).

I think aiohttp.client and ClientSession should not support more options for auth, or support handling of specific response codes (like 401, 403); these classes must be abstracted from specific auth scenarios: they already have lots to do, just to be good HTTP clients and client sessions.

In fact, I think that including support for basic auth in the constructor of ClientSession was a mistake in the first place. There are just too many scenarios: should the client support automatic login using Client Credentials flow, by client id and secret? Or maybe automatic handling of refresh_tokens to fetch new access_tokens when using Authorization code grant?

I think these things should be placed in dedicated libraries, outside of aiohttp.

[dadocsis]

Perhaps. But it would be useful to provide a means of doing so. Do you agree?

[RobertoPrevato]

I agree it would be useful, but not inside aiohttp. I think that handling specific scenarios should reside on a upper level, for example a class that internally uses an http client, and in dedicated libraries.

For example, if I had to support handling of refresh_tokens to obtain new access_tokens like I mentioned previously, I would do it in a specific class that uses an http client as one of its dependencies, and I wouldn't put it inside aiohttp, but in a dedicated library like aiohttp-oauth.

[dadocsis]

Agreed. However this would still require a refactor of the library's API to support something like ntlm auth where you will need to reuse connections. 

Sent from Yahoo Mail on Android

On Fri, Oct 12, 2018 at 11:16 AM, Roberto [e-mail] wrote:
I agree it would be useful, but not inside aiohttp. I think that handling specific scenarios should reside on a upper level, for example a class that internally uses an http client, and in dedicated libraries.

For example, if I had to support handling of refresh_tokens to obtain new access_tokens like I mentioned previously, I would do it in a specific class that uses an http client as one of its dependencies, and I wouldn't put it inside aiohttp, but in a dedicated library like aiohttp-oauth.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[RobertoPrevato]

In this case yes, if you need a way to reuse connections and this is not possible, but still I think it should be abstracted from NTLM. I just took a look at the implementation in client.py and it is currently bound to BasicAuth class. I understand you would need something like middlewares, but for client.

[aparamon]

Having interface compatible with requests.AuthBase would be really great.

[letrec]

Any plans to make this scenario easier to accommodate? I managed to get NTLM working, but it's not clear how to integrate it in a reliable way. My current thinking is to pass my own implementation of ClientResponse to ClientSession which delegates everything to the library implementation of ClientResponse with the exception of the start method, in which I could issue additional requests in case the request requires authentication. Is that something that can be recommended by the library developers?

[asvetlov]

not yet

[Mause]

@letrec any chance of you providing that code? I've been trying to do the same

[letrec]

@Mause, I'm in the middle of implementing the approach I outlined, but keep hitting all kinds of roadblocks starting from ClientResponse using not exported types in it's public interface and ClientSession directly accessing private fields of ClientResponse. Getting more and more pessimistic with every iteration.

[letrec]

Actually, I was wrong about the usage of private fields. Managed to get it working, but need some time to think on final design.

[letrec]

In NTLM one authenticates connections, not requests (https://blogs.technet.microsoft.com/mist/2018/02/14/windows-authentication-http-request-flow-in-iis/).
So if a request fails because it wasn't authenticated we need to follow it up with negotiate and authenticate requests on the same connection as the initial request.
If we just issues them on the same instance of ClientSession there's no guarantee that the connection will be the same.
This behaviour can be achieved if session is configured to use a connector with its connection limit set to 1, but it's not going to scale well.
It look like we need a special kind of connector which allocates connections from another connector and returns them to the parent connector only when it is closed.
We could create such a connector and set its connection limit to 1 and use it to create sessions that need such behaviour.
In this case the client code might look like this:

async def ntlm_get_text(connector: aiohttp.BaseConnector, url: str, domain: str, username: str, password: str) -> str: 
    async with ChildConnector(connector=connector) as child_connector:
        async with aiohttp.ClientSession(connector=child_connector) as session: 
            async with session.get(url) as response: 
                # NTLM-specific status handling with required retries 
                return await response.text() 

@asvetlov, does this approach seem feasible to you?

[asvetlov]

Honestly, looks overcomplicated.
I know nothing about NTLM though.

[letrec]

@asvetlov, you can ignore the NTLM part. The problem boils down to being able to create child connectors, which borrow a subset of connections from their parent connector.
It might be useful for other purposes as well, like having different limits for different hosts, for instance.
The issue is that create_child_connector has to be implemented in BaseConnector for all derived classes to benefit from it. I don't see how it can be implemented externally.

[asvetlov]

I still think that nested connectors drive to overcomplicated application design,

[letrec]

BaseConnector allows to set limit per host, but it's going to be the same limit for all hosts, if I understand it correctly. What I'm proposing is a generalization of this mechanism which could allow more fine-grained control over connections without introducing new abstractions. Limit per host could be accomplished in terms of this mechanism but with more flexibility.
In my mind this idea is conceptually similar to context managers or trio cancel scopes / nurseries.

[letrec]

My current approach is to create a connector with limit set to one and a new session per each request that requires authentication, so the problem is kinda solved albeit not in the most performant way, but good enough for my purposes.

[nernst]

I likewise have a need for more control over authentication, and like others that have already posted, to support NTLM SSPI.

Basically, the use case at my organization is that we have a number of webservices running under IIS on Windows that require authentication. Our python clients are also running from Windows.

Currently, we're using requests and requests_negotiate_sspi so that we don't have to explicitly specify credentials.

To achieve parallelism currently, we're using multiprocessing, which has a fair amount of per request overhead. The overhead is compounded when the python install is located on a network share due to our corporate mandated virus scan interfering on every new subprocess launched (seriously, it's about 1-2 minutes before a new Python process does anything useful when it's on a network share at my company, and no, we can't disable it because of hypersensitivity over corporate information security at a financial company).

aiohttp looked very promising when I cut a branch of our internal client to use it (saw an immediate 28% reduction in wall-clock time) when running against a local dev build of our webservice that didn't require auth. But, I cannot run my new version in either our QA or PROD environments because they require authentication (all I get is 401 errors).

I'm looking at adding support for SSPI in our client, but it is certainly not trivial. I think it's complicated enough and needed by enough people that it's worth adding to either aiohttp or a submodule (and looking around the project, submodule probably makes more sense). I don't currently have enough knowledge of aiohttp to really know how/where to integrate such an add-in, or if it's even really currently possible.

But yes, to me (and apparently others), authentication support beyond basic auth is a must have.

[letrec]

@nernst, do you actually have to use multiprocessing? Shouldn't multithreading do in this case?

[nernst]

Honestly, I haven't tested with multithreading yet, I'm working in some
legacy code I inherited this year. We have to break up our large requests
into a lot of smaller requests (our web service is returning years worth of
quant data and if we request too large of a chunk at a time, we get DB
timeouts-we're talking hundreds of millions of rows x > 500 columns).
Multithreading might help us, but asnyc seemed natural for what we needed
(lacking the authentication part).

On Tue, Sep 24, 2019, 7:24 PM letrec [e-mail] wrote:

@nernst https://github.com/nernst, do you actually have to use
multiprocessing? Shouldn't multithreading do in this case?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aio-libs/aiohttp/issues/434?email_source=notifications&email_token=ABOO2FNQ72HYMQIHWG22W6LQLKVSJA5CNFSM4BKUNJ5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7QGADQ#issuecomment-534798350,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOO2FIN6HYQG65VQMJFQUDQLKVSJANCNFSM4BKUNJ5A
.

[asvetlov]

We have no public client request class now.
I consider it as the main show stopper for requests like this.

[ryan-duve]

@letrec I would also be interested in seeing the NTLM solution you came up with, even if it starts a new session for each request. We are currently using multiprocessing with joblib to run a bunch of parallel calls with requests, where the session is authenticated like this:

session.auth = requests_ntlm.HttpNtlmAuth(username, password, session)

It works ok. I'm trying to move this to a more desired async/await flow, but stuck without NTLM auth:

import asyncio
from aiohttp import ClientSession
from requests_ntlm import HttpNtlmAuth

async def fetch(url, session):
    async with session.get(url) as response:
        print(f"url: {url} ({response.status})")
        return await response.read()

async def run():
    url = "http://server/page/{}"
    tasks = []

    conn = aiohttp.TCPConnector(limit=10)
    async with ClientSession(connector=conn) as session:
        session.auth = HttpNtlmAuth(username, password, session) # <-- CHANGED LINE
        for page in range(100):
            task = asyncio.ensure_future(fetch(url.format(page), session))
            tasks.append(task)

        responses = await asyncio.gather(*tasks)

loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run())
loop.run_until_complete(future)

Unfortunately, the above just returns 401s without explanation, but doesn't give a hint of where things are breaking down. I also tried putting the auth object into the ClientSession constructor but get the error:

TypeError: BasicAuth() tuple is required instead

[letrec]

@ryan-duve, when you get the 401 it means the connection is not authenticated. Check https://github.com/requests/requests-ntlm/blob/master/requests_ntlm/requests_ntlm.py#L74 to get an idea how to do that.