Requests: Ability to prefix session URLs

[foxx]

It would be nice to be able to prefix session URLs to avoid repetitive code.

Consider the following code;

from requests import Session
class MyTestCase(LiveServerTestCase):
    def test_url(self):
        s = Session()
        s.get("{}/api/people'.format(self.server_url))
        s.get("{}/api/cats'.format(self.server_url))

This could look a lot prettier by doing;

from requests import Session
class MyTestCase(LiveServerTestCase):
    def test_url(self):
        s = Session(prefix_url='http://example.com')
        s.get('/api/people')
        s.get('/api/cats')

In the mean time, I was able to achieve this by using a subclass;

class Session(requests.Session):
    """Helpers for performing API queries"""
    def __init__(self, *args, **kwargs):
        self.prefix_url = kwargs.get('prefix_url', None)
        super(Session, self).__init__(*args, **kwargs)

    def request(self, method, url, *args, **kwargs):
        if self.prefix_url:
            url = "{}{}".format(self.prefix_url, url)
        return super(Session, self).request(method, url, *args, **kwargs)

@kennethreitz would such a PR be accepted? Or does anyone know of a better way this could be done?

[sigmavirus24]

As we've described in previous issues (https://github.com/kennethreitz/requests/issues/2445, https://github.com/kennethreitz/requests/issues/133) this is not something we feel belongs in requests itself. There are several libraries that do this in differing ways.

The easiest way to implement this yourself would be to use the urljoin function provided by urllib/urlparse/etc. (depending on your version of Python).

[foxx]

I agree that prefix_url is not the most flexible solution, and hooks/subclasses would be better suited, as suggested by @kennethreitz in #133.

However it's not immediately clear from the docs how to do this, therefore I'd like to propose some sort of docs patch to make this clearer. Would such a patch be accepted?

[sigmavirus24]

133 is extremely old in this case (but represents the decision to not include this). Most of the hooks that existed when Kenneth responded to #133 no longer exist. Currently the only supported hook is the response hook which happens after a requests.Response object has been constructed but before it's given to the user. This, is very late in the process though.

[foxx]

Again, I'd argue this is a common use case which genuinely improves code quality, and is at least worthy of a docs patch. However if @kennethreitz votes down on this, then so be it.

[sigmavirus24]

and is at least worthy of a docs patch

I'm not sure I understand what needs to be added to the docs. Would you mind expounding on that?

[foxx]

Apologies for the slow response on this. After trying several different approaches, it seems the most flexible and cleanest approach is subclassing.

Here's an example that meets our particular needs of prepending a URL;

from requests import Session
from urlparse import urljoin

class LiveServerSession(Session):
    def __init__(self, prefix_url):
        self.prefix_url = prefix_url
        super(LiveServerSession, self).__init__()

    def request(self, method, url, *args, **kwargs):
        url = urljoin(self.prefix_url, url)
        return super(LiveServerSession, self).request(method, url, *args, **kwargs)

As for documentation, I'm thinking perhaps a "recipes" page of patterns and user contributed subclasses for achieving common goals, which aren't appropriate for inclusion into the core.

Thoughts @kennethreitz / @sigmavirus24 ?

[sigmavirus24]

So we have some recipes sprinkled through the documentation, but for the most part we haven't created one single recipes section. And the recipes that are in the docs are either minimal enough to just work or aren't used by anyone and so we haven't received any bug reports about them. The few really important ones have been abstracted into the requests-toolbelt which adds things as necessary.

You can open a PR there with tests to start a discussion about including this. There are a few things we'll probably want to change because I can already foresee some bug reports coming in if we did ship it in the toolbelt.

[foxx]

Ah, I didn't know about requests-toolbelt, I'll take a look and see about submitting a PR.

[jaraco]

@foxx Did you ever end up submitting a PR with this implementation? I wanted this feature today.

[sigmavirus24]

@jaraco double checking the toolbelt, no it does not seem as if @foxx ever did.

[antonagestam]

It would be nice with a canonical implementation of this that doesn't require either losing or replicating all type hints of the Session.request method. In that respect the toolbelt solution really isn't satisfactory.

[sigmavirus24]

Given the typeshed type hints are pretty terrifically insufficient in the first place, I don't think that's a priority for the toolbelt. Python's type hints are awfully insufficient for a library like Requests and there's enough documented on that topic.

[antonagestam]

@sigmavirus24 Sorry, but that just comes off as "because things aren't perfect they shouldn't be improved". I've gained a lot of value out of using the typeshed stubs for requests and I doubt I'm the only one.

[sigmavirus24]

I'm not saying they're not valuable, but they're wrong. They may provide value while being insufficient, incorrect, and misleading without being the right thing. It's also plausible that type hints can be valuable most of the time, but are insufficient for any kind of complexity - just like Go's type system isn't sufficient for higher order types. Regardless, if you want type hints for the toolbelt, you can add them. As it stands your comments aren't productive and as the person who started the toolbelt I won't copy junk around and do that work when it's of misleading quality nor am I about to try to provide accurate type hints. Whining to 1,400 people won't make your desires realized and this library is maintained for security purposes so you either need to take your own action or stop spamming folks

[antonagestam]

@sigmavirus24 I haven't asked anyone to do any work, if work is required to implement something here I might be able to put time in, I do make small contributions infrequently to some of the libraries I use.

I lifted this here because it seemed to me like the issue was closed because the toolbelt solution was deemed good enough. 5 years later and the Python landscape looks quite different today with static typing gaining a lot of support and becoming more of an important aspect for many. From that perspective I simply stated that the status quo isn't satisfactory.

In case I stepped on your toes, I'm sorry, perhaps I could have expressed myself differently.

[sigmavirus24]

@antonagestam I appreciate that. In short, 99% of people who make comments like yours are doing so to re-open an issue and get someone else to do whatever they want. It's given me a rather jaded perspective on comments like yours. The toolbelt probably meets the 99% use-case of people and this library has been in feature freeze for at leas 5 years (roughly since 2.0 was released). So the toolbelt is the best possible way forward for this for better or for worse.