Requests-futures: Add args to background callbacks

[earada]

Background callback args are fixed for now, but I need pass some additional information to the callback in my code.

I suggest this request function in order to allow callbacks with arguments passed by the caller code.

    def request(self, *args, **kwargs):
        """Maintains the existing api for Session.request.

        Used by all of the higher level methods, e.g. Session.get.

        The background_callback param allows you to do some processing on the
        response in the background, e.g. call resp.json() so that json parsing
        happens in the background thread.
        """
        func = sup = super(FuturesSession, self).request

        background_callback = kwargs.pop('background_callback', None)
        background_callback_args = kwargs.pop('background_callback_args', None)
        if background_callback:
            def wrap(*args_, **kwargs_):
                resp = sup(*args_, **kwargs_)
                if background_callback_args:
                    background_callback(self, resp, *background_callback_args)
                else:
                    background_callback(self, resp)
                return resp

            func = wrap

        return self.executor.submit(func, *args, **kwargs)

Example usage:

id = ...
future = session.get(url, background_callback=fun_cb, background_callback_args=(id,))

[ross]

common python-ish solutions would to either use a closure or lambda i.e.

future = session.get(url, background_callback=lambda sess, resp: fun_cb(sess, resp, id))

or

id = 'foo'

def fun_cb(sess, resp):
    # id can be accessed here
    pass

future = session.get(url, background_callback=fun_cb)

i haven't run across a situation with a pattern like background_callback_args in python, if you have/it's common i'd be open to the idea, just point me to some examples (unless lots of people are going to expect it i would prefer to avoid adding more ways to do things.)

best,
-rm

[earada]

Lambda one works like a charm ;)

Tnxs for the python-ish response. I think it is more adaptable and let a simpler code.

[ross]

cool. glad to help.

[yulyt]

Using the lambda doesn't quite work for me -- it's not really respecting the params I'm sending down (unless I'm having the wrong expectations):

When running this test:

def bg_call(sess, resp, hostname):
    # host from response.
    resp_host = urlparse.urlparse(resp.url).hostname.split('www.')[-1]
    log.debug('orig hostname %s, from resp %s', hostname, resp_host)

reqs_num_workers = 10
session = FuturesSession(max_workers=reqs_num_workers)

targets = ['google.com', 'foo.com']
types = ['http://', 'https://']
for hostname in targets:    
    log.debug('doing %s', hostname)
    for schema in types:
        session.get('%s%s' % (schema, hostname),
                    background_callback=lambda sess, resp: bg_call(sess, resp, hostname),
                    timeout=2)

I get this output (note how we end up with orig hostname google.com, from resp foo.com, meaning the original hostname I sent down in the request is NOT the one corresponding to the response):

07/02/2014 18:10:36 [DEBUG] [futures_test] [5057] - doing google.com                                                                                                          
07/02/2014 18:10:36 [DEBUG] [futures_test] [5057] - doing foo.com
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTPS connection (1): google.com
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTP connection (1): google.com
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTP connection (1): foo.com
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTPS connection (1): foo.com
07/02/2014 18:10:36 [DEBUG] [connectionpool] [5057] - "GET / HTTP/1.1" 301 219
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTP connection (1): www.google.com
07/02/2014 18:10:36 [DEBUG] [connectionpool] [5057] - "GET / HTTP/1.1" 301 220
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTPS connection (1): www.google.com
07/02/2014 18:10:36 [DEBUG] [connectionpool] [5057] - "GET / HTTP/1.1" 200 None
07/02/2014 18:10:36 [DEBUG] [futures_test] [5057] - orig hostname foo.com, from resp google.com
07/02/2014 18:10:36 [DEBUG] [connectionpool] [5057] - "GET / HTTP/1.1" 301 184
07/02/2014 18:10:36 [INFO] [connectionpool] [5057] - Starting new HTTP connection (1): www.foo.com
07/02/2014 18:10:36 [DEBUG] [connectionpool] [5057] - "GET / HTTP/1.1" 200 None
07/02/2014 18:10:36 [DEBUG] [futures_test] [5057] - orig hostname foo.com, from resp google.com
07/02/2014 18:10:36 [DEBUG] [connectionpool] [5057] - "GET / HTTP/1.1" 200 1494
07/02/2014 18:10:36 [DEBUG] [futures_test] [5057] - orig hostname foo.com, from resp foo.com

[ross]

i believe there's a common scope problem there. you're using in a variable hostname in the right side of the lambda that will have changed (many times) by the time the lambda is called.

nothing specific to requests-futures, it's just a python scope thing. this stack overflow post seems to do a good job of describing the situation and solution:
http://stackoverflow.com/questions/938429/scope-of-python-lambda-functions-and-their-parameters

[crazygit]

meet the same question, work out it followed as @ross

http://stackoverflow.com/questions/938429/scope-of-python-lambda-functions-and-their-parameters
When a lambda is created, it doesn't make a copy of the variables in the enclosing scope that it uses. It maintains a reference to the environment so that it can look up the value of the variable later. There is just one m. It gets assigned to every time through the loop. After the loop, the variable m has value 'mi'. So when you actually run the function you created later, it will look up the value of m in the environment that created it, which will by then have value 'mi'.

One common and idiomatic solution to this problem is to capture the value of m at the time that the lambda is created by using it as the default argument of an optional parameter. You usually use a parameter of the same name so you don't have to change the body of the code:

for m in ('do', 're', 'mi'):
   funcList.append(lambda m=m: callback(m))

so, @yulyt

 session.get('%s%s' % (schema, hostname),
                    background_callback=lambda sess, resp, hostname=hostname: bg_call(sess, resp, hostname),
                    timeout=2)

will work.

[yulyt]

I see, I understand the issue. Thanks for the references @ross and @crazygit :-)