Flask: Flask isn't doctestable

[sils]

lasse@lssteady ~/tmp/pytesttest % cat asd.py 
from flask import request

def addstuff(a,b):
    """
    Adds stuff:

    >>> addstuff(4,5)
    9

    See?
    """
    return a+b

lasse@lssteady ~/tmp/pytesttest % python3   
Python 3.5.1 (default, Dec  7 2015, 12:58:09) 
[GCC 5.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asd
>>> import doctest
>>> doctest.testmod(asd)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/doctest.py", line 1940, in testmod
    for test in finder.find(m, name, globs=globs, extraglobs=extraglobs):
  File "/usr/lib/python3.5/doctest.py", line 923, in find
    self._find(tests, obj, name, module, source_lines, globs, {})
  File "/usr/lib/python3.5/doctest.py", line 982, in _find
    if ((inspect.isroutine(inspect.unwrap(val))
  File "/usr/lib/python3.5/inspect.py", line 471, in unwrap
    while _is_wrapper(func):
  File "/usr/lib/python3.5/inspect.py", line 465, in _is_wrapper
    return hasattr(f, '__wrapped__')
  File "/usr/lib/python3.5/site-packages/werkzeug/local.py", line 343, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/usr/lib/python3.5/site-packages/werkzeug/local.py", line 302, in _get_current_object
    return self.__local()
  File "/usr/lib/python3.5/site-packages/flask/globals.py", line 20, in _lookup_req_object
    raise RuntimeError('working outside of request context')
RuntimeError: working outside of request context
>>> 

Anyone any idea what happens there? I'd love to doctest my Flask application since it's especially awesome to document endpoints.

[ThiefMaster]

doctests are useful for simple functions with no outside requirements, not for web applications and their controllers (flask's "view functions") which obviously need other setup steps like database access, the request context, etc.

[sils]

I find them pretty useful, e.g.:

@app.route('/brew/coffee')
def teapot():
    """
    A simple endpoint that informs the user that this teapot is not able to brew
    coffee. See RFC 2324 for more information about the Hyper Text Coffee Pot
    Control Protocol.

    >>> response = app.test_client().get('/brew/coffee')
    >>> response.status_code
    418
    >>> assert 'Unable to brew coffee.' in response.data.decode()

    :return: A 418 response as described in RFC 2324
    """
    response = jsonify(message='Unable to brew coffee. I\'m a teapot.')
    response.status_code = 418
    return response

It adds value to the docs while helping reducing test redundancy. Also this phenomenon means that I can't doctest any routine that is within a Python file that imports Flasks request object.

[ThiefMaster]

https://pytest.org/latest/doctest.html

You can probably combine this with a pytest fixture (possibly with autouse=True) that setups the app context.

[sils]

Maybe we have a misunderstanding... I am well aware of PyTest and the documentation. The same error gets raised with PyTest on test collection. I just tried to simplify it to you and make it more concrete thus provided the sample Python shell commands.

I'm saying that I apparently cannot use doctest on any Python module that _imports_ (not even uses) the Flask request object. I hoped we could figure out what kind of magic the request object does, what kind of magic doctest does around it and where it clashes. (This may actually well be an issue with doctest but that's what I'm trying to find out.)

[ThiefMaster]

Importing the request object is fine, accessing any attributes of it is not (or doing anything that's causes a call to any of its __...__ magic methods). See werkzeug's LocalProxy

But pytest might run autouse fixtures before accessing attributes of module globals (didn't test it), in that case it would probably work.

[sils]

Interesting:

>>> accounts.request.nonsense
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/site-packages/werkzeug/local.py", line 343, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/usr/lib/python3.5/site-packages/werkzeug/local.py", line 302, in _get_current_object
    return self.__local()
  File "/usr/lib/python3.5/site-packages/flask/globals.py", line 20, in _lookup_req_object
    raise RuntimeError('working outside of request context')
RuntimeError: working outside of request context
>>> accounts.request.__doc__
"Acts as a proxy for a werkzeug local.  Forwards all operations to\n    a proxied object.  The only operations not supported for forwarding\n    are right handed operands and any kind of assignment.\n\n    Example usage::\n\n        from werkzeug.local import Local\n        l = Local()\n\n        # these are proxies\n        request = l('request')\n        user = l('user')\n\n\n        from werkzeug.local import LocalStack\n        _response_local = LocalStack()\n\n        # this is a proxy\n        response = _response_local()\n\n    Whenever something is bound to l.user / l.request the proxy objects\n    will forward all operations.  If no object is bound a :exc:`RuntimeError`\n    will be raised.\n\n    To create proxies to :class:`Local` or :class:`LocalStack` objects,\n    call the object as shown above.  If you want to have a proxy to an\n    object looked up by a function, you can (as of Werkzeug 0.6.1) pass\n    a function to the :class:`LocalProxy` constructor::\n\n        session = LocalProxy(lambda: get_current_request().session)\n\n    .. versionchanged:: 0.6.1\n       The class can be instanciated with a callable as well now.\n    "
>>> inspect.unwrap(accounts.request)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/inspect.py", line 471, in unwrap
    while _is_wrapper(func):
  File "/usr/lib/python3.5/inspect.py", line 465, in _is_wrapper
    return hasattr(f, '__wrapped__')
  File "/usr/lib/python3.5/site-packages/werkzeug/local.py", line 343, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/usr/lib/python3.5/site-packages/werkzeug/local.py", line 302, in _get_current_object
    return self.__local()
  File "/usr/lib/python3.5/site-packages/flask/globals.py", line 20, in _lookup_req_object
    raise RuntimeError('working outside of request context')
RuntimeError: working outside of request context

Summarizing facts:

• Accessing __doc__ works.
• Accessing __wrapped__ does not. (https://docs.python.org/3/library/inspect.html#inspect.unwrap)
• Accessing any nonsense throws a RuntimeError
• doctest seems to handle an AttributeError, which is the usual thing to expect.

Is there a reason why request throws a RuntimeError whenever _any_ attribute (except some default ones apparently) is accessed? (I'm new to Flask so forgive me any non knowledge please:) Shouldn't the API of a request at least be defined? It certainly never will have a __wrapped__ attribute I'd guess...

[ThiefMaster]

It's a proxy object so any access to it (unless technically impossible, e.g. for the is operator which cannot be overloaded) is forwarded to the underlying object (which is only available during a request context in case of flask.request).

When you are in a request context, the attributes of flask.request are well-defined, but without one it's impossibly to do that without making the proxy much less generic.

[sils]

I guess that's NOTABUG then for you...

[ThiefMaster]

I don't know if pytest has a hook for this (doubt it), but if you can somehow make it check whether an object is a LocalProxy instance before trying to do things with it, you could avoid this kind of breakage.

[sils]

I've filed this as a doctest issue since they should catch other exceptions too IMO: http://bugs.python.org/issue25998

[sils]

FWIW I've created a custom request object with object.__setattr__(request, '__wrapped__', None) to work around this problem. In case somebody else wants to doctest anything that contains this import _somewhere_.

[immerrr]

Got bitten by this, too. I have utility functions which I'd like to doctest in utils.py next to various views_*.py modules that mention flask.request and everything goes up in flames when I try to do py.test --doctest-modules because py.test discovery .

Python devs don't think it's their problem either, as far as I can tell from http://bugs.python.org/issue25998. Could you guys figure it out with them?

[jaraco]

This bug bit me today also. I encounter the issue in Python 3.5 but not 2.7.

[jaraco]

@sils1297 Can you share the code you used to work around the issue? I've tried simply invoking the code you shared on flask.request in place of request, but that isn't sufficient to avoid the runtime error. Also, I'm seeing the runtime error for files that don't import request. A file with only this flask import has the same issue: from flask import current_app as app.

[jeffwidman]

@jaraco which point release of Python 3.5?

[sils]

Essentially I just did a module with:

from flask import request

# See https://github.com/mitsuhiko/flask/issues/1680
object.__setattr__(request, '__wrapped__', None) 

and then in every module import request from this module instead of importing it from flask directly

[RonnyPfannschmidt]

thas plain broken :)

[sils]

Well, what should we do... we want flask, we want doctest. Both are awesome but refuse to work together. It's a hack for sure and you probably don't wanna use it in production but it's still useful.

[jaraco]

which point release of Python 3.5?

3.5.1

[jaraco]

Also, current_app is affected:

import types
import doctest

import flask


def test_attr(name):
    mod = types.ModuleType('mod_under_test')
    attr = getattr(flask, name)
    setattr(mod, name, attr)
    try:
        doctest.testmod(mod)
    except RuntimeError:
        pass
    else:
        raise Exception("Did not raise")

    object.__setattr__(attr, '__wrapped__', None)

    doctest.testmod(mod)


affected_attrs = 'request', 'session', 'current_app'
list(map(test_attr, affected_attrs))

[jaraco]

Seems that session and current_app are effected as well.

After those discoveries, I was able to write a conftest.py file (for pytest) to implement the workaround:

import importlib


def pytest_configure():
    patch_flask_for_doctest()


def patch_flask_for_doctest():
    """
    Patch flask magic objects to keep them from raising
    RuntimeErrors during doctest discovery.
    https://github.com/pallets/flask/issues/1680
    """
    flask = importlib.import_module('flask')
    object.__setattr__(flask.request, '__wrapped__', None)
    object.__setattr__(flask.session, '__wrapped__', None)
    object.__setattr__(flask.current_app, '__wrapped__', None)

Perhaps I'll release that hack as a pytest plugin.

[immerrr]

It's better to go to the source and set werkzeug.local.LocalProxy.__wrapped__ = None, fixing them all at once.

[jeffwidman]

@untitaker your thoughts?

[RonnyPfannschmidt]

any doctest will have to push an known context to work correctly

simply setting something to None is bound to break other bits

[RonnyPfannschmidt]

also doctest is not awesome - its a toy that is hard to extend and falls down painfully whenever you need to deal with more than the most simple code examples

[untitaker]

I'd rather close this as it appears unfixable.

On 23 April 2016 08:46:15 CEST, Ronny Pfannschmidt [email protected] wrote:

also doctest is not awesome - its a toy that is hard to extend and
falls down painfully whenever you need to deal with more than the most
simple code examples

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/pallets/flask/issues/1680#issuecomment-213675904

Sent from my Android device with K-9 Mail. Please excuse my brevity.

[jaraco]

also doctest is not awesome - its a toy that is hard to extend and falls down painfully whenever you need to deal with more than the most simple code examples

If doctest is a toy that is not awesome, it should probably be removed from the stdlib. On the contrary, it's a model that works well for many use cases, including Python itself. Acknowledging that it doesn't have the setup/teardown semantics and some of the more advanced techniques necessary for mature projects and that sometimes a project will abuse doctests beyond their intended purpose, it doesn't follow that doctests should be deprecated, and especially not by a library like flask.

[This issue] appears unfixable.

If the issue is unfixable, doesn't it at least deserve an explanation for why it is unfixable? I haven't yet delved into what the issue is, but it seems that raising a RuntimeError during inspection of the code is something that the LocalProxy could at least in theory avoid.

[krzysztof-sikorski]

If the issue is unfixable, doesn't it at least deserve an explanation for why it is unfixable? I haven't yet delved into what the issue is, but it seems that raising a RuntimeError during inspection of the code is something that the LocalProxy could at least in theory avoid.

From what I understand (mind you, I'm not an active framework user, just a fan), the problem is that Request object shouldn't be used standalone because it needs at least partially initialised framework (the so-called "context"). Thus the RuntimeError instead of a "more standard" AttributeError, because accessing an attribute just reveals that context is not initialised.

I guess it would be hard to fix it in Werkzeug/Flask without redesigning most of the framework architecture, so the only sensible solution seem to be adding initialisation boilerplate to each and every doctest in which the Request object is used - which kinda defeats the purpose of using doctests instead of "normal" unit tests.

[untitaker]

@jaraco I am under the impression that this issue is unfixable in an absolutely clean way, without breaking API compat (such as changing RuntimeError to AttributeError -- tests already rely on that).

"Absolutely clean" is a requirement, because while the maturity and usefulness of doctests is IMO completely off-topic, I don't think it's worth working around its deficiencies. I closed this issue prematurely because I think we need to set strict priorities when the bugtracker is already trashy as is.

[untitaker]

That said, if you figure out the issue and are able to come up with a solution, please do open a PR.

[immerrr]

@untitaker I have created a PR against werkzeug that should fix this: https://github.com/pallets/werkzeug/pull/924.