Readthedocs.org: Support auto-generated API docs

[radix]

I use sphinx-apidoc to auto-generate API documentation for my project. Right now I have to commit these auto-generated files to my repository so that RTD can build them into HTML docs. It'd be cool if RTD could run sphinx-apidoc for me, since it's easy to forget to regen API docs and commit them to my repo after making changes to my code.

[radix]

fwiw I'd also be happy if RTD just allowed me to specify my own build process, so it wouldn't to specifically hardcode support for sphinx-apidoc. Though that may also be difficult to allow in a secure way.

[sontek]

I'm also trying to use sphinx-apidoc for my documentation. I use a tox -e docs to generate my docs

[jantman]

+1 on this... I'm sure, even documenting it in the release procedure, that I'll forget to generate the apidocs and commit them...

[gregmuellegger]

We are currently developing a new build system. And I assume we could integrate this in there somehow. However it's not there yet, so I assume we won't have a fix for this in the next couple of weeks, but more in the terms of months.

[ghost]

+1

Adding this support would be incredible and would significantly reduce the friction associated with keeping documentation current. If sphinx-apidoc was supported in readthedocs, a developer wouldn't even need the sphinx toolchain installed locally if they didn't want; their only responsibility would be to keep docstrings up to date.

[matteobachetti]

+1

[posita]

:+1:

[msarahan]

+1

[jsmedmar]

+1

[alyjak]

FYI, you can work around this by adding something like the following to your project's conf.py file:

def run_apidoc(_):
    modules = ['a_list_of',
               'python_module_directories',
               'in_your_project']
    for module in modules:
        cur_dir = os.path.abspath(os.path.dirname(__file__))
        output_path = os.path.join(cur_dir, module, 'doc')
        cmd_path = 'sphinx-apidoc'
        if hasattr(sys, 'real_prefix'):  # Check to see if we are in a virtualenv
            # If we are, assemble the path manually
            cmd_path = os.path.abspath(os.path.join(sys.prefix, 'bin', 'sphinx-apidoc'))
        subprocess.check_call([cmd_path, '-e', '-o', output_path, module, '--force'])

def setup(app):
    app.connect('builder-inited', run_apidoc)

This will run sphinx-apidoc during the builder-inited sphinx core event. It should work on readthedocs as well as locally, enabling you to autogenerate your api docs without having to store the output in your git repo. You can even cross reference your manual documentation with these API references.

[Maxyme]

Another solution which doesn't involve directly calling a subprocess is to use the following in conf.py:

from sphinx.apidoc import main
main(['-e', '-o', ...])

instead of

subprocess.check_call([cmd_path, '-e', '-o', output_path, module, '--force'])

This solved one of the issues I had was with long path names which are restricted on readthedocs.

[BowenFu]

@alyjak @Maxyme Thanks. And this works for me.

def run_apidoc(_):
    from sphinx.apidoc import main
    import os
    import sys
    sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
    cur_dir = os.path.abspath(os.path.dirname(__file__))
    module = '.'
    output_path = os.path.join(cur_dir, 'source')
    # main(['-e', '-o', output_path, module, '--force'])

def setup(app):
    app.connect('builder-inited', run_apidoc)

[TheChymera]

@BowenFu

Thank you! For your code to work, however, you would need to uncomment the main line. Not least of all, I found this incompatible with the directory structure I commonly see, where the .rst source files are all placed in the documentation directory (where config.py is also located). What worked for me was:

def run_apidoc(_):
    from sphinx.apidoc import main
    import os
    import sys
    sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
    cur_dir = os.path.abspath(os.path.dirname(__file__))
    module = os.path.join(cur_dir,"..","labbookdb")
    main(['-e', '-o', cur_dir, module, '--force'])

def setup(app):
    app.connect('builder-inited', run_apidoc)

[bpteague]

Two comments on @TheChymera 's code, which mostly works great and which I'm now using:

• You'll need to replace labbookdb with the name of your module
• Because option parsing starts with the second item in the list, -e doesn't get honored; if you want a separate page per module, you'll need to prepend None to the list passed to main.

Otherwise, a fine solution to the problem of automating API doc updates. Thanks!!

[agjohnson]

I'm not sure if this is a feature that we'll implement directly into RTD. Having the above solution in our docs would be a great step though! We have similar content under docs/guides.

If we implemented this anywhere, I think we could implement this in our YAML config.

[Guts]

Since last version, the behavior is the same as local doc build, as expected. So, I removed the workaround made in March.

Thanks for the fix.

[stsewd]

@Guts that means you just add the extension and it works as expected? Can we close this issue then?

[stsewd]

Also, maybe you wan to use https://github.com/rtfd/sphinx-autoapi

[Guts]

@Guts that means you just add the extension and it works as expected? Can we close this issue then?

Yes. I removed the workaround and it works. So I think it works as expected. To close the issue, Maybe we should wait for the author feedback? cc @radix

Also, maybe you wan to use https://github.com/rtfd/sphinx-autoapi

I've not really tested https://github.com/rtfd/sphinx-autoapi because it says Python 2 only (Python (2 only -- Epydoc doesn't support Python 3)) and I've not much time to give it a try and understand another Sphinx extension (I'm not a doc expert so it would requires me to really dive into).

[stsewd]

Thanks for reporting back, I will close this in a couple of days if the original author doesn't respond or anyone else is having this issue :)

[radix]

yeah, go ahead and close it if you have a resolution, I am not going to have a chance to verify it soon. Thanks very much!

[stsewd]

Ok, I'm closing, please let us know if you have the problem again or if the solution works for you too

[erwanp]

I've been using @TheChymera fix with great success until now, but I have to comment it when I want to build the docs locally.

Does the @Guts update mean I can drop this fix, add extensions = ['autoapi.extension'] in my conf.py, and it will run both locally and on RTD?

Edit: I didnt realize at the time, but autoapi and sphinx-apidoc are too different libraries with different parsing methods. Output may be slightly different.

[stsewd]

Does the @Guts update mean I can drop this fix, add extensions = ['autoapi.extension'] in my conf.py, and it will run both locally and on RTD?

I think so, if you require to execute code conditionally you can see https://docs.readthedocs.io/en/latest/faq.html#how-do-i-change-behavior-for-read-the-docs

[Guts]

Does the @Guts update mean I can drop this fix, add extensions = ['autoapi.extension'] in my conf.py, and it will run both locally and on RTD?

That's what it means. At least, this is what I did: a workaround to handle local/RTD build envs, then removing it with the RTD update.

[jakirkham]

Sphinx 1.6 to 1.7 broke some things that the aforementioned workaround relied on. A Sphinx 1.6 and 1.7+ compatible version is included below.

def run_apidoc(_):
    ignore_paths = [
        ...
    ]

    argv = [
        "-f",
        "-T",
        "-e",
        "-M",
        "-o", ".",
        ".."
    ] + ignore_paths

    try:
        # Sphinx 1.7+
        from sphinx.ext import apidoc
        apidoc.main(argv)
    except ImportError:
        # Sphinx 1.6 (and earlier)
        from sphinx import apidoc
        argv.insert(0, apidoc.__file__)
        apidoc.main(argv)


def setup(app):
    app.connect('builder-inited', run_apidoc)

[jakirkham]

Looks like there is some work going on upstream to add autogeneration of API docs to Sphinx.

xref: https://github.com/sphinx-doc/sphinx/pull/4101

[stephenfin]

Looks like there is some work going on upstream to add autogeneration of API docs to Sphinx.

xref: sphinx-doc/sphinx#4101

This is also supported out-of-tree using the sphinxcontrib.apidoc extension. This is used extensively in the OpenStack community and works quite well for our purposes.

[SBCV]

With Sphinx 3.1.1 I needed to import
from sphinx.ext.apidoc import main
instead of
from sphinx.apidoc import main