Sphinx: Specify multiple source folder in the conf.py file

[rweickelt]

Current state: Only one source directory is possible and must be provided as a command-line argument.

Suggestion:

• Introduce a conf file parameter source_paths. This parameter can be a list of directories where Sphinx looks for source files.

Use case

• Documentation for a family of slightly different products in a product family.
• The product family is made of multiple modules and each module contributes one or more documentation chapters.
• Each product is a bundle of different modules.
• There is product-specific documentation, for instance the root index file.

Desired solution

/project/
.
|-- module-a-doc-dir
|    `-- module-a
|         `-- index.rst
|-- module-b-doc-dir
|    `-- module-b
|         `-- index.rst
|-- module-c-doc-dir
|    `-- module-c
|         `-- index.rst
|-- product-1-specific
|    `-- index.rst
|-- product-2-specific
|    `-- index.rst 
|-- conf-1.py
`-- conf-2.py

The modules are not necessarily in the same repository. They could be somewhere in the file system.
In the config files I would then write something like:

src_paths = [ 'product-1-specific', 'module-a-doc-dir', 'module-b-doc-dir' ]

(Different conf files are part of ##3133). This would produce the following output:

/build/html/
.
|-- index.html    (from product-1-specific/)
|-- module-a
|   `-- index.html
`-- module-b
    `-- index.html

Workaround 1

• Keep everything in a single repository
• Set a environment variable when building to specify the product we want to build.
• Use if/else for that variable in the conf.py and
• exclude source folders of unused modules and products (exclude_patterns in conf.py).

/project/sources/
.
|-- module-a 
|    `-- index.rst
|-- module-b
|    `-- index.rst
|-- module-c
|    `-- index.rst
|-- product-1-specific
|    `-- index.rst
|-- product-2-specific
|    `-- index.rst 
`-- conf.py

The result looks then like:

/build/html/
.
|-- index.html    (redirect to product-1-specific/index.html)
|-- module-a
|    `-- index.html
|-- module-b
|    `-- index.html
`-- product-1-specific -> index.html   (contains the toctree for the whole documentation)

The root index.html is just a redirect into the product-1-specific subdir.

Remarks

• Module documentation must be kept together with the sources
• The "root" file is in a subdir which is not elegant.
• Specify exclude (negative) patterns for each product instead of an include (positive) list

Workaround 2

• Merge all source directories as a preliminary build step and execute sphinx-build in a separate folder.
• rsync could be used to efficiently sync the files in the source directories and the build dir
• Set a environment variable when building to specify the product we want to build.
• Use if/else for that variable in the conf.py and

Contra

• Requires an additional build step.
• Requires copying files.

Pro

• Could achieve the desired output structure.
• Is actually not as bad as I thought in the first place.

[tk0miya]

Is this really useful for everyone? I don't know your case is general or not.

[terepaii]

This seems like it would be useful. Say you have a directory structure like the following:

Project/
----Product1/
--------Docs1/
----Product2/
--------Docs2/
----Product3/
--------Docs3/

Where you want to document multiple products separately for a customer rather than having all docs for all products in one directory.

[rweickelt]

@terepaii: exactly. I have updated the proposal above.

[shimizukawa]

-1 IMO the feature will complicate the specification and implementation of Sphinx.
In such situation, I usually use symlink or intersphinx or git/hg submodule feature.

[zygoloid]

We also want this. In our case, we have some hand-written .rst files (in src/docs/) and some generated .rst files (which will be put into build/docs/), and want to build a single set of documentation from these (there may be arbitrary links between the two).

[Mark90]

I feel like I'm in need of a similar feature. We need to build 2 different documents for an API project, one targeted at developers and the other at users of the service. The developers will need to see full details, whereas users should only see a subset of the information. (This isn't too weird a usecase, right?)

I started out slightly different from @rweickelt but, if I'm not mistaken, with the same goal:

/sphinx-docs/
|--/full-doc/
|----conf.py
|----index.rst
|--/user-doc/
|----conf.py
|----index.rst
|--/shared/
|----shared_section_1.rst

The idea being to include the shared sections from within full-doc and user-doc, but alas referencing files outside the root is not allowed. There is a SO post suggesting it can work, but it doesn't for me. Symlinking does the trick but is kind of hacky. Intersphinx only seems to take care of referencing between Sphinx documents.

Another (possibly better) solution might be to define multiple build configurations and include RST files depending on the build, but I don't think that's possible either.

Edit
Ended up with this simple workaround in the buildscript.

• Copy shared/ into full-doc/ and user-doc/
• Build either doc
• Remove shared/ from full-doc/ and user-doc/

[shane-axiom]

Another +1 here, this would help us avoid duplicating content when building docs for products make up of various collections of subcomponents.

[ofloveandhate]

i would like this, too.

[Letme]

+1 here - It would make for a lot cleaner build if we could specify multiple source directories in conf.py (and a lot more portable).

[rowanG077]

Is this planned at all? I have the same use case as @zygoloid. We generate a part of our rst file and the other part is in our source directory. Currently we copy the entire source docs directory to the build directory and then build it using sphinx but we would really like to use autobuild to view our documentation in real-time. There is no solution for me right now but this feature would fix that.

[tk0miya]

We don't have any plan for this. So I set milestone as "some future version".
Any PRs are always welcome.

Thanks,

[Mark90]

Well, I thought about giving the PR a shot so did some looking around. I see about 108 usages of Sphinx.srcdir (232 when counting tests). All of these locations would have to be changed to take into account not just one, but multiple extra directories. I now understand the complexity point made by @shimizukawa and don't think anyone will care enough to implement this.

I'll stick with my workaround :-)

[rowanG077]

Yeah I also looked at making a PR but that single input directory is very interwoven with sphinx and isn't fixed easily.

Instead I have implemented a project that offers this functionality. You specify multiple input directories, a temp directory and an output directory and it will merge all directories and build. It also offers build on change functionality:

https://github.com/rowanG077/sphinx-multibuild

[carlescufi]

cc @ru-fu

[lucieleonard]

Hello.
I think this could help for my use case, but I let you judge (I'm not really technical sorry).

Context: We have several products, in different repositories, all using sphinx for the docs generation.
For now, I generate all docs one-by-one, and push them to our website individually. It gives 1 html space for each product.

Goal: I'd like to be able to generate 1 HTML containing all docs from those repositories, preferably with a "master index" to be able to tweak my index and organize my html.

It would give:

"master index":

• product1 installation
• product2 installation
• configuration
• etc.

Will this help me or do you know something that could achieve that?

Thanks!

[Letme]

what you have is a problem in this issue, but there are also workarounds presented here.

You will need to do staging all into one directory under which all documentation is linked, if you cant make products as a subdirectory of that main documentation directory anyway. This for me is however a nasty workaround as we end up replicating whole repository of documentation while we could simply add include paths (like you do for compilers) to make it easy. And then also dependency tracking would make it easy as original files would actually change and be monitored, while now symlinks are monitored.

[Mekk]

I just wrote another conf.py with long exclude list just to make smaller „extract” from the bigger document. And I keep forgetting to exclude new items. And I can't find non-hacky way to have different source for top-level index.html.

In general including seems much more natural than excluding.