Sphinx: Sphinx doesn't render autodoc_docstring_signature correctly w/ Python 2

[mengxr]

Subject: Some pyspark.ml docs were generated without rendering the docstring signature when Python 2 is used.

Problem

See https://issues.apache.org/jira/browse/SPARK-24530. We use autodoc_docstring_signature in PySpark to render some ML class docs correctly. A correct version looks like the following:

And when it doesn't render correctly, it shows

We tried different Python/Sphinx versions and it seems it only appears with Python 2.

Procedure to reproduce the problem

# use python 2
git checkout [email protected]:apache/spark.git
cd spark/python/docs
make html
open _build/html/pyspark.ml.html

Check the generated doc of LogisticRegression

Error logs / results

Didn't notice any suspicious warning messages (-v -w warn.txt).

Reproducible project / your project

apache/spark

Environment info

• OS: macOS
• Python version: 2
• Sphinx version: v1.6.3

[tk0miya]

Okay, I'll take a look later.

[tk0miya]

inspect.getargspec() can't recognize its argument well. It seems some decorator hides its arguments.

$ python
Python 2.7.15 (default, May  1 2018, 22:52:16)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyspark.ml.classification import LogisticRegression
>>> import inspect
>>> inspect.getargspec(LogisticRegression.__init__)
ArgSpec(args=['self'], varargs='args', keywords='kwargs', defaults=None)

AFAIK, there are no way to obtain original signature of the methods in python 2.x.
So I feel it is hard to fix this.

[HyukjinKwon]

Hi, @tk0miya. Thanks for checking this out. I was hesitant to leave a comment and I am glad that we had similar investigations. Yup, the cause looked indeed functools.wraps fails to copy its signature in Python 2 whereas Python 3 succeeds, and I agree there could be no way to find out the original signature in this case.

One thing I am wondering is though, I thought we could override the signature by autodoc feature when the feature is enabled and the first line of docstring specifies its signature.

[tk0miya]

It seems the first line of the docstring is blank. Unfortunately, the signature is defined at next line. So autodoc_docstring_signature does not work.

$ python
Python 3.6.5 (default, Mar 31 2018, 01:44:52)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyspark.ml.classification import LogisticRegression
>>> LogisticRegression.__init__.__doc__.splitlines()[0]
''
>>> LogisticRegression.__init__.__doc__.splitlines()[1]
'        __init__(self, featuresCol="features", labelCol="label", predictionCol="prediction",                  maxIter=100, regParam=0.0, elasticNetParam=0.0, tol=1e-6, fitIntercept=True,                  threshold=0.5, thresholds=None, probabilityCol="probability",                  rawPredictionCol="rawPrediction", standardization=True, weightCol=None,                  aggregationDepth=2, family="auto",                  lowerBoundsOnCoefficients=None, upperBoundsOnCoefficients=None,                  lowerBoundsOnIntercepts=None, upperBoundsOnIntercepts=None):'

Could you check your docstring please?

[HyukjinKwon]

Ah, thank you so much @tk0miya. Will give some updates here after checking and fixing it within Spark side

[tk0miya]

PEP-257 expects multiline-docstring starts just after triple-quotes like this.

def complex(real=0.0, imag=0.0):
    """Form a complex number.

    Keyword arguments:
    real -- the real part (default 0.0)
    imag -- the imaginary part (default 0.0)
    """
    if imag == 0.0 and real == 0.0:
        return complex_zero
    ...

So the rule of autodoc_docstring_signature is not strict to me.

[HyukjinKwon]

Hmmm .. but

def complex(real=0.0, imag=0.0):
    """
    Form a complex number.

    Keyword arguments:
    real -- the real part (default 0.0)
    imag -- the imaginary part (default 0.0)
    """
    if imag == 0.0 and real == 0.0:
        return complex_zero
    ...

this also complies PEP 257:

The summary line may be on the same line as the opening quotes or on the next line.

[tk0miya]

Oh, I'd overlooked it. +1 for support the next line.

@shimizukawa what do you think?

[shimizukawa]

Well, I think that it is not a rare case that the first letter appears in the second line.

def function(args):
    """
    First chars appear after triple-quote line
    """

I think that it is good to reference the behavior of inspect.getdoc.

>>> def foo(args):
...     """first
...
...     second
...     """
...
>>> def bar(args):
...     """
...     first
...
...     second
...     """
...
>>> print(foo.__doc__)
first

    second

>>> print(bar.__doc__)

    first

    second

>>> import inspect
>>> print(inspect.getdoc(foo))
first

second
>>> print(inspect.getdoc(bar))
first

second
>>>

[tk0miya]

Okay, let's improve it :-)

[tk0miya]

Oh, I'm very sorry. My investigation was wrong. The blank line is not related with this problem. autodoc correctly skips it on processing.

• inspect.getargspec() which is used by python 2.7 does not support to obtain an argument list of decorated function (as commented above).
• autodoc only uses a docstring of class by default. It means a docstring of __init__() method is not used (see doc of autoclass_content: http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autoclass_content).
• autodoc_docstring_signature expects the first line is a signature. But the docstring of LogisticRegression.__init__() is not considered as signature because it contains : at tail. So it will be ignored even if autoclass_content = 'both' or ... = 'init'.

So I think this is not a bug of autodoc_docstring_signature.

[HyukjinKwon]

Ah, thanks for taking a look and investigating this. Looks I should have taken a closer look. Will get back to you within few days after trying out.

[HyukjinKwon]

Confirmed. Thanks. At the very least, the configuration should be changed with some additional docstring changes within Spark side - it would need actually some more efforts since some other docs are broken within Spark side when autoclass_content is configured but it looks rather a problem within Spark side.

Thank you so much @tk0miya for your time and efforts.

[HyukjinKwon]

I am pretty sure this can be closed for the current status.

[tk0miya]

:-)