Google-api-python-client: Content-ID header over 64 characters gets an extra newline; breaks batch requests

[ShadowLNC]

I'm working with batch requests, and adding custom request_id values. With a request_id value of length 28, the total length of the Content-ID value inside the < and > characters is 64.

Somehow, for >=64 characters, "\n " (including trailing space) gets added before the Content-ID value, but after the "Content-ID: ", resulting in a malformed Content-ID value which makes the round trip back (response takes the form Content-ID: response- <uuid+request_id> instead of Content-ID: <response-uuid+request_id>) and throws an exception (a BatchError for a malformed Content-ID value).

I've narrowed this down to this patch of googleapiclient/http.py starting at line 1234:

# encode the body: note that we can't use `as_string`, because
# it plays games with `From ` lines.
print(message) #no newline here
fp = StringIO()
g = Generator(fp, mangle_from_=False)
g.flatten(message, unixfrom=False)
body = fp.getvalue()
print(body,[body]) #newline appears here, printing a list shows newline as \n

This can be reproduced without requiring an authorised call, using the following test code:

from httplib2 import Http
from apiclient.discovery import build
directory = build('admin', 'directory_v1')

def errorTest(id_length):
    req_id = "abcdefghijklmnopqrstuvwxyz0123456789"
    try:
        batch_dir = directory.new_batch_http_request()
        batch_dir.add(directory.groups().list(), request_id=req_id[0:id_length])
        batch_dir.execute(http=Http()) #BatchError, unprintable (separate issue)
    except Exception as e:
        print(e.reason)
        raise

for i in [27,28]:
    print(i,"############################################",i) #readability
    try:
        errorTest(i)
    except AttributeError:
        #only occurs if call not authorised; BatchError occurs first and isn't caught
        pass

[nathanielmanistaatgoogle]

Thank you for the very clear report!

[ShadowLNC]

By Design: Line Folding (RFC2822) (Unrecognised by server)

I've found that this line-folding is by design (for lines with length >78), thanks to this StackOverflow Question.

This ordinarily would not be an issue, but for the Content-ID header, it contains no spaces. As such, when the total length is over 78, line folding will occur at the first available whitespace character, which happens to be immediately after the colon, and the server does not recognise this as folding.

(I am unsure if folding is supposed to occur at the beginning of a header, as that does not appear to be explicitly described by RFC2822, but I would assume the implication is that folding may occur at the beginning of a header value.)

If I insert a space within the header value at an appropriate location, it folds there instead, and is unfolded correctly by the server, however the server does not re-fold it in the response (given an absolute maximum 998 character limit in RFC2822, this is irrelevant).

"Content-ID: <>" is 14 characters, leaving a maximum of 64 within the greater than/less than symbols (angle brackets) before folding will occur. The greater than/less than symbols have no effect on the folding (apart from altering the line length).

The folding only occurs in the MIMEMultipart object, not the MIMENonMultipart object - and only in Python3, not Python2.

Possible Workaround

My proposal, if the server-side issue cannot be fixed, is to store the request_id values separately, and uniquely identify them with a base 10 integer in the actual request, as follows:

request_id_lookup = { 0:"my_request_id", 1:"my_second_request" }
#alternatively, a list, because they are sorted and integer-indexed
request_id_lookup = ["my_request_id", "my_second_request"]

Then, the header would look something like Content-ID: <uuid+1234> - 2^64 is only 20 digits in base 10, which only pushes the line to 71 characters. As a matter of fact, the order list already behaves like request_id_lookup, so a simple enumerate call in creating the request is all that would be necessary (plus a lookup to decode it).

I am willing to write this workaround, but would like confirmation that this is the way to proceed, as it is a workaround and does not address the actual issue.

[mcdonc]

I'm currently reviewing ways to avoid the folding.

[mcdonc]

This appears to be the very simplest fix:

https://github.com/google/google-api-python-client/compare/master...mcdonc:content-id-linefolding-164

(once I finish tests, I'll submit a PR)

Thanks so much for the precise bug report (and apologies for the interminable wait for a response).