Kibana: [Canvas] PDF Report Failures with Watcher in V6.7.2 due to invalid escape characters in job params

[MorrieAtElastic]

Kibana version: 6.7.2

Elasticsearch version: 6.7.2

Server OS version:

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):

Describe the bug:
User reports that he is unable to generate reports from Canvas and receives the following error:

{"type":"log","@timestamp":"2020-05-03T08:56:03Z","tags":["error","reporting","download"],"pid":13319,"message":"Report k9qrfl8c0a9zdd1f18490rvz has non-OK status: [500] Reason: [{\"message\":\"Reporting generation failed\",\"reason\":\"Max attempts reached (3)\"}]"}
{"type":"response","@timestamp":"2020-05-03T08:56:03Z","tags":["api"],"pid":13319,"method":"get","statusCode":500,"req":{"url":"/api/reporting/jobs/download/k9qrfl8c0a9zdd1f18490rvz","method":"get","headers":{"accept-charset":"UTF-8","kbn-xsrf":"reporting","content-length":"0","host":"logger01.nonpci.aws-ire.credorax.com:5601","connection":"Keep-Alive","user-agent":"Apache-HttpClient/4.5.2 (Java/12.0.1)","accept-encoding":"gzip,deflate"},"remoteAddress":"10.100.82.22","userAgent":"10.100.82.22"},"res":{"statusCode":500,"responseTime":35,"contentLength":9},"message":"GET /api/reporting/jobs/download/k9qrfl8c0a9zdd1f18490rvz 500 35ms - 9.0B"}

User has no problem generating PDF reports from basic Kibana and that the errors only come from Canvas.

The above error message generated when verbose logging was enabled: the log will be attached to this ticket. I am unable to find any other helpful diagnostics, and this does not appear to be a timeout issue. Extending the timeout extends the time before an error message is generated, but it seems that the timeout can be extended indefinitely and the report still fails. Further, the messages generated do not mention timeouts but rather state that the report is failing after 3 unsuccessful attempts.

Steps to reproduce:
I have not tried to reproduce user's results, however user claims this problem can be reproduced when trying top print a report generated from a single canvas workpad.

Any additional context:
Working with Kibana developers (specifically @tsullivan) prior to filing this bug, we are now of the opinion that the problem lies with the way that Canvas generates the POST URL which would be included in a watch, and that problems arise specifically if the URL contains a %20 (URL-encoded space character). We believe that we have reproduced user's errors with the following curl command:

curl \
-XPOST \
-uelastic:changeme \
-H"kbn-version: 8.0.0" \
https://spicy.local:/kbn/api/reporting/generate/printablePdf\?jobParams\=\(browserTimezone:America%2FPhoenix,layout:\(dimensions:\(height:720,width:1080\),id:preserve_layout\),objectType:'canvas%20workpad',relativeUrls:\!\(%2Fapp%2Fcanvas%23%2Fexport%2Fworkpad%2Fpdf%2Fworkpad-e08b9bdb-ec14-4339-94c4-063bddfd610e%2Fpage%2F1,%2Fapp%2Fcanvas%23%2Fexport%2Fworkpad%2Fpdf%2Fworkpad-e08b9bdb-ec14-4339-94c4-063bddfd610e%2Fpage%2F2,%2Fapp%2Fcanvas%23%2Fexport%2Fworkpad%2Fpdf%2Fworkpad-e08b9bdb-ec14-4339-94c4-063bddfd610e%2Fpage%2F3\),title:'%5BeCommerce%5D%20Revenue%20Tracking'\)

This returns the following response:

{"statusCode":400,"error":"Bad Request","message":"invalid rison: (browserTimezone:America/Phoenix,layout:(dimensions:(height:720,width:1080),id:preserve_layout),objectType:canvas workpad,relativeUrls:!(/app/canvas#/export/workpad/pdf/workpad-e08b9bdb-ec14-4339-94c4-063bddfd610e/page/1,/app/canvas#/export/workpad/pdf/workpad-e08b9bdb-ec14-4339-94c4-063bddfd610e/page/2,/app/canvas#/export/workpad/pdf/workpad-e08b9bdb-ec14-4339-94c4-063bddfd610e/page/3),title:[eCommerce] Revenue Tracking)"}

Replacing "%20" with an "_" eliminates the error.

It was subsequently found that the generated POST URL would work if it was enclosed in quote pairs, i.e.,

curl \
-XPOST \
-uelastic:changeme \
-H 'kbn-version: 6.8.3' \
"https://virtual-spice:/kibana/api/reporting/generate/printablePdf?jobParams=(browserTimezone:America%2FPhoenix,layout:(dimensions:(height:720,width:1080),id:preserve_layout),objectType:'canvas%20workpad',relativeUrls:!(%2Fapp%2Fcanvas%23%2Fexport%2Fworkpad%2Fpdf%2Fworkpad-e08b9bdb-ec14-4339-94c4-063bddfd610e%2Fpage%2F1,%2Fapp%2Fcanvas%23%2Fexport%2Fworkpad%2Fpdf%2Fworkpad-e08b9bdb-ec14-4339-94c4-063bddfd610e%2Fpage%2F2),title:'%5BeCommerce%5D%20Revenue%20Tracking')"

The end user has now confirmed that manually executing his POST URL enclosed by a pair of quotation marks did allow the report to get generated.

[elasticmachine]

Pinging @elastic/kibana-canvas (Team:Canvas)

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[clintandrewhall]

I believe this is a reporting issue. Canvas does indeed set up the job params:

https://github.com/elastic/kibana/blob/master/x-pack/plugins/canvas/public/components/workpad_header/share_menu/utils.ts#L59

Then we rison-encode it and perform a fetch.post with the resulting URL. Should it be somehow encoded differently?

The objectType string, with the space, matches the expectations in the reporting API and tests, AFAICT. For example, the unit test for the report listing expects canvas workpad rather than canvas_workpad.

@tsullivan is there something I'm missing here?

[tsullivan]

@clintandrewhall It looks like there is a missing detail to this issue: the bug is that the POST URL being generated doesn't always work with Watcher, when copied and pasted into a Watcher script. I've updated the issue title.

@MorrieAtElastic please correct me if I am wrong, but that is how you have experienced the issue, correct?

• Canvas PDF export does work if you click "Generate PDF" in the browser
• it does not work if you click "Copy POST URL" and paste the URL string into a Watcher script
  
  
  
  • The error can be reproduced if you paste the URL string into a cUrl request without double-quotes around it
  
  
  • As @clintandrewhall noted, we have POST URL strings in API tests that do not produce these errors.
  
  

I wanted to have this issue open to get a chance to look at how the POST URL is generated for Canvas. Other than the rison encoding of job parameters, the string also should be URI-encoded.

I agree with @clintandrewhall this doesn't seem to be a Canvas issue. Going from the link to the source, I see that the encoding I expected to find is there: https://github.com/elastic/kibana/blob/master/x-pack/plugins/canvas/public/components/workpad_header/share_menu/utils.ts#L74-L76

I'd like to get someone from the ES team or Watcher to weigh in on this, since Watcher is the HTTP client in this scenario.

1. For this URL, why does Watcher behave the same way curl behaves when the URL is not double-quoted? Is that expected?
2. Is there a different way that Watcher should handle the string that could produce the desired behavior?

[MorrieAtElastic]

@tsullivan: Your perception of the problem is correct. the PDF can be generated in the browser: the problem comes when trying to use it in a watch and sending emails.

It does not surprise me to hear thatfailures do not come on every URL->watch as it appears that the problem is limited to processing of blank characters.

[clintandrewhall]

@tsullivan In the interim, perhaps we should consider disallowing empty spaces in the objectType parameter (and others?) considering the intention is to create a url...? Shouldn't be hard to find/replace all of those instances... unless there's a good reason to keep a space?

[tsullivan]

@clintandrewhall I think the objectType is fine, and blank characters should not cause an issue. The Workpad title also appears in job parameters, and could have quotes or a space: e.g. Tim's Workpad. Another reason I would prefer not to make changes to the Canvas objectType is that it is the key in our usage/telemetry data to identify the trends specific to Canvas reporting.

The description says:

problems arise specifically if the URL contains a %20 (URL-encoded space character).

However, back in the original issue, I tracked it to:

The single-quotes that wrap the text have to be escaped.

I'm not really certain if the issue is with single quotes, or spaces, or both. If the workaround had to be done in a Watch JSON, then I'd like to focus on what the actual workaround was. If the Reporting automation is actually using Watcher, the URL string would have to be surrounded by double quotes regardless as a Watcher "script" is JSON and that is a very restrictive syntax.

@MorrieAtElastic can you confirm that:

• it's using Watcher for Reporting automation, not a shell script or some other language.
• If it is Watcher, what was the actual workaround change needed? We should have the before & after.

Once we have those questions answered, I will loop in Elasticsearch devs that are familiar with Watcher.

[MorrieAtElastic]

• As mentioned above, the problem comes when using watcher to send the pdf. It is possible to manually generate the pdf in the reporting window without problem.

• I thought that the workaround was provided by @tsullivan was understood: surround the entire pdf URL in quotation pairs.

[tsullivan]

Hi, apologies but I still don't see how an issue like this can happen with Watcher and Reporting.

An example of creating a Watch that emails a PDF report is here: https://www.elastic.co/guide/en/kibana/current/automating-report-generation.html#_use_watcher

The syntax is JSON:

"email": {
  "to": "'Recipient Name <[email protected]>'",
  "subject": "Error Monitoring Report",
  "attachments" : {
    "error_report.pdf" : {
      "reporting" : {
        "url": "http://0.0.0.0:5601/api/reporting/generate/printablePdf?jobParams=...", 
        "retries":40, 
        "interval":"15s", 
        "auth":{ 
          "basic":{
            "username":"elastic",
            "password":"changeme"
          }
        }
      }
    }
  }
}

A string in singlequotes would be a JSON syntax error: strings must always be in doublequotes.

I'm thinking the issue is with custom scripts where the string has to be handled properly. As far as I can tell, there wouldn't be a reason why Watcher isn't handling it properly.

[aterreno]

I can confirm I've just had the same issue and fixed replacing the encoded space in the report %20 with an underscore