Kibana: Fatal error in discover app after upgrade from 6.3.2 to 6.4.0

Created on 24 Aug 2018 · 26Comments · Source: elastic/kibana

**Original install method (e.g. download page, yum, from source, etc.):yum

**Describe the bug:Fatal error in discover app after upgrade from 6.3.2 to 6.4.0

Steps to reproduce:
1.open the discover app
2.a fatal error is printed : "Courier fetch: Cannot read property 'forEach' of undefined"
3.

Expected behavior:

Screenshots (if relevant):

Errors in browser console (if relevant):
TypeError: Cannot read property 'forEach' of undefined
at https://kibana/bundles/commons.bundle.js:3:1085607
at Array.forEach ()
at _callee$ (https://kibana/bundles/commons.bundle.js:3:1085555)
at tryCatch (https://kibana/bundles/vendors.bundle.js:36:138904)
at Generator.invoke [as _invoke] (https://kibana/bundles/vendors.bundle.js:36:142786)
at Generator.prototype.(anonymous function) [as next] (https://kibana/bundles/vendors.bundle.js:36:140027)
at step (https://kibana/bundles/commons.bundle.js:3:1081249)
at https://kibana/bundles/commons.bundle.js:3:1081375
at

The error appears in this line :
segregatedResponses.forEach(function(responses, strategyIndex) {

Provide logs and/or server output (if relevant):

Any additional context:

Discover KibanaApp Operations bug

Source

dav3860

Most helpful comment

@carolineLe Do you happen to have a proxy in front of Elasticsearch, similar to what @florian-asche has set up? This will be the proxy located at the elasticsearch.url defined in your Kibana config file. If so then the error could be due to the proxy responding with some sort of XML or HTML response, for an example an error page due to a 502 or similar.

I've created an issue to address this problem (https://github.com/elastic/elasticsearch-js/issues/701). We currently suggest users configure their elasticsearch.url to point directly to Elasticsearch. In production, if you want high availability built into your deployment, you can configure an Elasticsearch coordinating node on the Kibana host, which can act as your endpoint. Let me know if you'd like to explore this option and I'll pull together some information for you.

Thank you everyone for your help with this issue. I'm going to close it since it's not a problem we can solve on the Kibana side. Feel free to continue leaving questions here if you have any.

cjcenizal on 24 Sep 2018

👍2

All 26 comments

I have a similar problem here. After upgrading from 6.X to 6.4.0 i get the following error message:

Error: Bad Gateway
ErrorAbstract@http://192.168.0.8/bundles/vendors.bundle.js:313:132203
StatusCodeError@http://192.168.0.8/bundles/vendors.bundle.js:313:135090
respond@http://192.168.0.8/bundles/vendors.bundle.js:313:149378
checkRespForFailure@http://192.168.0.8/bundles/vendors.bundle.js:313:148589
AngularConnector.prototype.request/<@http://192.168.0.8/bundles/vendors.bundle.js:313:157823
processQueue@http://192.168.0.8/bundles/vendors.bundle.js:197:199684
scheduleProcessQueue/<@http://192.168.0.8/bundles/vendors.bundle.js:197:200647
$digest@http://192.168.0.8/bundles/vendors.bundle.js:197:210409
$apply@http://192.168.0.8/bundles/vendors.bundle.js:197:213205
done@http://192.168.0.8/bundles/vendors.bundle.js:197:132704
completeRequest@http://192.168.0.8/bundles/vendors.bundle.js:197:136327
requestLoaded@http://192.168.0.8/bundles/vendors.bundle.js:197:135223

But it only appears for metricbeat, not for heartbeat.

I found out, that the request on /elasticsearch url get an 502 instead of 200. The nginx write in logfile, that the header is to big:

2018/08/26 21:39:59 [error] 17551#17551: *855 upstream sent too big header while reading response header from upstream, client: 193, server: kibana, request: "POST /elasticsearch/_msearch HTTP/1.1", upstream: "http://8:5601/elasticsearch/_msearch", host: "8", referrer: "http://8/app/kibana"

i fixed it by adding:

  proxy_buffers 4 256k;
  proxy_buffer_size 128k;
  proxy_busy_buffers_size 256k;

to nginx config.

florian-asche on 26 Aug 2018

👍1

It doesn't seem to be the same error for me, as the request on /elasticsearch gets a 200 when the error appears :
10.0.28.176 - MYUSER [27/Aug/2018:10:46:19 +0200] "POST /elasticsearch/_msearch HTTP/1.1" 200 56177 "https://kibana/app/kibana" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

dav3860 on 27 Aug 2018

Same error after upgrade from 6.3.2 to 6.4.0 when i open a dashboard

joschi99 on 28 Aug 2018

Same error over here:

Courier fetch: Cannot read property 'forEach' of undefined

I noticed the error goes away when I remove all "Scripted Fields" for the index pattern.

musskopf on 29 Aug 2018

Are there errors in your proxy? For example nginx?

florian-asche on 29 Aug 2018

I did an upgrade from 6.3.2 to 6.4.0 and got the following similar error:
Courier fetch: responses is undefined
Version: 6.4.0
Build: 17929

Error: responses is undefined

```_callee$/<@https://mydomainname/bundles/commons.bundle.js:3:1085597
_callee$@https://mydomainname/bundles/commons.bundle.js:3:1085535
tryCatch@https://mydomainname/bundles/vendors.bundle.js:36:138901
invoke@https://mydomainname/bundles/vendors.bundle.js:36:142786
defineIteratorMethods/https://mydomainname/bundles/vendors.bundle.js:36:140022
step@https://mydomainname/bundles/commons.bundle.js:3:1081241
step/<@https://mydomainname/bundles/commons.bundle.js:3:1081375
run@https://mydomainname/bundles/vendors.bundle.js:36:107333
notify/<@https://mydomainname/bundles/vendors.bundle.js:36:107573
flush@https://mydomainname/bundles/vendors.bundle.js:153:55572


the debugger console is a bit more verbose:

```vendors.bundle.js:197:155519
Possibly unhandled rejection: {}
TypeError: responses is undefined
Stack trace:
_callee$/<@https://mydomainname/bundles/commons.bundle.js:3:1085597
_callee$@https://mydomainname/bundles/commons.bundle.js:3:1085535
tryCatch@https://mydomainname/bundles/vendors.bundle.js:36:138901
invoke@https://mydomainname/bundles/vendors.bundle.js:36:142786
defineIteratorMethods/</prototype[method]@https://mydomainname/bundles/vendors.bundle.js:36:140022
step@https://mydomainname/bundles/commons.bundle.js:3:1081241
step/<@https://mydomainname/bundles/commons.bundle.js:3:1081375
run@https://mydomainname/bundles/vendors.bundle.js:36:107333
notify/<@https://mydomainname/bundles/vendors.bundle.js:36:107573
flush@https://mydomainname/bundles/vendors.bundle.js:153:55572

carolineLe on 5 Sep 2018

For us, the error appears randomly but very often.

dav3860 on 5 Sep 2018

@cjcenizal I'm wondering if this is a side effect of the courier refactor, any chance you could take a look?

Bargs on 10 Sep 2018

@dav3860 @carolineLe Could you look under the network tab in your browser's dev tools and tell me what the response is from the call to _msearch? It should be the most recent call in the network activity log.

If the response is too large to easily share or if you want to keep your data confidential then you can just share the first part of the response with me. It should look something like this:

{"responses":[{"took":132,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":81,"max_score":null

cjcenizal on 11 Sep 2018

@dav3860 @carolineLe Could you also share the request payload with me? That would help a lot.

I'm able to reproduce the error you've found, but I'm having trouble identifying the root cause. So far the information I have points to the problem lying within the response we get back from Elasticsearch, but more details on the particular requests you're making will help me.

cjcenizal on 11 Sep 2018

Here is the request that is sent:
{"index":"index_name-*","ignore_unavailable":true,"timeout":30000,"preference":1536647765711}
{"version":true,"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"_source":{"excludes":[]},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"30s","time_zone":"Europe/Berlin","min_doc_count":1}}},"stored_fields":["*"],"script_fields":{},"docvalue_fields":["@timestamp", "OtherTimeField1", "OtherTimeField2", "OtherTimeField3", "OtherTimeField4"],"query":{"bool":{"must":[{"match_all":{}},{"range":{"@timestamp":{"gte":1536646868523,"lte":1536647768524,"format":"epoch_millis"}}}],"filter":[],"should":[],"must_not":[]}},"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"*":{}},"fragment_size":2147483647}}

And the response associated:

{"responses":[{"took":414,"timed_out":false,"_shards":{"total":100,"successful":100,"skipped":90,"failed":0},
"hits":{"total":450519,"max_score":null,"hits":[...]},
"aggregations":{"2":{"buckets":[{"key_as_string":"2018-09-11T08:21:00.000+02:00","key":1536646860000,"doc_count":7342},{"key_as_string":"2018-09-11T08:21:30.000+02:00","key":1536646890000,"doc_count":14881},{"key_as_string":"2018-09-11T08:22:00.000+02:00","key":1536646920000,"doc_count":12212}, ... ]}},"status":200}]}

I had the bug with another index pattern and noticed the following warning: "Doc-value field [@timestamp] is not using a format.". Setting the format for the timestamps fields in this index "fixed" the problem, but it did not change anything for the one I'm having trouble with right now.

carolineLe on 11 Sep 2018

@carolineLe Thank you for this info! For the response you shared, could you provide the entire shape of the object? For example, the response example I gave had a responses property as well.

cjcenizal on 11 Sep 2018

Sorry, the formatting removed the first line, I edited my answer.

carolineLe on 12 Sep 2018

👍1

@dav3860 When you say "randomly but very often" can you confirm for me that you mean that you're using the same query, which sometimes yields the error and sometimes doesn't? E.g. you have Discover open with "Absolute" time range selected (to ensure the time doesn't change), and you hit refresh 10 times in your browser, and out of those 10 times 2 or 3 will fail?

cjcenizal on 12 Sep 2018

@carolineLe I'm able to reproduce the original error, but your error Courier fetch: responses is undefined is turning out to be trickier to reproduce. Can you click on the top line in the stack trace and show me which line of code is generating that error? Thanks.

cjcenizal on 19 Sep 2018

In terms of the original error, "Courier fetch: Cannot read property 'forEach' of undefined, this is causing a fatal error for the same reason that https://github.com/elastic/kibana/issues/22466 originally did. 6.4.0 contains a regression in which errors thrown by CallClient result in fatal errors, as in the case when a call to the es client results in a Promise rejection. This has been fixed in 6.4.1 (https://github.com/elastic/kibana/pull/22558).

Note that this just fixes the regression, which is an exacerbated symptom of an underlying error. The underlying error "Courier fetch: Cannot read property 'forEach' of undefined is due to a failed request sent through the es client, which I can't diagnose with the information in this issue.

cjcenizal on 19 Sep 2018

columnNumber: 1085597
fileName: "https://mydomainname/bundles/commons.bundle.js"
lineNumber: 3
message: "responses is undefined"

Right before this error, I get another weird error. The "XML" appears malformed but there is no XML involved, the response is in JSON.

Erreur d’analyse XML : mal formé
Emplacement : https://mydomainname/elasticsearch/_msearch
Numéro de ligne 1, Colonne 1 :
_msearch:1:1
_callee$/<@https://mydomainname/bundles/commons.bundle.js:3:1085597
_callee$@https://mydomainname/bundles/commons.bundle.js:3:1085535
tryCatch@https://mydomainname/bundles/vendors.bundle.js:36:138901
invoke@https://mydomainname/bundles/vendors.bundle.js:36:142786
defineIteratorMethods/</prototype[method]@https://mydomainname/bundles/vendors.bundle.js:36:140022
step@https://mydomainname/bundles/commons.bundle.js:3:1081241
step/<@https://mydomainname/bundles/commons.bundle.js:3:1081375
run@https://mydomainname/bundles/vendors.bundle.js:36:107333
notify/<@https://mydomainname/bundles/vendors.bundle.js:36:107573
flush@https://mydomainname/bundles/vendors.bundle.js:153:55572

Thank you for investigating this !

carolineLe on 19 Sep 2018

@cjcenizal OK, I did change the response from the server and added the "Content-type: application/json" and it fixed my problem.
Should I report it to the elasticsearch team ?

carolineLe on 19 Sep 2018

Hi @carolineLe thank you for the information. That sounds like an appropriate issue for the elasticsearch-js repo.

In terms of hunting down the origin of that error within our source, I'm still wondering if you can help me do that? Could you refer to the animation below and click on the second link in the commons.bundle.js stack trace in the dev tools console, to identify the line of code which originates the error? Thank you so much!

call_stack

cjcenizal on 19 Sep 2018

6.4.1 parity with 6.3.0

I've done some investigation to identify whether 6.4.1 is on par with 6.3.0 in terms of how errors within CallClient are surfaced.

Long-story short, I believe 6.4.1 is on par with 6.3.0, if not slightly above par. Here are my findings.

6.3.0

Throwing an error in `getFetchParams`

This results in a Notifier error.

throw_error

`msearch` returning a rejected promise

This results in a Notifier error.

throw_error

`msearch` resolving with `undefined`

This results in a different Notifier error: “Cannot read property responses of undefined”.

resolve_with_undefined

`msearch` resolving with an empty object

This results in a fatal error.

resolve_with_empty_object

6.4.1

Throwing an error in `getFetchParams`

This results in a Notifier error.

throw_error

`msearch` returning a rejected promise

This results in a Notifier error.

throw_error_msearch

`msearch` resolving with `undefined`

This results in a different Notifier error: “Cannot read property responses of undefined”.

resolve_with_undefined

`msearch` resolving with an empty object

This results in a different Notifier error: “Cannot read property forEach of undefined”.

resolve_with_object

cjcenizal on 19 Sep 2018

Here is where the exception occurs, at the "responses.forEach":

case 12:
  segregatedResponses = _context2.sent; responsesInOriginalRequestOrder = new Array(searchRequestsAndStatuses.length); segregatedResponses.forEach(function (r\
esponses, strategyIndex) {
    responses.forEach(function (response, responseIndex) {
      var searchRequest = searchStrategiesWithRequests[strategyIndex].searchRequests[responseIndex];
      var requestIndex = searchRequestsAndStatuses.indexOf(searchRequest);
      responsesInOriginalRequestOrder[requestIndex] = response
    })

carolineLe on 20 Sep 2018

Thank you @carolineLe! This is very helpful.

cjcenizal on 20 Sep 2018

Looks like the two error messages we've been seeing (Cannot read property 'forEach' of undefined and responses is undefined) are the same error, just surfaced in different ways depending on whether you're using Chrome or Firefox. They're both due to the msearch request resolving with an empty object (for some reason).

cjcenizal on 20 Sep 2018

@cjcenizal Maybe this is due to the JSON wrongly parsed as XML/HTML because the mimetype header is missing, what do you think ? I don't know if the missing mimetype is specific to my installation.

carolineLe on 21 Sep 2018

Looks like the two error messages we've been seeing (Cannot read property 'forEach' of undefined and responses is undefined) are the same error, just surfaced in different ways depending on whether you're using Chrome or Firefox. They're both due to the msearch request resolving with an empty object (for some reason).

see https://github.com/elastic/kibana/issues/22355#issuecomment-416004716

florian-asche on 23 Sep 2018

👍1

Thank you everyone for your help with this issue. I'm going to close it since it's not a problem we can solve on the Kibana side. Feel free to continue leaving questions here if you have any.

cjcenizal on 24 Sep 2018

👍2

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Kibana still spamming logs when different version from Elasticsearch

ctindel · 3Comments

Visualize: Authorization Exception Error when loading dashboard with "too many" visualizations

mark54g · 3Comments

Upgrade Assistant - Scripting settings - from 5.6 to 6.x

LukeMathWalker · 3Comments

Kibana 4 - Courier Fetch: X of 5 shards failed

Ginja · 3Comments

Integrate Subqueries in toolbar

treussart · 3Comments

Kibana: Fatal error in discover app after upgrade from 6.3.2 to 6.4.0

Most helpful comment

All 26 comments

6.4.1 parity with 6.3.0

6.3.0

Throwing an error in getFetchParams

msearch returning a rejected promise

msearch resolving with undefined

msearch resolving with an empty object

6.4.1

Throwing an error in getFetchParams

msearch returning a rejected promise

msearch resolving with undefined

msearch resolving with an empty object

Related issues

Throwing an error in `getFetchParams`

`msearch` returning a rejected promise

`msearch` resolving with `undefined`

`msearch` resolving with an empty object

Throwing an error in `getFetchParams`

`msearch` returning a rejected promise

`msearch` resolving with `undefined`

`msearch` resolving with an empty object