Kibana: [ML] Forever loading indicators in regression job list and results

[lcawl]

Kibana version:

7.9.0 (first build candidate)

Elasticsearch version:

7.9.0 (first build candidate)

Server OS version:

MacOS 10.15

Browser version:

Safari 13.1

Describe the bug:

If I create a regression job by following the Kibana-specific steps in this tutorial (https://www.elastic.co/guide/en/machine-learning/7.x/flightdata-regression.html), I do not receive evaluation results. Instead, I see the same "loading indicators" as mentioned in https://github.com/elastic/kibana/pull/69422. However, if I run the evaluate API from the console, it returns results successfully there.

I have also replicated this problem on master last week.

Steps to reproduce:

1. Install Elasticsearch and Kibana. Start a trial license and enable security (I have not tried recreating with security disabled, since I don't see any security-related errors).
2. Log into Kibana and add the "sample flights" data.
3. Create a regression job as described in https://www.elastic.co/guide/en/machine-learning/7.x/flightdata-regression.html. Accept defaults for all advanced options.
4. Wait for the job to complete.
5. Expand the job details in the "Data Frame Analytics" page and notice that there are no results provided for the "generalized mean squared error", "generalization mean squared logarithmic error", etc.
6. View the job results and notice that there are no numbers provided for those evaluation metrics there either.
7. Verify that the API works. For example:

POST _ml/data_frame/_evaluate { "index": "df-flight-delays", "query": { "bool": { "filter": [{ "term": { "ml.is_training": false } }] } }, "evaluation": { "regression": { "actual_field": "FlightDelayMin", "predicted_field": "ml.FlightDelayMin_prediction", "metrics": { "r_squared": {}, "mse": {}, "msle": {}, "huber": {} } } } }

Expected behavior:

I expected results as in previous releases (shown here): https://www.elastic.co/guide/en/machine-learning/7.x/flightdata-regression.html#flightdata-regression-results

Something equivalent to what's returned by the API. For example:

{ "regression" : { "huber" : { "value" : 34.58652313095805 }, "mse" : { "value" : 3864.093389072879 }, "msle" : { "value" : "NaN" }, "r_squared" : { "value" : 0.5665066497474046 } } }

Screenshots (if relevant):

Job creation:

Missing evaluation details from the job management page:

Missing evaluation details from the results page:

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[peteharverson]

I am able to reproduce this, and see the following errors in the browser console:

[droberts195]

@przemekwitek please could you investigate where the NaNs are coming from. Maybe we calculate log(0) at some point.

You can get the data set into an Elasticsearch you've built locally by also building and running Kibana locally using yarn kbn bootstrap; yarn start. Then when you've logged in go to "Add sample data" on the home page and add the "Sample flight data". Then you should be able to add extra debug to your locally built Elasticsearch and see what's going wrong.

[przemekwitek]

@przemekwitek please could you investigate where the NaNs are coming from. Maybe we calculate log(0) at some point.

@peteharverson: IIUC the issue with MSLE metric returning NaN will be solved in the UI layer (i.e. NaN will be treated as correct return value for the MSLE metric in the evaluation API). Is that correct?

[droberts195]

the issue with MSLE metric returning NaN will be solved in the UI layer (i.e. NaN will be treated as correct return value for the MSLE metric in the evaluation API). Is that correct?

@przemekwitek the UI side will be changed, but please can you investigate and check that the cause is that one of the predicted values was -1. If it was then it's true that no backend changes can really be made, and it's just that the loss function is incompatible with negative numbers. But there may be a bug somewhere instead that could affect usage on data sets where MSLE is a valid and sensible loss function to use.

[przemekwitek]

but please can you investigate and check that the cause is that one of the predicted values was -1

I'll do that and will posts the results here.

[przemekwitek]

I was able to reproduce the issue on the flights dataset. While the ground truth field value (FlightDelayMin) is never negative so the log function is well defined, the prediction field value (ml.FlightDelayMin_prediction) happens to be negative.
During MSLE calculation we apply the same offset to both actual (y) and predicted (y') but in the case of predicted the offset is not enough. log function is not defined for negative arguments so it yields NaN.

 * equation: msle = 1/n * Σ(log(y + offset) - log(y´ + offset))^2

I don't think the metric can return anything more sensible than NaN in this case so IMO we should allow it and handle correctly in the UI.

[droberts195]

I don't think the metric can return anything more sensible than NaN in this case so IMO we should allow it and handle correctly in the UI.

I agree. Thanks for investigating @przemekwitek.