Kibana: Migration from Kibana 7.7.1 to 7.8 fails with : Document contains at least one immense term ... whose UTF8 encoding is longer than the max length 32766

[nicenemo]

Kibana version: 7.7.1 to 7.8

Elasticsearch version: 7.8

Server OS version: Debian 10

Original install method (e.g. download page, yum, from source, etc.):
apt install

Describe the bug:

When upgrading our stack fro 7.7.1 to 7.7.8 the Kibana index migration fails and Kibana refuses to continue starting up.

Upgrade Elasticsearch, Logstash just with apt update && apt upgrade
Then upgrade Kibana.

1. It moans about encryption keys and cross cluster migration in a basic license. I never configured the later. And I am not interested in cross cluster migration yet. I set the encryption keys. to silence that. With or withouth keys, it fails with:

Then it fails with:

kibana[12316]:  FATAL  Error: Document contains at least one immense term in field="url.url.keyword" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[47, 97, 112, 112, 47, 107, 105, 98, 97, 110, 97, 35, 47, 100, 105, 115, 99, 111, 118, 101, 114, 63, 95, 103, 61, 40, 102, 105, 108, 116]...', original message: bytes can be at most 32766 in length; got 34822

Expected behavior:

Upgrading succeeds.

Provide logs and/or server output (if relevant):

Any additional context:
3 nodes with elasticsearch, one runs kibana too, another runs logstash.

I manually reinstalled the 7.7.1 deb with dpkg again, That still runs against a 7.8 Elasticsearch,

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[pgayvallet]

@rudolf Do we have any way to find which SO type / migration is causing this error?

[rudolf]

@nicenemo Thanks for reporting this, it looks like a regression introduced by https://github.com/elastic/kibana/pull/64043 because the mappings for the url field of the url saved object type no longer includes the ignore_above: 2048.

[rudolf]

The best workaround is to remove all url objects, perform the upgrade, update the mappings, and then import all the url objects again:

1. Rollback to the last working version of Kibana
2. Take a backup snapshot of the .kibana* indices
3. Use the saved objects management UI of your outdated Kibana to export and then delete all url saved object types.
4. Upgrade to Kibana 7.8.0, this time the migration should succeed
5. After Kibana has started up, update the mappings with:
   curl -u elastic:changeme -X PUT "localhost:9200/.kibana/_mapping" -H 'Content-Type: application/json' --data-binary "@/payload.json"

{
    "properties": {
        "url": {
            "properties": {
                "url": {
                    "fields": {
                        "keyword": {
                            "ignore_above": 2048,
                            "type": "keyword"
                        }
                    },
                    "type": "text"
                }
            }
        }
    }
}

1. Import the url objects from step (1) using the saved objects management UI

You might just have one or two url's that are long enough to cause this problem in which case you can delete just those objects.
Find and then delete sharing short url's which link to very long url's:
curl -u elastic:changeme -X POST "localhost:9200/.kibana/_search?pretty" -H 'Content-Type: application/json' --data-binary "@./payload.json"

payload.json:

{"query": {"bool" : {"filter" : {"script" : {"script" : {"source" : "doc['url.url.keyword'].size() > 0 && doc['url.url.keyword'].value.length() > 8191"}}}}}}

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)