Kibana: No handler for type [flattened] declared on field [state]

[designermonkey]

Kibana version: 7.5.0

Elasticsearch version: 7.5.0

Server OS version: Ubuntu 18.04

Browser version: n/a

Browser OS version: n/a

Original install method (e.g. download page, yum, from source, etc.): Using elastic apt repo and then apt install

Describe the bug: Continuous Kibana server is not ready yet in browser led me to journald logs, where the below output cycles over and over again as systemd tries to start kibana.

I have verified that ES is contactable, as the .kibana_task_manager_1 index is created perfectly fine.

It can't create the mapping for .kibana_1. After a whole day of trying to find out what this is, I was led to understand that _doc is deprecated in ES 7. I don't know enough about this stuff, but this seems to be a big problem, no?

I cannot get Kibana to install and migrate.

Steps to reproduce:

1. Just install it

Expected behavior:

Dec 05 17:37:50 kibana001-lon1-ops kibana[947]:  FATAL  [mapper_parsing_exception] No handler for type [flattened] declared on field [state] :: {"path":"/.kibana_1","query":{},"body":"{\"mappings\":{\"dynamic\":\"strict\",\"properties\":{\"config\":{\"dynamic\":\"true\",\"properties\":{\"buildNum\":{\"type\":\"keyword\"}}},\"migrationVersion\":{\"dynamic\":\"true\",\"type\":\"object\"},\"type\":{\"type\":\"keyword\"},\"namespace\":{\"type\":\"keyword\"},\"updated_at\":{\"type\":\"date\"},\"references\":{\"type\":\"nested\",\"properties\":{\"name\":{\"type\":\"keyword\"},\"type\":{\"type\":\"keyword\"},\"id\":{\"type\":\"keyword\"}}},\"graph-workspace\":{\"properties\":{\"description\":{\"type\":\"text\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"numLinks\":{\"type\":\"integer\"},\"numVertices\":{\"type\":\"integer\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"},\"wsState\":{\"type\":\"text\"}}},\"space\":{\"properties\":{\"name\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":2048}}},\"description\":{\"type\":\"text\"},\"initials\":{\"type\":\"keyword\"},\"color\":{\"type\":\"keyword\"},\"disabledFeatures\":{\"type\":\"keyword\"},\"imageUrl\":{\"type\":\"text\",\"index\":false},\"_reserved\":{\"type\":\"boolean\"}}},\"ml-telemetry\":{\"properties\":{\"file_data_visualizer\":{\"properties\":{\"index_creation_count\":{\"type\":\"long\"}}}}},\"apm-telemetry\":{\"properties\":{\"has_any_services\":{\"type\":\"boolean\"},\"services_per_agent\":{\"properties\":{\"python\":{\"type\":\"long\",\"null_value\":0},\"java\":{\"type\":\"long\",\"null_value\":0},\"nodejs\":{\"type\":\"long\",\"null_value\":0},\"js-base\":{\"type\":\"long\",\"null_value\":0},\"rum-js\":{\"type\":\"long\",\"null_value\":0},\"dotnet\":{\"type\":\"long\",\"null_value\":0},\"ruby\":{\"type\":\"long\",\"null_value\":0},\"go\":{\"type\":\"long\",\"null_value\":0}}}}},\"map\":{\"properties\":{\"description\":{\"type\":\"text\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"},\"bounds\":{\"type\":\"geo_shape\"},\"mapStateJSON\":{\"type\":\"text\"},\"layerListJSON\":{\"type\":\"text\"},\"uiStateJSON\":{\"type\":\"text\"}}},\"maps-telemetry\":{\"properties\":{\"mapsTotalCount\":{\"type\":\"long\"},\"timeCaptured\":{\"type\":\"date\"},\"attributesPerMap\":{\"properties\":{\"dataSourcesCount\":{\"properties\":{\"min\":{\"type\":\"long\"},\"max\":{\"type\":\"long\"},\"avg\":{\"type\":\"long\"}}},\"layersCount\":{\"properties\":{\"min\":{\"type\":\"long\"},\"max\":{\"type\":\"long\"},\"avg\":{\"type\":\"long\"}}},\"layerTypesCount\":{\"dynamic\":\"true\",\"properties\":{}},\"emsVectorLayersCount\":{\"dynamic\":\"true\",\"properties\":{}}}}}},\"canvas-workpad\":{\"dynamic\":false,\"properties\":{\"name\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\"}}},\"@timestamp\":{\"type\":\"date\"},\"@created\":{\"type\":\"date\"}}},\"canvas-element\":{\"dynamic\":false,\"properties\":{\"name\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\"}}},\"help\":{\"type\":\"text\"},\"content\":{\"type\":\"text\"},\"image\":{\"type\":\"text\"},\"@timestamp\":{\"type\":\"date\"},\"@created\":{\"type\":\"date\"}}},\"infrastructure-ui-source\":{\"properties\":{\"name\":{\"type\":\"text\"},\"description\":{\"type\":\"text\"},\"metricAlias\":{\"type\":\"keyword\"},\"logAlias\":{\"type\":\"keyword\"},\"fields\":{\"properties\":{\"container\":{\"type\":\"keyword\"},\"host\":{\"type\":\"keyword\"},\"pod\":{\"type\":\"keyword\"},\"tiebreaker\":{\"type\":\"keyword\"},\"timestamp\":{\"type\":\"keyword\"}}},\"logColumns\":{\"type\":\"nested\",\"properties\":{\"timestampColumn\":{\"properties\":{\"id\":{\"type\":\"keyword\"}}},\"messageColumn\":{\"properties\":{\"id\":{\"type\":\"keyword\"}}},\"fieldColumn\":{\"properties\":{\"id\":{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"}}}}}}},\"metrics-explorer-view\":{\"properties\":{\"name\":{\"type\":\"keyword\"},\"options\":{\"properties\":{\"metrics\":{\"type\":\"nested\",\"properties\":{\"aggregation\":{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"},\"color\":{\"type\":\"keyword\"},\"label\":{\"type\":\"keyword\"}}},\"limit\":{\"type\":\"integer\"},\"groupBy\":{\"type\":\"keyword\"},\"filterQuery\":{\"type\":\"keyword\"},\"aggregation\":{\"type\":\"keyword\"}}},\"chartOptions\":{\"properties\":{\"type\":{\"type\":\"keyword\"},\"yAxisMode\":{\"type\":\"keyword\"},\"stack\":{\"type\":\"boolean\"}}},\"currentTimerange\":{\"properties\":{\"from\":{\"type\":\"keyword\"},\"to\":{\"type\":\"keyword\"},\"interval\":{\"type\":\"keyword\"}}}}},\"inventory-view\":{\"properties\":{\"name\":{\"type\":\"keyword\"},\"metric\":{\"properties\":{\"type\":{\"type\":\"keyword\"}}},\"groupBy\":{\"type\":\"nested\",\"properties\":{\"label\":{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"}}},\"nodeType\":{\"type\":\"keyword\"},\"view\":{\"type\":\"keyword\"},\"customOptions\":{\"type\":\"nested\",\"properties\":{\"text\":{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"}}},\"boundsOverride\":{\"properties\":{\"max\":{\"type\":\"integer\"},\"min\":{\"type\":\"integer\"}}},\"autoBounds\":{\"type\":\"boolean\"},\"time\":{\"type\":\"integer\"},\"autoReload\":{\"type\":\"boolean\"},\"filterQuery\":{\"properties\":{\"kind\":{\"type\":\"keyword\"},\"expression\":{\"type\":\"keyword\"}}}}},\"siem-ui-timeline\":{\"properties\":{\"columns\":{\"properties\":{\"aggregatable\":{\"type\":\"boolean\"},\"category\":{\"type\":\"keyword\"},\"columnHeaderType\":{\"type\":\"keyword\"},\"description\":{\"type\":\"text\"},\"example\":{\"type\":\"text\"},\"indexes\":{\"type\":\"keyword\"},\"id\":{\"type\":\"keyword\"},\"name\":{\"type\":\"text\"},\"placeholder\":{\"type\":\"text\"},\"searchable\":{\"type\":\"boolean\"},\"type\":{\"type\":\"keyword\"}}},\"dataProviders\":{\"properties\":{\"id\":{\"type\":\"keyword\"},\"name\":{\"type\":\"text\"},\"enabled\":{\"type\":\"boolean\"},\"excluded\":{\"type\":\"boolean\"},\"kqlQuery\":{\"type\":\"text\"},\"queryMatch\":{\"properties\":{\"field\":{\"type\":\"text\"},\"displayField\":{\"type\":\"text\"},\"value\":{\"type\":\"text\"},\"displayValue\":{\"type\":\"text\"},\"operator\":{\"type\":\"text\"}}},\"and\":{\"properties\":{\"id\":{\"type\":\"keyword\"},\"name\":{\"type\":\"text\"},\"enabled\":{\"type\":\"boolean\"},\"excluded\":{\"type\":\"boolean\"},\"kqlQuery\":{\"type\":\"text\"},\"queryMatch\":{\"properties\":{\"field\":{\"type\":\"text\"},\"displayField\":{\"type\":\"text\"},\"value\":{\"type\":\"text\"},\"displayValue\":{\"type\":\"text\"},\"operator\":{\"type\":\"text\"}}}}}}},\"description\":{\"type\":\"text\"},\"favorite\":{\"properties\":{\"keySearch\":{\"type\":\"text\"},\"fullName\":{\"type\":\"text\"},\"userName\":{\"type\":\"text\"},\"favoriteDate\":{\"type\":\"date\"}}},\"kqlMode\":{\"type\":\"keyword\"},\"kqlQuery\":{\"properties\":{\"filterQuery\":{\"properties\":{\"kuery\":{\"properties\":{\"kind\":{\"type\":\"keyword\"},\"expression\":{\"type\":\"text\"}}},\"serializedQuery\":{\"type\":\"text\"}}}}},\"title\":{\"type\":\"text\"},\"dateRange\":{\"properties\":{\"start\":{\"type\":\"date\"},\"end\":{\"type\":\"date\"}}},\"sort\":{\"properties\":{\"columnId\":{\"type\":\"keyword\"},\"sortDirection\":{\"type\":\"keyword\"}}},\"created\":{\"type\":\"date\"},\"createdBy\":{\"type\":\"text\"},\"updated\":{\"type\":\"date\"},\"updatedBy\":{\"type\":\"text\"}}},\"siem-ui-timeline-note\":{\"properties\":{\"timelineId\":{\"type\":\"keyword\"},\"eventId\":{\"type\":\"keyword\"},\"note\":{\"type\":\"text\"},\"created\":{\"type\":\"date\"},\"createdBy\":{\"type\":\"text\"},\"updated\":{\"type\":\"date\"},\"updatedBy\":{\"type\":\"text\"}}},\"siem-ui-timeline-pinned-event\":{\"properties\":{\"timelineId\":{\"type\":\"keyword\"},\"eventId\":{\"type\":\"keyword\"},\"created\":{\"type\":\"date\"},\"createdBy\":{\"type\":\"text\"},\"updated\":{\"type\":\"date\"},\"updatedBy\":{\"type\":\"text\"}}},\"upgrade-assistant-reindex-operation\":{\"dynamic\":true,\"properties\":{\"indexName\":{\"type\":\"keyword\"},\"status\":{\"type\":\"integer\"}}},\"upgrade-assistant-telemetry\":{\"properties\":{\"ui_open\":{\"properties\":{\"overview\":{\"type\":\"long\",\"null_value\":0},\"cluster\":{\"type\":\"long\",\"null_value\":0},\"indices\":{\"type\":\"long\",\"null_value\":0}}},\"ui_reindex\":{\"properties\":{\"close\":{\"type\":\"long\",\"null_value\":0},\"open\":{\"type\":\"long\",\"null_value\":0},\"start\":{\"type\":\"long\",\"null_value\":0},\"stop\":{\"type\":\"long\",\"null_value\":0}}},\"features\":{\"properties\":{\"deprecation_logging\":{\"properties\":{\"enabled\":{\"type\":\"boolean\",\"null_value\":true}}}}}}},\"file-upload-telemetry\":{\"properties\":{\"filesUploadedTotalCount\":{\"type\":\"long\"}}},\"lens\":{\"properties\":{\"title\":{\"type\":\"text\"},\"visualizationType\":{\"type\":\"keyword\"},\"state\":{\"type\":\"flattened\"},\"expression\":{\"index\":false,\"type\":\"keyword\"}}},\"lens-ui-telemetry\":{\"properties\":{\"name\":{\"type\":\"keyword\"},\"type\":{\"type\":\"keyword\"},\"date\":{\"type\":\"date\"},\"count\":{\"type\":\"integer\"}}},\"query\":{\"properties\":{\"title\":{\"type\":\"text\"},\"description\":{\"type\":\"text\"},\"query\":{\"properties\":{\"language\":{\"type\":\"keyword\"},\"query\":{\"type\":\"keyword\",\"index\":false}}},\"filters\":{\"type\":\"object\",\"enabled\":false},\"timefilter\":{\"type\":\"object\",\"enabled\":false}}},\"index-pattern\":{\"properties\":{\"fieldFormatMap\":{\"type\":\"text\"},\"fields\":{\"type\":\"text\"},\"intervalName\":{\"type\":\"keyword\"},\"notExpandable\":{\"type\":\"boolean\"},\"sourceFilters\":{\"type\":\"text\"},\"timeFieldName\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"type\":{\"type\":\"keyword\"},\"typeMeta\":{\"type\":\"keyword\"}}},\"visualization\":{\"properties\":{\"description\":{\"type\":\"text\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"savedSearchRefName\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"uiStateJSON\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"},\"visState\":{\"type\":\"text\"}}},\"search\":{\"properties\":{\"columns\":{\"type\":\"keyword\"},\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"sort\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"dashboard\":{\"properties\":{\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"optionsJSON\":{\"type\":\"text\"},\"panelsJSON\":{\"type\":\"text\"},\"refreshInterval\":{\"properties\":{\"display\":{\"type\":\"keyword\"},\"pause\":{\"type\":\"boolean\"},\"section\":{\"type\":\"integer\"},\"value\":{\"type\":\"integer\"}}},\"timeFrom\":{\"type\":\"keyword\"},\"timeRestore\":{\"type\":\"boolean\"},\"timeTo\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"url\":{\"properties\":{\"accessCount\":{\"type\":\"long\"},\"accessDate\":{\"type\":\"date\"},\"createDate\":{\"type\":\"date\"},\"url\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":2048}}}}},\"server\":{\"properties\":{\"uuid\":{\"type\":\"keyword\"}}},\"kql-telemetry\":{\"properties\":{\"optInCount\":{\"type\":\"long\"},\"optOutCount\":{\"type\":\"long\"}}},\"sample-data-telemetry\":{\"properties\":{\"installCount\":{\"type\":\"long\"},\"unInstallCount\":{\"type\":\"long\"}}},\"telemetry\":{\"properties\":{\"enabled\":{\"type\":\"boolean\"},\"sendUsageFrom\":{\"ignore_above\":256,\"type\":\"keyword\"},\"lastReported\":{\"type\":\"date\"},\"lastVersionChecked\":{\"ignore_above\":256,\"type\":\"keyword\"},\"userHasSeenNotice\":{\"type\":\"boolean\"}}},\"timelion-sheet\":{\"properties\":{\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"timelion_chart_height\":{\"type\":\"integer\"},\"timelion_columns\":{\"type\":\"integer\"},\"timelion_interval\":{\"type\":\"keyword\"},\"timelion_other_interval\":{\"type\":\"keyword\"},\"timelion_rows\":{\"type\":\"integer\"},\"timelion_sheet\":{\"type\":\"text\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"ui-metric\":{\"properties\":{\"count\":{\"type\":\"integer\"}}}},\"_meta\":{\"migrationMappingPropertyHashes\":{\"config\":\"87aca8fdb053154f11383fce3dbf3edf\",\"migrationVersion\":\"4a1746014a75ade3a714e1db5763276f\",\"type\":\"2f4316de49999235636386fe51dc06c1\",\"namespace\":\"2f4316de49999235636386fe51dc06c1\",\"updated_at\":\"00da57df13e94e9d98437d13ace4bfe0\",\"references\":\"7997cf5a56cc02bdc9c93361bde732b0\",\"graph-workspace\":\"cd7ba1330e6682e9cc00b78850874be1\",\"space\":\"c5ca8acafa0beaa4d08d014a97b6bc6b\",\"ml-telemetry\":\"257fd1d4b4fdbb9cb4b8a3b27da201e9\",\"apm-telemetry\":\"07ee1939fa4302c62ddc052ec03fed90\",\"map\":\"23d7aa4a720d4938ccde3983f87bd58d\",\"maps-telemetry\":\"a4229f8b16a6820c6d724b7e0c1f729d\",\"canvas-workpad\":\"b0a1706d356228dbdcb4a17e6b9eb231\",\"canvas-element\":\"7390014e1091044523666d97247392fc\",\"infrastructure-ui-source\":\"ddc0ecb18383f6b26101a2fadb2dab0c\",\"metrics-explorer-view\":\"53c5365793677328df0ccb6138bf3cdd\",\"inventory-view\":\"84b320fd67209906333ffce261128462\",\"siem-ui-timeline\":\"1f6f0860ad7bc0dba3e42467ca40470d\",\"siem-ui-timeline-note\":\"8874706eedc49059d4cf0f5094559084\",\"siem-ui-timeline-pinned-event\":\"20638091112f0e14f0e443d512301c29\",\"upgrade-assistant-reindex-operation\":\"a53a20fe086b72c9a86da3cc12dad8a6\",\"upgrade-assistant-telemetry\":\"56702cec857e0a9dacfb696655b4ff7b\",\"file-upload-telemetry\":\"0ed4d3e1983d1217a30982630897092e\",\"lens\":\"21c3ea0763beb1ecb0162529706b88c5\",\"lens-ui-telemetry\":\"509bfa5978586998e05f9e303c07a327\",\"query\":\"11aaeb7f5f7fa5bb43f25e18ce26e7d9\",\"index-pattern\":\"66eccb05066c5a89924f48a9e9736499\",\"visualization\":\"52d7a13ad68a150c4525b292d23e12cc\",\"search\":\"181661168bbadd1eff5902361e2a0d5c\",\"dashboard\":\"d00f614b29a80360e1190193fd333bab\",\"url\":\"c7f66a0df8b1b52f17c28c4adb111105\",\"server\":\"ec97f1c5da1a19609a60874e5af1100c\",\"kql-telemetry\":\"d12a98a6f19a2d273696597547e064ee\",\"sample-data-telemetry\":\"7d3cfeb915303c9641c59681967ffeb4\",\"telemetry\":\"358ffaa88ba34a97d55af0933a117de4\",\"timelion-sheet\":\"9a2a2748877c7a7b582fef201ab1d4cf\",\"ui-metric\":\"0d409297dc5ebe1e3a1da691c6ee32e3\"}}},\"settings\":{\"number_of_shards\":1,\"auto_expand_replicas\":\"0-1\"}}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [state]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}},\"status\":400}"}

[timroes]

@designermonkey This Exception you're getting looks a bit like you're trying to run a "Basic" Kibana version against an Open Source Elasticsearch cluster. Could you please verify, that you're running not an OSS Elasticsearch, by going to https://<your-es-host>:9200/ and paste the JSON you'll be seeing here?

[designermonkey]

{
  "name" : "elasticsearch001-lon1-ops",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "NVI9gbnmSW6TIz3EI9GdRg",
  "version" : {
    "number" : "7.4.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "fc0eeb6e2c25915d63d871d344e3d0b45ea0ea1e",
    "build_date" : "2019-10-22T17:16:35.176724Z",
    "build_snapshot" : false,
    "lucene_version" : "8.2.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

I don't understand the difference tbh. I am installing all the parts of ELK via the apt installers and they don't work together.

[designermonkey]

I've downgraded from 7.5.0 to 7.4.1 on all components too.

[timroes]

@designermonkey Yeah, both should be the same "default" distribution, and it also is for your case, just wanted to be 100% sure, because the error message looked a bit like you were running the pure Apache-2.0 licensed version, which does not support the flattened field type.

Just to understand your downgrade process. Did you run 7.5 first then it didn't work, you downgraded all the components to 7.4.1, and tried to run again? Also is that error message you posted from the 7.4.1 or 7.5.0 state?

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[designermonkey]

@timroes nah, I use ansible and digitalocean so destroyed the server completely. I'll need to fully retest whether the error persists with 7.4.1, but the elasticsearch output is definitely not different (apart from the version number obviously).

Out of interest, I cannot understand why the Oss version vs default version has different support for field types, when I have read that the only difference should be xpack inclusion? I find all this very odd indeed.

[timroes]

Some fieldttypes (like flattened are part of x-pack). With 7.4.1 you should not have the problem at all, since the mapping that caused your issue was introduced by Lens in 7.5.0, so there should be no errors in 7.4.1 at all.

[designermonkey]

Ok, I think I'll stick to 7.4.1 then. There may have been a mix up of active/inactive xpack. I don't want to use xpack features and am struggling to figure out how to deactivate it.

Thanks for the feedback.

[timroes]

Yes, that I assume was then the reason. You were most likely running an x-pack enabled Kibana with an x-pack disabled Elasticsearch cluster, which would reveal that error.

You can also use a slightly different APT repository to download only the Apache 2 licensed code. You'll find instructions into the installation guide. If you're looking through it there's always the alternative repositories and URLs for the Apache 2 licensed OSS build in each step.

[ttpro1995]

so, how do you fix this problem ?

[lpanichi]

I ran into this error while running elasticsearch and kibana with docker.
I changed the images
from docker.elastic.co/elasticsearch/elasticsearch-oss:7.5.1 to elasticsearch:7.5.1
and from docker.elastic.co/kibana/kibana-oss:7.5.1 to kibana:7.5.1
Then I delete all the indexes related to kibana in ES and restarted kibana. Everything is fine now.

[designermonkey]

I have always been in favour of the elastic stack and have built a few websites around it, but after a year or two away from usage, to come back to find things like this has made me quite distrusting of the stack.

x-pack is a pay for feature and seems to be turned on by default, with imo not enough detail for developers on how to _not_ include it or to deactivate it. My team and I have spent far too long trying to figure out what the issues were around this for us to have a favourable view of the elastic stack any more, which is sad for me.

The documentation mentioned above states:

This package is free to use under the Elastic license. It contains open source and free commercial features and access to paid commercial features. Start a 30-day trial to try out all of the paid commercial features. See the Subscriptions page for information about Elastic license levels.

As far as I have seen, the paid for features are on by default and nothing on this page explains how to deactivate them. Yes there is a link to a subscription model but if I'm looking at OSS, then reading subscription information is the last thing on my mind. That statement above does not explain that I have to apply a license or apply a subscription.

My overall comment is: x-pack should be disabled by default as it is not part of the OSS package. If a user then decides to use it, they should explicitly activate it and apply a license model. The issue I had above was that there is no way I can find to turn it off in Kibana!

I'm not alluding to this being done intentionally to push sales of features, but this does seem like a poor methodology of providing OSS software.

Also, I am open to correction if I have clearly missed an obvious explanation of the features and their configuration.

[timroes]

Hi John,

sorry for the confusion here, let me try to provide a hopefully helpful explanation here.

If you download any part of the Elastic Stack via the homepage or docker, (e.g. the kibana-7.6.0 or elasticsearch-7.6.0) image, you'll always get the so called "Basic" version, which contains the Apache-2.0 licensed plus x-pack. Part of x-pack is free (like Maps, Lens, Authentication, etc.) and not requiring any license or registration, and that is also the only part enabled in that downloaded version. You should not see any paid features enabled by default. From the history in this issue, it also doesn't look like any paid feature was enabled for you after installation (and if so, we should investigate that further).

If you're interested in a pure Apache-2.0 licensed version (without Maps, Lens, flattened field type, etc.) there is an OSS image provided for every part of the stack. You'll easily detect that, because it always has -oss in its name. For download you'll e.g. find them here (there's a link on every download page below the packages, pointing to the pure Apache-2.0 download) or via docker you can expand every docker image there to see the default and the pure OSS version.

You must always make sure you use OSS or default version for every part of the stack, so running a Basic Kibana against an OSS Elasticsearch won't work. You can also find a detailed difference between the OSS, the Basic and the paid features on the Subscription page.

I hope this helped clarify a bit on the differences in the versions. Please feel free to reach out for any further questions.

Cheers,
Tim

[designermonkey]

Thanks for the response Tim.

[ebuildy]

Kibana should check elasticsearch server have the right licence , and print a clear message.

[hyprstack]

For those of us using elasticsearch7.4 on AWS and that need the flattened type for a field, how do we work around this? I am unable to set field type to flattened for my mapping.

{
  "name" : "xxxxxxxxx",
  "cluster_name" : "xxxxxxxx",
  "cluster_uuid" : "xxxxxxxx",
  "version" : {
    "number" : "7.4.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2020-05-05T04:47:06.936807Z",
    "build_snapshot" : false,
    "lucene_version" : "8.2.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[wylieconlon]

@hyprstack The flattened field is available in the free Basic distribution that Elastic offers, but is not available in Amazon Elasticsearch. You could switch to purchasing Elastic instances from the Amazon marketplace, or from https://elastic.co directly.

[hyprstack]

Thanks :) How easily would this integrate with existing AWS infrastructures?
Also before I recommend any purchase of extra services to my employer, could having a single level list (array) also lead to memory explosion?
It seems to be stored as a text data type.

[wylieconlon]

@hyprstack This issue is not the right place to ask questions about our products: there are multiple ways of getting started depending on your needs. I recommend you read about it on https://www.elastic.co/ to see which one is best for you.

[hyprstack]

Thanks, although I also asked a second question.

Could having a single level list (array) also lead to memory explosion?

[naisanzaa]

Fixed. elasticsearch-oss and kibana was the issue. Had to change it to kibana-oss

[mindscratch]

@naisanzaa I had to do the same. I'm running a local fork of kibana and would normally start it with yarn start, but when I switched to using OpenDistro for Elasticsearch I ran into problems until I started it with yarn start --oss (happens to be documented)