Kibana: Index pattern creation API

[aruntomar]

It would be helpful if we could have an api to configure the initial "configure an Index pattern" page to be used to used by automation tools or scripts.

[rashidkpc]

Agreed, this would be nice to have. It would also be awesome to move the mapping parsing to the server side to improve the handling of large mappings.

[pg99bassist]

I third this. I've been banging on cURL requests to do this for the last 2 hours and I've got nothing. I will keep trying... but an initial Index Pattern creation method to run programatically from other languages would be extremely beneficial.

[scaph01]

yeah I too was faced with this, my solution (workaround) is similar to yours, I am writing directly to the .kibana index , essentially, in Ansible I will end up doing something like this in a post deploy task:

curl -XPUT http://<es node>:9200/.kibana/index-pattern/events-* -d '{"title" : "events-*",  "timeFieldName": "EventTime"}'
curl -XPUT http://<es node>:9200/.kibana/config/4.1.1 -d '{"defaultIndex" : "events-*"}'

I don't really want to be writing to that index but I also did not want to use the Kibana HTTP flows...So I am interested doing this in a more supported manner.

[madAndroid]

It would be great if this was automated in some fashion - we're using Elasticdump to achieve this at the moment, and it's a fairly good solution, but it requires the index to be created first 'by hand', exported, and then added to our puppet code.

[rashidkpc]

Closing in favor of #5199

[alainchiasson]

This was closed in favor of 5199, but 5199 was dropped - any chance of getting this back ?

[leesei]

Kibana 5 webpage is POSTing to

• http://localhost:5601/es_admin/.kibana/index-pattern/filebeat-*/_create' to create index
• http://localhost:5601/api/kibana/settings/defaultIndex to set default
  But it requires kbn-xsrf header.

I was able to add index to ES 5 with this POST

curl -XPOST -H 'Content-Type: application/json' \
  'http://localhost:9200/.kibana/index-pattern/filebeat-*' \
  -d'{"title":"filebeat-*","timeFieldName":"@timestamp","notExpandable":true}'

But I cannot figure out where the defaultIndex is stored.

POSTing to http://localhost:9200/.kibana/config/ no longer works. (It worked on 4.x).

[falkenbt]

You can set the kbn-xsrfheader to anything, it doesn't seem to get checked:

curl ${KIBANASERVER}/api/kibana/settings/defaultIndex \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/plain, */*" \
-H "kbn-xsrf: anything" -H "Connection: keep-alive" \
--data-binary ${YOURDEFAULTINDEX} -w "\n" --compressed 

[tedder]

Hmm, was hoping to do a GET against this in 5.x.

$ curl http://localhost:9200/api/kibana/settings/defaultIndex
No handler found for uri [/api/kibana/settings/defaultIndex] and method [GET]
$ curl http://localhost:9200/.kibana/config/
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No endpoint or operation is available at [config]"}],"type":"illegal_argument_exception","reason":"No endpoint or operation is available at [config]"},"status":400}

[falkenbt]

Looks like you fired your first GET at elastic (port 9200) and not at Kibana.

[hobti01]

For kibana 6.0 using curl and jq:

url="http://localhost:5601"
index_pattern="logstash-*"
time_field="@timestamp"
# Create index pattern and get the created id
# curl -f to fail on error
id=$(curl -f -XPOST -H "Content-Type: application/json" -H "kbn-xsrf: anything" \
  "$url/api/saved_objects/index-pattern" \
  -d"{\"attributes\":{\"title\":\"$index_pattern\",\"timeFieldName\":\"$time_field\"}}" \
  | jq -r '.id')
# Create the default index
curl -XPOST -H "Content-Type: application/json" -H "kbn-xsrf: anything" \
  "$url/api/kibana/settings/defaultIndex" \
  -d"{\"value\":\"$id\"}"

[fletchowns]

Thanks @hobti01 ! That seems to also work for Kibana 5.6.4

[nh2]

I found that in Kibana 5.0 (and probably also 6.0), you can set a specific Index pattern ID instead of having a randomly generated one as @hobti01's script stores in id=$(... | jq -r '.id'); that is equivalent to setting the following field in the UI:

Setting a given ID is especially convenient for automatic Kibana deployments, as your exported Dashboards etc then will always refer to a fixed ID instead of a random one.

You can do so by POSTing that ID to /api/saved_objects/index-pattern/THE_ID.

For example:

curl -f -XPOST -H 'Content-Type: application/json' -H 'kbn-xsrf: anything' 'http://localhost:5601/api/saved_objects/index-pattern/logstash-*' '-d{"attributes":{"title":"logstash-*","timeFieldName":"@timestamp"}}'

@hobti01's script adjusted to use a fixed ID:

#!/usr/bin/env bash
# From https://github.com/elastic/kibana/issues/3709
set -euo pipefail
url="http://localhost:5601"
index_pattern="logstash-*"
id="logstash-*"
time_field="@timestamp"
# Create index pattern
# curl -f to fail on error
curl -f -XPOST -H "Content-Type: application/json" -H "kbn-xsrf: anything" \
  "$url/api/saved_objects/index-pattern/$id" \
  -d"{\"attributes\":{\"title\":\"$index_pattern\",\"timeFieldName\":\"$time_field\"}}"
# Make it the default index
curl -XPOST -H "Content-Type: application/json" -H "kbn-xsrf: anything" \
  "$url/api/kibana/settings/defaultIndex" \
  -d"{\"value\":\"$id\"}"

When the ID already exists, the script will fail with curl: (22) The requested URL returned error: 409 Conflict and exit code 22.

[hobti01]

@nh2 you can remove the pipe to jq when using a static id - thanks for the suggestion!

[nh2]

you can remove the pipe to jq when using a static id

Oops, right! Edited.

[nh2]

When the ID already exists, the script will fail with curl: (22) The requested URL returned error: 409 Conflict and exit code 22.

If you don't want that, and want overwrite behaviour instead, use "$url/api/saved_objects/index-pattern/$id?overwrite=true" for the first curl invocation.

You will then get responses where the version counts up each time you do it, like:

{"id":"logstash-*","type":"index-pattern","version":2,"attributes":{"title":"logstash-*","timeFieldName":"@timestamp"}}
{"id":"logstash-*","type":"index-pattern","version":3,"attributes":{"title":"logstash-*","timeFieldName":"@timestamp"}}

[nh2]

I have also just written an export.json CLI importer tool for Kibana: https://github.com/nh2/kibana-importer

Together with this and the above index-pattern-creation curl, I can now deploy Kibana in a fully declarative fashion without having to load anything manually into it.

[matthewadams]

FYI, trying this against es 6.2.2 fails:

/ # curl http://es-api.kube-system.svc.cluster.local:9200
{
  "name" : "Yw3vJ4X",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "HWh0uIo2Q6uL4LOZXnau5Q",
  "version" : {
    "number" : "6.2.2",
    "build_hash" : "10b1edd",
    "build_date" : "2018-02-16T19:01:30.685723Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
/ # curl -f -XPOST -H "Content-Type: application/json" -H "kbn-xsrf: anything" 'http://es-api.kube-system.svc.cluster
.local:9200/api/saved_objects/index-pattern/logstash-*' -d'{"attributes":{"title":"logstash-*","timeFieldName":"@time
stamp"}}'
curl: (22) The requested URL returned error: 400 Bad Request

Any suggestions?

[hobti01]

@matthewadams it looks like you are making the request against elasticsearch without using the '.kibana' endpoint. Try making the POST directly to kibana.

[sandstrom]

If you would prefer a method for loading dashboards and index patterns from the filesystem (for use with Chef, Puppet, Salt, etc) please vouch for it: https://github.com/elastic/kibana/issues/2310

[matthewadams]

@sandstrom If by "vouched", you mean put a thumbs up on the issue, I did that. :) Good to know I'm not the only one.