Elasticsearch: Include client IP/host in slowlog entries

[ppf2]

Sometimes it may be helpful to be able to log the the host/IP of the end client that submitted a search or indexing request (eg. somebody has been sending these really bad queries to the cluster and we want to find out the host/IP of the offending search client application/host). Would be nice to have this as part of the slowlog entries.

[s1monw]

I agree this can be useful. It can also be useful in the context of shield to log user information who issues the request. We can forward the headers from the request to the SearchContenxt and then log it? I mean we can even go further and allow users to specify a mustache template to make up their log line here?

[cfeio]

+1 This would be very useful in troubleshooting where slow queries are coming from

[martijnvg]

+1 there is no doubt about the usefulness of this enhancement

[justinvaldez]

Please add this - It's way to hard to locate the offending bad queries at this point in time.

[sherry-ger]

+1

[seang-es]

+1

[charlesfeng]

+1

[paullovessearch]

+1

But also have need for application ID, user ID, transaction ID, etc. A generic log data parameter is best. Thanks!

[s1monw]

It seems pretty straight forward to add to me we can use the ThreadContext (5.0 only to pull information to log which is already forwarding all http headers that a user provides. We can have some predefined ones here?

[paullovessearch]

Also, the 2.X logs are missing the shard ID. Can we please add back the shard ID?

Version 1.7.X:

[2016-04-19 02:12:07,305][INFO ][index.search.slowlog.query]
[es-01] [twitter][4] took[3.9s], took_millis[3931], types[], stats[],
search_type[DFS_QUERY_THEN_FETCH], total_shards[10], source[…], extra_source[],

Version 2.3

[2016-05-13 23:28:55,886][INFO ][index.search.slowlog.query]
[twitter]took[37.5ms], took_millis[37], types[], stats[],
search_type[QUERY_THEN_FETCH], total_shards[5], source[…], extra_source[],

[benjoyce]

+1

Could really do with the client/source IP for the search query entries.

[shsander]

+1
I am trying to track down a bad search request. I have stopped all clients that I know are using ES. The only searches remaining in slowlog (with threshold set to 0s) are the ones that I am trying to tack down, but I have no idea who is making them. Client/Source IP would be very useful

[mahdibh]

+1 on allowing clients to specify some sort of clientId as part of their query request that also gets logged with the query (in addition to client ip, which can be obtained off-hand without the client making any changes). This can be extremely useful for tracing an execution all the way up from a user call down to a slow query hitting ES (using something like zipkin).

See https://discuss.elastic.co/t/recommendations-for-tracking-down-slow-queries/61649

[nezhazheng]

+1

[clintongormley]

Discussed in FixItFriday - we should allow the requestor to specify a custom header (eg X-Opaque-ID) which is then logged in the search and index slow logs.

[nmeisels]

+1 This would be extremely useful

[higee]

+1

[kovyrin]

+1 for adding user IP and the opaque id to slow logs (and, maybe, the user name as well)

[IngaFeick]

+1 Really really need this, but for all logs, not just slow logs.

[lwjli]

The x-pack plugin provides similar functionality, see https://www.elastic.co/guide/en/x-pack/current/auditing.html
But it just log the client ip for some events, not slowlog.

[mylinuxsa]

+1
I need

[kevinconaway]

It looks like the X-Opaque-Id is now available in the slowlog via https://github.com/elastic/elasticsearch/pull/31539

[pgomulka]

closing as this is being supported by x-opaque-id