Elasticsearch-net: [meta] 6.4.0 release

[Mpdreamz]

This is a meta issue to keep track of the work we want to do for the 6.4.0 release individual tickets with the v6.4.0 tag will be created for most of these.

Feature parity work

Deprecations

• [x] Add deprecation warnings / changes for nGram and edgeNGram - https://github.com/elastic/elasticsearch/pull/30209
• [x] Investigate copy_settings default - https://github.com/elastic/elasticsearch/pull/30404
• [x] Default shards changing from 5 to 1 - https://github.com/elastic/elasticsearch/pull/30587
• [x] Deprecate not copy settings and explicitly disallow - https://github.com/elastic/elasticsearch/pull/30404 (https://github.com/elastic/elasticsearch-net/pull/3416)
• [x] Deprecate accepting malformed requests in stored script API - https://github.com/elastic/elasticsearch/pull/28939 Check, suspect we are OK
• [x] In the field capabilities API, deprecate support for providing fields in the request body. https://github.com/elastic/elasticsearch/pull/30157
• [x] Deprecate indexing and querying a context completion field without context - https://github.com/elastic/elasticsearch/pull/30712

New Features

• [x] Add WeightedAvg metric aggregation - https://github.com/elastic/elasticsearch/pull/31037 (https://github.com/elastic/elasticsearch-net/pull/3417)
• [x] Add a MovingFunction pipeline aggregation, deprecate MovingAvg - https://github.com/elastic/elasticsearch/pull/29594 (https://github.com/elastic/elasticsearch-net/pull/3419)
• [x] Add missing_bucket option in the composite agg - https://github.com/elastic/elasticsearch/pull/29465
• [x] Expose lucenes RemoveDuplicatesTokenFilter - https://github.com/elastic/elasticsearch/pull/31275
• [x] Multiplexing token filter - https://github.com/elastic/elasticsearch/pull/31208
• [x] Adds a new analysis plugin called analysis_nori that exposes the Lucene Korean analysis module - https://github.com/elastic/elasticsearch/pull/30397
• [x] Adding a char_group tokenizer - https://github.com/elastic/elasticsearch/pull/24186
• [ ] Add Kerberos authentication support - https://github.com/elastic/elasticsearch/pull/32263
• [ ] ML. Detectors now support custom rules that enable the user to improve machine learning results by providing some domain-specific knowledge in the form of rule. - https://github.com/elastic/elasticsearch/issues/31110
• [ ] ML. Reverse engineer Grok patterns from categorization results - https://github.com/elastic/elasticsearch/pull/30125
• [x] Add support for field aliases - https://github.com/elastic/elasticsearch/pull/32172
• [x] Add an option to split keyword field on whitespace at query time - https://github.com/elastic/elasticsearch/pull/30691
• [x] The new _ignored field enables you to know which fields got ignored at index time because of the ignore_malformed option - https://github.com/elastic/elasticsearch/pull/29658
• [x] Add more contexts to painless execute api https://github.com/elastic/elasticsearch/pull/30511
• [x] Index phrases https://github.com/elastic/elasticsearch/pull/30450
• [ ] Add a format option to docvalue_fields. - https://github.com/elastic/elasticsearch/pull/29639

Enhancements

• [x] Add exclusion option to keep_types token filter - https://github.com/elastic/elasticsearch/pull/32012
• [x] Added lenient flag for synonym token filter - https://github.com/elastic/elasticsearch/pull/31484
• [x] Consistent encoder names - https://github.com/elastic/elasticsearch/pull/29492
• [x] Add support for ignore_unmapped to geo sort - https://github.com/elastic/elasticsearch/pull/31153
• [x] Add Index UUID to /_stats Response - https://github.com/elastic/elasticsearch/pull/31871
• [x] INGEST: Extend KV Processor - https://github.com/elastic/elasticsearch/pull/32232/files
• [x] date_index_name processor template resolution. https://github.com/elastic/elasticsearch/pull/31841
• [x] Introduction of a bytes processor - https://github.com/elastic/elasticsearch/pull/31733
• [x] Extend allowed characters for grok field names - https://github.com/elastic/elasticsearch/pull/31653
• [x] Ingest: Add ignore_missing option to RemoveProc - https://github.com/elastic/elasticsearch/pull/31693
• [x] Add region ISO code to GeoIP Ingest plugin - https://github.com/elastic/elasticsearch/pull/31669
• [x] Add ingest-attachment support for per document indexed_chars limit - https://github.com/elastic/elasticsearch/pull/31352
• [ ] Return statistics about forecasts as part of the job stats and usage API - https://github.com/elastic/elasticsearch/pull/31647
• [ ] [ML] Add description to ML filters - https://github.com/elastic/elasticsearch/pull/31330
• [x] Remove RestGetAllMappingsAction https://github.com/elastic/elasticsearch/pull/31129
• [x] _cluster/state should always return cluster_uuid - https://github.com/elastic/elasticsearch/pull/30143
• [ ] Rename ranking evaluation quality_level to metric_score - https://github.com/elastic/elasticsearch/pull/32168
• [ ] Rename ranking evaluation response unknown_docs section - https://github.com/elastic/elasticsearch/pull/32166
• [ ] Add Expected Reciprocal Rank metric - https://github.com/elastic/elasticsearch/pull/31891
• [ ] Add details section for dcg ranking metric - https://github.com/elastic/elasticsearch/pull/31177
• [x] Add second level of field collapsing - https://github.com/elastic/elasticsearch/pull/31808
• [x] Added max_expansion param to span_multi - https://github.com/elastic/elasticsearch/pull/30913
• [x] Store username on watch execution - https://github.com/elastic/elasticsearch/pull/31873

Bug fixes

• [x] Ensure that ip_range aggregations always return bucket keys. - https://github.com/elastic/elasticsearch/pull/30701
• [x] Add support for indexed shape routing in geo_shape query - https://github.com/elastic/elasticsearch/pull/30760
• [x] Add support for is_write_index in put-alias body parsing - https://github.com/elastic/elasticsearch/pull/31674
• [x] Make sure that field collapsing supports field aliases - https://github.com/elastic/elasticsearch/pull/32648
• [x] Get Mapping API to honour allow_no_indices and ignore_unavailable - https://github.com/elastic/elasticsearch/pull/31507
• [x] Fix _cluster/state to always return cluster_uuid - https://github.com/elastic/elasticsearch/pull/30656
• [x] Reject forcemerge requests with a body - https://github.com/elastic/elasticsearch/pull/30792

New API's

There a quite a few new API's that need to be mapped:

• [ ] xpack.license.get_basic_status.json
• [ ] xpack.license.post_start_basic.json
• [ ] xpack.ml.delete_calendar.json
• [ ] xpack.ml.delete_calendar_event.json
• [ ] xpack.ml.delete_calendar_job.json
• [ ] xpack.ml.get_calendar_events.json
• [ ] xpack.ml.get_calendars.json
• [ ] xpack.ml.info.json
• [ ] xpack.ml.post_calendar_events.json
• [ ] xpack.ml.put_calendar.json
• [ ] xpack.ml.put_calendar_job.json
• [ ] xpack.ml.get_calendar_job.json
• [x] xpack.rollup.delete_job.json
• [x] xpack.rollup.get_jobs.json
• [x] xpack.rollup.get_rollup_caps.json (https://github.com/elastic/elasticsearch/pull/30401)
• [x] xpack.rollup.put_job.json
• [x] xpack.rollup.rollup_search.json
• [x] xpack.rollup.start_job.json
• [x] xpack.rollup.stop_job.json
• [x] xpack.sql.clear_cursor.json (https://github.com/elastic/elasticsearch-net/pull/3414)
• [x] xpack.sql.query.json (https://github.com/elastic/elasticsearch-net/pull/3414)
• [x] xpack.sql.translate.json (https://github.com/elastic/elasticsearch-net/pull/3414)
• [ ] xpack.ssl.certificates.json
• [x] scripts_painless_execute.json
• [ ] xpack.ml.update_filter.json
• [ ] xpack.rollup.get_rollup_index_caps.json
• [ ] xpack.security.delete_privileges.json
• [ ] xpack.security.get_privileges.json
• [ ] xpack.security.has_privileges.json
• [ ] xpack.security.put_privilege.json
• [ ] xpack.security.put_privileges.json

Tests.Throughput

An unscientific throughput application to measure our different IConnection implementations on how many messages we can pump. For now we only plan to measure _bulk and _search and we'll use the machine learning data set MachineLearningCluster already pulls down.

Will warrant its own [meta] ticket which will be linked here later.

Netcoreapp2.1

or 2.2 whichever comes first 😸.

• [ ] Explicitly ship this TFM and get some numbers how the managed handler performs on linux/windows through Tests.Throughput vs the already existing implementations/
• [ ] Move our dotnet core IHttpConnection over to HttpClientFactory. We currently do much of the same but if we don't have to maintain our own pooling of HttpClient instances that would eleviate some complexity in our codebase.

Elasticsearch.Net.Abstractions

• [ ] Use differ from this repository see: https://github.com/elastic/elasticsearch-net-abstractions/pull/13
• [ ] Move ApiGeneration over.
• [ ] Move DocGeneration over.

[Mpdreamz]

Add deprecation warnings / changes for nGram and edgeNGram - elastic/elasticsearch#30209

No action needed we are using the non deprecated formats 😸

[Mpdreamz]

Default shards changing from 5 to 1 - elastic/elasticsearch#30587
No action needed on our end, integration tests still run successfully where we do not explicitly state the number of shards.

[Mpdreamz]

Deprecate accepting malformed requests in stored script API - elastic/elasticsearch#28939

No action need on our end.

[Mpdreamz]

In the field capabilities API, deprecate support for providing fields in the request body. - elastic/elasticsearch#30157

Our mapping of this API always uses GET so no action needed

[Mpdreamz]

Deprecate indexing and querying a context completion field without context - elastic/elasticsearch#30712

We expose a handy CompletionField that folks can use on their POJO's but the validation for this should live on the server.

[Mpdreamz]

Consistent encoder names - elastic/elasticsearch#29492

Ascii docs change only.

[Mpdreamz]

The new _ignored field enables you to know which fields got ignored at index time because of the ignore_malformed option - elastic/elasticsearch#29658

This new field gets automatically stored. Doesn't like like an action is needed on our end.

[Mpdreamz]

date_index_name processor template resolution. - elastic/elasticsearch#31841

Doesn't look like this requires an action on our end (no additional properties).

[Mpdreamz]

Extend allowed characters for grok field names - elastic/elasticsearch#31653

No action needed on our end.

[Mpdreamz]

Add region ISO code to GeoIP Ingest plugin - elastic/elasticsearch#31669

No action needed unless we want to include another default implementation for the GeoIP target field class.

[Mpdreamz]

Remove RestGetAllMappingsAction - elastic/elasticsearch#31129

This has no consequences on our end afaict.

[Mpdreamz]

Added max_expansion param to span_multi - elastic/elasticsearch#30913
No action needed on our part, PR ended up reusing and adhering to using bool max clauses rather than introducing a new option.

[Mpdreamz]

Add support for indexed shape routing in geo_shape query- elastic/elasticsearch#30760

Our IFieldLookup already exposes routing 😸

[Mpdreamz]

Make sure that field collapsing supports field aliases - elastic/elasticsearch#32648

Requires no changes on our end.

[Mpdreamz]

Fix _cluster/state to always return cluster_uuid- elastic/elasticsearch#30656

No action needed, fixes a behaviour from a previous PR which we covered earlier as well.

[Mpdreamz]

Get Mapping API to honour allow_no_indices and ignore_unavailable - elastic/elasticsearch#31507

We have tests for getting mappings from indices that do not exist that still run OK. Im happy to skip this one.

`

[Mpdreamz]

Reject forcemerge requests with a body - elastic/elasticsearch#30792

We are not sending a body and our integrations are happy, skipping.

[Mpdreamz]

Closing this in favour of https://github.com/elastic/elasticsearch-net/issues/3457

Moved remaining items that we bumped to that ticket.