Elasticsearch: Implement an @Awaitsfix equivalent for YML tests

[alpar-t]

Right now we rely on the tests.rest.blacklist system property passed to the rest test runner to be able to mute tests.
This is a cumbersome and too far from the test. The muted tests are not included in the report @polyfractal sends out, and this information can't be consumed easily picked up by @elastic/es-clients .

We should have a way to do this in the rest syntax.
Maybe @javanna has some ideas on the syntax.

[elasticmachine]

Pinging @elastic/es-core-infra

[javanna]

Yaml tests support a skip section, where either a version range can be specified (we have used 'all' in the past as well I think), or a list of features that need to be supported by runner for the test to run. Haven't we been using this section to skip tests as well? As far as I remember, the blacklist was introduced (not sure it's still used that way) in the java runner to not run certain tests with certain setups, for instance when security is installed indices resolution is different and a couple of tests from our rest-api-spec/test collection fail.

[polyfractal]

Yeah, skip: 'all' is the current way to "awaits fix" a yaml test. The email script digs through the yaml tests looking for any 'all' skips and includes them at the bottom of the email.

Here's a (very old) example: https://github.com/elastic/elasticsearch/blob/master/rest-api-spec/src/main/resources/rest-api-spec/test/update/85_fields_meta.yml#L4-L6

[karmi]

Just confirming that we _heavily_ use the skip feature of YAML tests in all the language clients.

[alpar-t]

I know I used blacklists for muting in the past, but I checked the code base and couldn't find examples of this so I guess we should be fine. I'll close this issue, thanks for the explanations.

[polyfractal]

It might be useful to document this somewhere though, since I'm not sure how widespread knowledge of skip: all is to the general ES team?

E.g. I know the yaml test readme state you can skip, but perhaps we should add a note wherever we document awaitsfix too?

[alpar-t]

The way I got here was that I wanted to mute a test for #38711 which fails for smoke-test-multi-node but not for others. Guess I'll have to mute the test for all other runs too then.

[javanna]

@atorok I think that's correct, seems like this test fails when run against multiple nodes? If that's the case it should be muted completely as we can't assume that all the language client runners are running on a single node.