Azure-sdk-for-net: [BUG] Azure Search - Cannot $filter by fields containing special characters, even when escaped

[PrimeHydra]

Describe the bug
When a filter query includes special characters that have been escaped with backslashes, as described in the SDK docs, it does not match fields containing the same text. (Also, single quote ' is apparently escaped by doubling it: ''.) In this case, the test document contains a field like:

"FirmName": "Crazy Charz Inc. ' + - && ! ( ) { } [ ] ^ \" ~ * ? : \\ /"

Expected behavior
The document above should be returned.

Actual behavior (include Exception or Stack Trace)
Instead, zero results are returned.

To Reproduce

1. Create an index with the document mentioned in the description.
2. Using the SDK, set SearchParams.Filter to search.in(FirmName, 'Crazy Charz Inc. '' \+ \- \&\& \! \( \) \{ \} \[ \] \^ \" \~ \* \? \: \\ \/', '|'). (That's how the value looks when I inspect in in VS.)
3. Observe that no results are returned (not an error; see SO link below for details on what happens in the Search Explorer with the same query).

Environment:

• Microsoft.Azure.Search v10.1.0
• Windows 10
• Search is performed in a .NET Standard 2.0 library and consumed by a .NET Core 3.1 library.
• Visual Studio 16.4.3

Also see this SO post.

Not to be confused with this feature request; the issue is that the filter doesn't work _even when escaping the characters_, not the need for a character-escaping method. Unless I'm doing something wrong, in which case, please tell me how to pass this test case. Thanks!

[brjohnstmsft]

@PrimeHydra I wrote a console app to repro this behavior, and it's not a bug -- you're using the encoding scheme for a different query syntax (Simple query syntax for full-text search, not the OData syntax for filters). I provided more details in my answer on StackOverflow.

[PrimeHydra]

Thanks, @brjohnstmsft . Though this was user error, I feel the SDK docs could have made the filter expression character escaping a bit clearer? Three different syntaxes in one request is a lot for search newbies like myself to keep track of. :) Maybe I missed that page in the docs. Appreciate your response.

[brjohnstmsft]

@PrimeHydra I agree it isn’t discoverable enough. It’s probably in a note somewhere. I’ll change the filter docs to make it more obvious.

We’re currently working on a new generation of client libraries for Azure Cognitive Search, and one of our goals is to provide a mechanism for encoding each syntax so you don’t have to deal with it anymore.

[brjohnstmsft]

@PrimeHydra I've updated the documentation. The changes should go live some time later today. I added a note at the top of the "simple" and "full Lucene" syntax pages explaining that they're not the same as the filter syntax, then added a section here explaining how string escaping works in OData.

Thanks for reporting this -- your feedback helps us make our service better!