Azure-sdk-for-net: [BUG] Search SDK v11 - SearchIndexerSkill optional properties throw exceptions upon deserialization on CreateSkillSet()

[glennmusa]

Describe the bug
In these two examples the Azure Search dotnet SDK SearchIndexerSkill deserializer throws exceptions on nullable properties and empty collections.

To get around this we have to put some value of the correct type in those properties of any SearchIndexerSkill.

The deserializer: https://github.com/Azure/azure-sdk-for-net/blob/a03cb7b0cdcf5fd1010eb043b7b2c9383dc5c189/sdk/search/Azure.Search.Documents/src/Generated/Models/WebApiSkill.Serialization.cs#L101

Expected behavior
Users shouldn't have to provide values for optional properties.

Actual behavior and Reproduction

Example 1 - Nullable properties are required to specified:

If you don't specify what are documented and implemented as nullable properties for a SearchIndexerSkill, the deserializer will throw exceptions.

The EntityRecognitionSkill class definition for reference.

The WebApiSkill class definition for reference.

When you add this EntityRecognitionSkill to a skillset:

var entityRecognitionSkill = new EntityRecognitionSkill(inputs, outputs)
{
    Context = "/document/finalText/pages/*"
};

Skillset creation will throw this ArgumentNullException:

An unhandled exception of type 'System.ArgumentNullException' occurred in System.Private.CoreLib.dll: 'Value cannot be null.'
at Azure.Search.Documents.Indexes.Models.EntityRecognitionSkillLanguage..ctor(String value) at Azure.Search.Documents.Indexes.Models.EntityRecognitionSkill.DeserializeEntityRecognitionSkill(JsonElement element) at Azure.Search.Documents.Indexes.Models.SearchIndexerSkill.DeserializeSearchIndexerSkill(JsonElement element) at Azure.Search.Documents.Indexes.Models.SearchIndexerSkillset.DeserializeSearchIndexerSkillset(JsonElement element) at Azure.Search.Documents.SkillsetsRestClient.Create(SearchIndexerSkillset skillset, CancellationToken cancellationToken) at Azure.Search.Documents.Indexes.SearchIndexerClient.CreateSkillset(SearchIndexerSkillset skillset, CancellationToken cancellationToken)
An unhandled exception of type 'System.ArgumentNullException' occurred in System.Private.CoreLib.dll: 'Value cannot be null.'
at Azure.Search.Documents.Indexes.Models.EntityRecognitionSkillLanguage..ctor(String value) at Azure.Search.Documents.Indexes.Models.EntityRecognitionSkill.DeserializeEntityRecognitionSkill(JsonElement element) at Azure.Search.Documents.Indexes.Models.SearchIndexerSkill.DeserializeSearchIndexerSkill(JsonElement element) at Azure.Search.Documents.Indexes.Models.SearchIndexerSkillset.DeserializeSearchIndexerSkillset(JsonElement element) at Azure.Search.Documents.SkillsetsRestClient.Create(SearchIndexerSkillset skillset, CancellationToken cancellationToken) at Azure.Search.Documents.Indexes.SearchIndexerClient.CreateSkillset(SearchIndexerSkillset skillset, CancellationToken cancellationToken)

Or a WebApiSkill like this to a skillset:

var webApiSkill = new WebApiSkill(inputs, outputs, uri)
{
    Description = "Upload image data to the annotation store",
    Context = "/document/normalized_images/*"
};

Skillset creation will throw this FormatException:

An unhandled exception of type 'System.FormatException' occurred in System.Private.CoreLib.dll: 'The string '' is not a valid TimeSpan value.'
at System.Xml.XmlConvert.ToTimeSpan(String s) at Azure.Search.Documents.Indexes.Models.WebApiSkill.DeserializeWebApiSkill(JsonElement element) at Azure.Search.Documents.Indexes.Models.SearchIndexerSkill.DeserializeSearchIndexerSkill(JsonElement element) at Azure.Search.Documents.Indexes.Models.SearchIndexerSkillset.DeserializeSearchIndexerSkillset(JsonElement element) at Azure.Search.Documents.SkillsetsRestClient.Create(SearchIndexerSkillset skillset, CancellationToken cancellationToken) at Azure.Search.Documents.Indexes.SearchIndexerClient.CreateSkillset(SearchIndexerSkillset skillset, CancellationToken cancellationToken) 

If you assign values to the nullable properties of these skills like below you're able to successfully create the skillset:

var entityRecognitionSkill = new EntityRecognitionSkill(inputs, outputs)
{
    Context = "/document/finalText/pages/*",
    DefaultLanguageCode = EntityRecognitionSkillLanguage.En //this is a nullable that serialization enforces as required
};

var webApiSkill = new WebApiSkill(inputs, outputs, uri)
{
    Description = "Upload image data to the annotation store",
    Context = "/document/normalized_images/*",
    BatchSize = 1, //this is a nullable that serialization enforces as required
    DegreeOfParallelism = 1, //this is a nullable that serialization enforces as required
    Timeout = System.TimeSpan.FromSeconds(30) //this is a nullable that serialization enforces as required
};

Example 2 - An empty dictionary will throw an Invalid Operation Exception

An empty HttpHeaders dictionary in a WebApiSkill throws a InvalidOperationException at deserialization.

The WebApiSkill class definition for reference

When you add this WebApiSkill to a skillset:

var webApiSkill = new WebApiSkill(inputs, outputs, uri)
{
    Description = "Generate HOCR for webpage rendering",
    Context = "/document",
    BatchSize = 1, //this is a nullable that serialization enforces as required
    DegreeOfParallelism = 1, //this is a nullable that serialization enforces as required
    Timeout = System.TimeSpan.FromSeconds(30), //this is a nullable that serialization enforces as required          
};

The skillset creation will throw this InvalidOperationException:

An unhandled exception of type 'System.InvalidOperationException' occurred in System.Private.CoreLib.dll: 'The requested operation requires an element of type 'Object', but the target element has type 'Null'.'
at System.Text.Json.JsonElement.EnumerateObject() at Azure.Search.Documents.Indexes.Models.WebApiSkill.DeserializeWebApiSkill(JsonElement element) at Azure.Search.Documents.Indexes.Models.SearchIndexerSkill.DeserializeSearchIndexerSkill(JsonElement element) at Azure.Search.Documents.Indexes.Models.SearchIndexerSkillset.DeserializeSearchIndexerSkillset(JsonElement element) at Azure.Search.Documents.SkillsetsRestClient.Create(SearchIndexerSkillset skillset, CancellationToken cancellationToken) at Azure.Search.Documents.Indexes.SearchIndexerClient.CreateSkillset(SearchIndexerSkillset skillset, CancellationToken cancellationToken)

To get the skillset to create, you have to put some value in the dictionary even if you don't need it:

var webApiSkill = new WebApiSkill(inputs, outputs, uri)
{
    Description = "Generate HOCR for webpage rendering",
    Context = "/document",
    BatchSize = 1, //this is a nullable that serialization enforces as required
    DegreeOfParallelism = 1, //this is a nullable that serialization enforces as required
    Timeout = System.TimeSpan.FromSeconds(30), //this is a nullable that serialization enforces as required
};

//this dictionary requires _some_ value
webApiSkill.HttpHeaders["foo"] = "bar";

Environment:

• Name and version of the Library package used:
  
  
  
  • Azure.Search.Documents 11.1.1
  
  

• Hosting platform or OS and .NET runtime version:

  .NET Core SDK (reflecting any global.json):
   Version:   3.1.401
   Commit:    39d17847db

  Runtime Environment:
  OS Name:     ubuntu
  OS Version:  20.04
  OS Platform: Linux
  RID:         ubuntu.20.04-x64

• IDE and version:
  
  
  
  • Visual Studio Code Version: 1.49.0
  
  

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[heaths]

Thanks for finding these issues. There appears to be some discrepancy between swagger and our source generator. All required properties should be hoisted to the constructor, which didn't happen here. We can't change existing constructors, but will consider options to make it more obvious what are required properties across different skills.