Azure-sdk-for-net: Search SDK: FieldBuilder tries to read a property marked with JsonIgnore

[jnhaffey]

We are using Microsoft.Azure.Search.dll (3.0.0) from NuGet Package 3.0.3

Here are our Interfaces:

public interface IResourceSearchDocument
{
    string RecordType { get; set; }
    string Title { get; set; }
    string Description { get; set; }
    DateTime? PublishedDate { get; set; }
    DateTime? LastUpdatedDate { get; set; }
    string ImageUrl { get; set; }
    string LinkToResource { get; set; }
}

public interface IEventSearchDocument
{
    string EventId { get; set; }
    string ProductId { get; set; }
    string EventCode { get; set; }
    string EventType { get; set; }
    string Location { get; set; }
    DateTime? EventStartDate { get; set; }
    DateTime? EventEndDate { get; set; }
}

This is our base class:

[SerializePropertyNamesAsCamelCase]
public abstract class SearchDocument : ISearchDocument
{
    /// <summary>
    /// Gets or sets the ID for the document. For consistency, this should never be updated or retrieved manually.
    /// All IDs should be set through the <see cref="Id"/> property.
    /// </summary>
    [Key]
    [JsonProperty("Id")]
    public string AzureId { get; set; }

    /// <summary>
    /// Gets or sets any unique IDs or compound IDs that might contain characters unsafe for transmission via URL.
    /// For consistency, all IDs should be set through this property.
    /// </summary>
    [JsonIgnore]
    [IsRetrievable(false)]
    public string Id
    {
        get
        {
            return AzureId.FromBase64EncodedString();
        }
        set
        {
            AzureId = value.ToBase64EncodedString();
        }
    }
}

Finally, this is the WebSiteSearchDocument:

[SerializePropertyNamesAsCamelCase]
public class WebSearchDocument : SearchDocument, IEventSearchDocument, IResourceSearchDocument
{
    #region Implementation of IEventSearchDocument
    [IsRetrievable(true)]
    public string EventId { get; set; }
    [IsRetrievable(true)]
    public string ProductId { get; set; }
    [IsRetrievable(true)]
    public string EventCode { get; set; }
    [IsRetrievable(true), IsFilterable]
    public string EventType { get; set; }
    public string Location { get; set; }
    [IsRetrievable(true), IsFilterable]
    public DateTime? EventStartDate { get; set; }
    [IsRetrievable(true), IsFilterable]
    public DateTime? EventEndDate { get; set; }

    #endregion Implementation of IEventSearchDocument

    #region Implementation of IResourceSearchDocument
    [IsRetrievable(true), IsFilterable]
    public string RecordType { get; set; }

    [JsonIgnore]
    [IsRetrievable(false)]
    public WebSearchRecordType RecordTypeEnum { get; set; }

    [IsRetrievable(true), IsSearchable]
    public string Title { get; set; }
    [IsRetrievable(true), IsSearchable]
    public string Description { get; set; }
    [IsRetrievable(true)]
    public DateTime? PublishedDate { get; set; }
    [IsRetrievable(true)]
    public DateTime? LastUpdatedDate { get; set; }
    [IsRetrievable(true)]
    public string ImageUrl { get; set; }
    [IsRetrievable(true)]
    public string LinkToResource { get; set; }

    #endregion Implementation of IResourceSearchDocument
}

We have set up an IndexManager and this method is called to 'Create the Index':

public Index Create<T>(string indexName) where T : class, ISearchDocument
{
    if (Exists(indexName))
    {
        throw new ArgumentException($"An index by the name of {indexName} already exists.", nameof(indexName));
    }

    var indexDefinition = new Index
    {
        Name = indexName,
        Fields = FieldBuilder.BuildForType<T>()
    };

    return _serviceClient.Indexes.Create(indexDefinition);
}

This is the exception we are getting back when we try to set the enum to JsonIgnore and IsRetrievable(false)

{"Property recordTypeEnum has unsupported type BACP.Web.Search.WebSearchRecordType\r\nParameter name: propertyType"}
Data: {System.Collections.ListDictionaryInternal}
HResult: -2147024809
HelpLink: null
IPForWatsonBuckets: {140706498970467}
InnerException: null
IsTransient: false
Message: "Property recordTypeEnum has unsupported type BACP.Web.Search.WebSearchRecordType\r\nParameter name: propertyType"
ParamName: "propertyType"
RemoteStackTrace: null
Source: "Microsoft.Azure.Search"
StackTrace: " at Microsoft.Azure.Search.FieldBuilder.GetDataType(Type propertyType, String propertyName)\r\n at Microsoft.Azure.Search.FieldBuilder.BuildForTypeT\r\n at Felinesoft.Search.IndexManager.d__51.MoveNext() in E:\\Felinesoft\\BACP\\DEV\\WEB\\BACP.Web\\Felinesoft.Search\\IndexManager.cs:line 86\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\r\n at Felinesoft.Search.IndexManager.d__7`1.MoveNext() in E:\Felinesoft\BACP\DEV\WEB\BACP.Web\Felinesoft.Search\IndexManager.cs:line 117\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtim
e.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.GetResult()\r\n at BACP.Web.Presentation.Infrastructure.Azure.SearchIndexBuilder.<>c__DisplayClass26_0.<b__4>d.MoveNext() in E:\Felinesoft\BACP\DEV\WEB\BACP.Web\BACP.Web.Presentation\Infrastructure\Azure\SearchIndexBuilder.cs:line 180\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.GetResult()\r\n at BACP.Web.Presentation.Controllers.AzureSearchToolsController.d__6.MoveNext() in E:\Felinesoft\BACP\DEV\WEB\BACP.Web\BACP.Web.Presentation\Controllers\AzureSearchToolsController.cs:line 52"
TargetSite: {Microsoft.Azure.Search.Models.DataType GetDataType(System.Type, System.String)}
WatsonBuckets: null
_HResult: -2147024809
_className: null
_data: {System.Collections.ListDictionaryInternal}
_dynamicMethods: null
_exceptionMethod: {Microsoft.Azure.Search.Models.DataType GetDataType(System.Type, System.String)}
_exceptionMethodString: null
_helpURL: null
_innerException: null
_ipForWatsonBuckets: {140706498970467}
_message: "Property recordTypeEnum has unsupported type BACP.Web.Search.WebSearchRecordType"
_remoteStackIndex: 0
_remoteStackTraceString: null
_safeSerializationManager: {System.Runtime.Serialization.SafeSerializationManager}
_source: "Microsoft.Azure.Search"
_stackTrace: {sbyte[768]}
_stackTraceString: null
_watsonBuckets: null
_xcode: -532462766
_xptrs: {0}
m_paramName: "propertyType"

It is my understanding that putting the JsonIgnore and IsRetrievable(false) attributes on the RecordTypeEnum property, then the Microsoft.Azure.Search.Models.DataType GetDataType(System.Type, System.String) should ignore it.

[brjohnstmsft]

@jnhaffey Thanks for reporting this. We'll investigate and see what needs to be fixed.

In the meantime, to unblock yourself you can create the index fields directly by using the various Field class constructors. From the error message I'm pretty sure the issue is in the FieldBuilder class, so creating the Field objects directly should unblock you.

[ghost]

Ccq

Get Outlook for Androidhttps://aka.ms/ghei36

---

From: Jason N Haffey [e-mail]
Sent: Thursday, June 1, 2017 5:47:06 AM
To: Azure/azure-sdk-for-net
Cc: Subscribed
Subject: [Azure/azure-sdk-for-net] Azure Search SDK trying to read property I have told it to ignore (#3309)

We are using Microsoft.Azure.Search.dll (3.0.0) from NuGet Package 3.0.3

Here are our Interfaces:

public interface IResourceSearchDocument
{
string RecordType { get; set; }
string Title { get; set; }
string Description { get; set; }
DateTime? PublishedDate { get; set; }
DateTime? LastUpdatedDate { get; set; }
string ImageUrl { get; set; }
string LinkToResource { get; set; }
}

public interface IEventSearchDocument
{
string EventId { get; set; }
string ProductId { get; set; }
string EventCode { get; set; }
string EventType { get; set; }
string Location { get; set; }
DateTime? EventStartDate { get; set; }
DateTime? EventEndDate { get; set; }
}

This is our base class:

[SerializePropertyNamesAsCamelCase]
public abstract class SearchDocument : ISearchDocument
{
///

/// Gets or sets the ID for the document. For consistency, this should never be updated or retrieved manually.
/// All IDs should be set through the property.
///

[Key]
[JsonProperty("Id")]
public string AzureId { get; set; }

    /// <summary>
    /// Gets or sets any unique IDs or compound IDs that might contain characters unsafe for transmission via URL.
    /// For consistency, all IDs should be set through this property.
    /// </summary>
    [JsonIgnore]
    [IsRetrievable(false)]
    public string Id
    {
            get
            {
                    return AzureId.FromBase64EncodedString();
            }
            set
            {
                    AzureId = value.ToBase64EncodedString();
            }
    }

}

Finally, this is the WebSiteSearchDocument:

[SerializePropertyNamesAsCamelCase]
public class WebSearchDocument : SearchDocument, IEventSearchDocument, IResourceSearchDocument
{
#region Implementation of IEventSearchDocument
[IsRetrievable(true)]
public string EventId { get; set; }
[IsRetrievable(true)]
public string ProductId { get; set; }
[IsRetrievable(true)]
public string EventCode { get; set; }
[IsRetrievable(true), IsFilterable]
public string EventType { get; set; }
public string Location { get; set; }
[IsRetrievable(true), IsFilterable]
public DateTime? EventStartDate { get; set; }
[IsRetrievable(true), IsFilterable]
public DateTime? EventEndDate { get; set; }

    #endregion Implementation of IEventSearchDocument

    #region Implementation of IResourceSearchDocument
    [IsRetrievable(true), IsFilterable]
    public string RecordType { get; set; }

    [JsonIgnore]
    [IsRetrievable(false)]
    public WebSearchRecordType RecordTypeEnum { get; set; }

    [IsRetrievable(true), IsSearchable]
    public string Title { get; set; }
    [IsRetrievable(true), IsSearchable]
    public string Description { get; set; }
    [IsRetrievable(true)]
    public DateTime? PublishedDate { get; set; }
    [IsRetrievable(true)]
    public DateTime? LastUpdatedDate { get; set; }
    [IsRetrievable(true)]
    public string ImageUrl { get; set; }
    [IsRetrievable(true)]
    public string LinkToResource { get; set; }

    #endregion Implementation of IResourceSearchDocument

}

We have set up an IndexManager and this method is called to 'Create the Index':

public Index Create(string indexName) where T : class, ISearchDocument
{
if (Exists(indexName))
{
throw new ArgumentException($"An index by the name of {indexName} already exists.", nameof(indexName));
}

    var indexDefinition = new Index
    {
            Name = indexName,
            Fields = FieldBuilder.BuildForType<T>()
    };

    return _serviceClient.Indexes.Create(indexDefinition);

}

This is the exception we are getting back when we try to set the enum to JsonIgnore and IsRetrievable(false)

{"Property recordTypeEnum has unsupported type BACP.Web.Search.WebSearchRecordType\r\nParameter name: propertyType"}
Data: {System.Collections.ListDictionaryInternal}
HResult: -2147024809
HelpLink: null
IPForWatsonBuckets: {140706498970467}
InnerException: null
IsTransient: false
Message: "Property recordTypeEnum has unsupported type BACP.Web.Search.WebSearchRecordType\r\nParameter name: propertyType"
ParamName: "propertyType"
RemoteStackTrace: null
Source: "Microsoft.Azure.Search"
StackTrace: " at Microsoft.Azure.Search.FieldBuilder.GetDataType(Type propertyType, String propertyName)\r\n at Microsoft.Azure.Search.FieldBuilder.BuildForTypeT\r\n at Felinesoft.Search.IndexManager.d__51.MoveNext() in E:\Felinesoft\BACP\DEV\WEB\BACP.Web\Felinesoft.Search\IndexManager.cs:line 86\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\r\n at Felinesoft.Search.IndexManager.d__7`1.MoveNext() in E:\Felinesoft\BACP\DEV\WEB\BACP.Web\Felinesoft.Search\IndexManager.cs:line 117\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtim
e.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.GetResult()\r\n at BACP.Web.Presentation.Infrastructure.Azure.SearchIndexBuilder.<>c__DisplayClass26_0.d.MoveNext() in E:\Felinesoft\BACP\DEV\WEB\BACP.Web\BACP.Web.Presentation\Infrastructure\Azure\SearchIndexBuilder.cs:line 180\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.GetResult()\r\n at BACP.Web.Presentation.Controllers.AzureSearchToolsController.d__6.MoveNext() in E:\Felinesoft\BACP\DEV\WEB\BACP.Web\BACP.Web.Presentation\Controllers\AzureSearchToolsController.cs:line 52"
TargetSite: {Microsoft.Azure.Search.Models.DataType GetDataType(System.Type, System.String)}
WatsonBuckets: null
_HResult: -2147024809
_className: null
_data: {System.Collections.ListDictionaryInternal}
_dynamicMethods: null
_exceptionMethod: {Microsoft.Azure.Search.Models.DataType GetDataType(System.Type, System.String)}
_exceptionMethodString: null
_helpURL: null
_innerException: null
_ipForWatsonBuckets: {140706498970467}
_message: "Property recordTypeEnum has unsupported type BACP.Web.Search.WebSearchRecordType"
_remoteStackIndex: 0
_remoteStackTraceString: null
_safeSerializationManager: {System.Runtime.Serialization.SafeSerializationManager}
_source: "Microsoft.Azure.Search"
_stackTrace: {sbyte[768]}
_stackTraceString: null
_watsonBuckets: null
_xcode: -532462766
_xptrs: {0}
m_paramName: "propertyType"

It is my understanding that putting the JsonIgnore and IsRetrievable(false) attributes on the RecordTypeEnum property, then the Microsoft.Azure.Search.Models.DataType GetDataType(System.Type, System.String) should ignore it.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/Azure/azure-sdk-for-net/issues/3309, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AQQ1kbojVlg-TgSF2ock09p5hrGWVeZFks5r_rLKgaJpZM4Ns7bu.

[bartroozendaal]

I am running into this too. Is there any way to tell FieldBuilder to ignore fields? If not, I'll do a post-processing, removing fields that I do not want.

[brjohnstmsft]

@bartroozendaal As a workaround, you can create the index fields directly by using the various Field class constructors.

We have confirmed that this is an issue with the FieldBuilder class. There's a fix on the way, but no ETA yet.

[nprieto95]

A more elegant workaround could be to call the overload of BuildForType that takes an IContractResolver and provide it with a contract resolver that inherits (in this case) from CamelCasePropertyNamesContractResolver and which removes unsupported properties from the result before returning it in the CreateProperties override. In my case, I have a "products" index with an unsupported property called "prices", and my solution looks like this:

searchServiceClient.Indexes.CreateOrUpdate(
                new Index(
                    "products",
                    FieldBuilder.BuildForType<Product>(new ProductSearchContractResolver())
                )
            );

with ProductSearchContractResolver defined as:

public class ProductSearchContractResolver : CamelCasePropertyNamesContractResolver
{
    protected override IList<JsonProperty> CreateProperties(Type type, MemberSerialization memberSerialization)
    {
        IList<JsonProperty> properties = base.CreateProperties(type, memberSerialization);
        properties = properties.Where(p => p.PropertyName != "prices").ToList();
        return properties;
    }
}

It's working like a charm on our side, although I am eager to have a solution right here on the SDK to remove the workaround.

[nprieto95]

Note also that this solution makes it pointless to have the model class annotated as SerializePropertyNamesAsCamelCase.

[Gansted]

To make your workaround slightly more generic you could change

properties = properties.Where(p => p.PropertyName != "prices").ToList();

to

properties = properties.Where(p => !p.Ignored).ToList();

That way all properties marked with the JsonIgnore attribute will be ignored when building your fields.

[brjohnstmsft]

The fix for this issue shipped with version 3.0.5 to NuGet.