Azure-sdk-for-net: Using Azure .NET SDK with NewtonSoft.Json 9.0.2-beta results in SerializationBinder error

[fuocor]

I am getting a TargetInvocationException on an index with the innerExcpetion of "Cannot get SerializationBinder because an ISerializationBinder was previously set." from Newtonsoft.Json.

The code is pretty straight forward:

        try
        {
            if(await SearchClient.Indexes.ExistsAsync(INDEX))
            {
                Logger.LogInformation("{0}", "Deleting index..\n");
                await SearchClient.Indexes.DeleteAsync(INDEX);
            }
            if(await SearchClient.DataSources.ExistsAsync(DATA_SOUCE))
            {
                Logger.LogInformation("{0}", "Deleting data source...\n");
                await SearchClient.DataSources.DeleteAsync(DATA_SOUCE);
            }
            if(await SearchClient.Indexers.ExistsAsync(INDEXER))
            {
                Logger.LogInformation("{0}", "Deleting indexer...\n");
                await SearchClient.Indexers.DeleteAsync(INDEXER);
            }
        }
        catch (Exception ex)
        {
            Logger.LogError("Error deleting indexing resources: {0}\r\n", ex.Message);
            throw;
        }

Any help is appreciated

rich

PS - I'm using Windows.Azure.Search (2.0.6-preview)

[fuocor]

It appears that downgrading Newtonsoft.Json to 9.0.1 from 9.0.2-beta resolves the issue

[brjohnstmsft]

@fuocor We're aware of this issue. It seems like a compatibility issue with the latest beta of JSON.NET, but we haven't been able to track it down yet.

[fuocor]

thanks

[brjohnstmsft]

@fuocor I'm having trouble getting a minimal repro of this problem. Would you be willing to share your code, or whatever minimal subset of it triggers the issue? Just calling ExistsAsync in a console app doesn't seem to be enough.

[laultman]

I am getting the same error. Following this example:
https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-csharp?toc=%2fazure%2fvirtual-machines%2fwindows%2ftoc.json
Console app .NET Core 1.1.0
My project.json file content:
{
"version": "1.0.0-*",
"buildOptions": {
"emitEntryPoint": true
},

"dependencies": {
"Microsoft.Azure.Management.ResourceManager": "1.4.0-preview",
"Microsoft.IdentityModel.Clients.ActiveDirectory": "3.13.6",
"Microsoft.Rest.ClientRuntime.Azure": "3.3.3",
"Newtonsoft.Json": "9.0.2-beta1",
"Microsoft.NETCore.App": {
"type": "platform",
"version": "1.1.0"
},
"Microsoft.Azure.Management.Storage": "5.0.2-preview",
"Microsoft.Azure.Management.Network": "8.0.0-preview",
"Microsoft.Azure.Management.Compute": "13.1.0-prerelease"
},
"runtimes": {
"win10-x64": {}
},
"frameworks": {
"netcoreapp1.0": {
"imports": [
"dnxcore50",
"portable-net45+win8"
]
}
}
}

[fuocor]

Just downgrade to 9.1 of NewtonSoft to work around it
Rich

Sent from my Windows 10 phone

From: Larry Aultmanmailto:[email protected]
Sent: Sunday, November 20, 2016 6:14 PM
To: Azure/azure-sdk-for-netmailto:[email protected]
Cc: Richard Fuocomailto:[email protected]; Mentionmailto:[email protected]
Subject: Re: [Azure/azure-sdk-for-net] Search SDK: Indexes.ExistsAsync faulting (#2552)

I am getting the same error. Following this example:
https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-windows-csharp-template?toc=%2fazure%2fvirtual-machines%2fwindows%2ftoc.json
Console app .NET Core 1.1.0
My project.json file content:
{
"version": "1.0.0-*",
"buildOptions": {
"emitEntryPoint": true
},

"dependencies": {
"Microsoft.Azure.Management.ResourceManager": "1.4.0-preview",
"Microsoft.IdentityModel.Clients.ActiveDirectory": "3.13.6",
"Microsoft.Rest.ClientRuntime.Azure": "3.3.3",
"Newtonsoft.Json": "9.0.2-beta1",
"Microsoft.NETCore.App": {
"type": "platform",
"version": "1.1.0"
},
"Microsoft.Azure.Management.Storage": "5.0.2-preview",
"Microsoft.Azure.Management.Network": "8.0.0-preview",
"Microsoft.Azure.Management.Compute": "13.1.0-prerelease"
},
"runtimes": {
"win10-x64": {}
},
"frameworks": {
"netcoreapp1.0": {
"imports": [
"dnxcore50",
"portable-net45+win8"
]
}
}
}

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/Azure/azure-sdk-for-net/issues/2552#issuecomment-261813909, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AO7kwRGkFBq18Uqfy1VU5SR2QapUuqUKks5rANRLgaJpZM4K0sJB.

[brjohnstmsft]

Thanks @laultman. This confirms that it is a more general issue not specific to the Azure Search SDK. I'll update the title of this issue accordingly and try to get the right people looking into it.

In the meantime, the workaround is to downgrade as @fuocor suggested.

[brjohnstmsft]

FYI @hovsepm @markcowl

[markcowl]

Description

Get a minimal repro of the issue and determine the root cause. File issues for fix.

Cost: 2

[efeurich]

Thanks guys you save me a lot of time.

[jsgoupil]

Ah snap, I have the same problem with Newtonsoft.Json v10.0.2
Downgrading to 9.0.1 fixes the problem.

[bmeredith]

Also seeing this issue with Newtonsoft.Json v10.0.2.

[blumu]

Is there any fix in sight for this bug? It's not always possible to downgrade to Newtonsoft v9.
For instance, F# 4.1 requires Newtonsoft.Json v10.0: https://github.com/JamesNK/Newtonsoft.Json/issues/1180

[brjohnstmsft]

@markcowl Any update on this?

[idg10]

Just in case it's a useful data point, I've encountered this in what might be an error scenario (with Json.NET 10.0.2) because it looks like a CloudErrorJsonConverter was involved:

System.AggregateException: One or more errors occurred. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Cannot get SerializationBinder because an ISerializationBinder was previously set.
   at Newtonsoft.Json.JsonSerializer.get_Binder()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
   at Microsoft.Rest.Azure.JsonSerializerExtensions.WithoutConverter(JsonSerializer serializer, JsonConverter converterToExclude)
   at Microsoft.Rest.Azure.CloudErrorJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Microsoft.Rest.Serialization.SafeJsonConvert.DeserializeObject[T](String json, JsonSerializerSettings settings)
   at Microsoft.Azure.Management.Compute.VirtualMachinesOperations.<GetWithHttpMessagesAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Valhalla.Web.SystemTests.Tests.Pre.DeprovisionSingleVmSet.<CheckVmInAzure>d__9.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Valhalla.Web.SystemTests.Tests.Pre.DeprovisionSingleVmSet.RunWeb()
   at Valhalla.Web.SystemTests.Tests.TestBaseWithWebDriver.RunCore()
---> (Inner Exception #0) System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Cannot get SerializationBinder because an ISerializationBinder was previously set.
   at Newtonsoft.Json.JsonSerializer.get_Binder()
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
   at Microsoft.Rest.Azure.JsonSerializerExtensions.WithoutConverter(JsonSerializer serializer, JsonConverter converterToExclude)
   at Microsoft.Rest.Azure.CloudErrorJsonConverter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Microsoft.Rest.Serialization.SafeJsonConvert.DeserializeObject[T](String json, JsonSerializerSettings settings)
   at Microsoft.Azure.Management.Compute.VirtualMachinesOperations.<GetWithHttpMessagesAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Valhalla.Web.SystemTests.Tests.Pre.DeprovisionSingleVmSet.<CheckVmInAzure>d__9.MoveNext()<---

[markcowl]

@fuocor @brjohnstmsft This is firmly in plan to look at in May, and from there we will have a plan to fix in runtime and/or generated code. It is absolutely in our mission to support the latest version

[parido]

This is great news as this issue is impacting our development here (downgrade JSON is not an option right now).
Thanks

[Bartmax]

thanks, just hit this. Saved me tons of times!

[galvesribeiro]

Guys, any updates on that? using v10+ still having the same error...

Thank you.

[brjohnstmsft]

@galvesribeiro Which Azure library are you using, and which version is it?

[galvesribeiro]

@brjohnstmsft

<PackageReference Include="Microsoft.Azure.Management.Fluent" Version="1.0.0" />
<PackageReference Include="Newtonsoft.Json" Version="9.0.1"/>

I can only get it to work with 9.0.1. I tried with 10+ and same error as state on this issue.

[brjohnstmsft]

@galvesribeiro It might be because Microsoft.Azure.Management.Fluent depends on an older version of Microsoft.Rest.ClientRuntime.Azure. Try adding a binding redirect to version 3.3.6 and see if that helps.

[galvesribeiro]

@brjohnstmsft I'm on .Net Core... No binding redirects :(

[brjohnstmsft]

@galvesribeiro Got it. Sorry for the inconvenience. :(

Since my last reply I've seen an update from @markcowl that this work is still in progress.

[galvesribeiro]

Ok, thanks @brjohnstmsft will keep the older version for now.

[bottomup]

Same here. The problem occurred when I added 2 fields to an index. I was on NewtonSoft.Json 10, a downgrade to 9 made it work again.

Probably it works when updating to NewtonSoft.Json 10 if the index is untouched. Changing the index causes the problem.

[brjohnstmsft]

@bottomup Note that this issue covers all Azure .NET libraries. It sounds like you're talking about the Azure Search library in particular -- is that correct? If so, which version are you using?

[bottomup]

@brjohnstmsft correct. I'm using Microsoft.Azure.Search 3.0.3.
However, I'm also using other Microsoft.Azure packages, but didn't experience these issues there.

[brjohnstmsft]

@bottomup Can you try upgrading to 3.0.4 and let me know if you still experience the problem?

[jbreuer]

I'm having the same issue with Azure Search. Upgraded to 3.0.4 and I get the error "Cannot get SerializationBinder because an ISerializationBinder was previously set.". Using Newtonsoft.Json version 10.0.2.

[galvesribeiro]

Its important to note that this is not a particular nuget issue... Any nuget from Azure SDK that is using 9.0.1 and you try to use 10.x.x+, you will get that problem...

[jbreuer]

Thanks for the info. So do we need to wait until a new version comes out that uses 10.x.x+ ? I can't downgrade to 9.

[galvesribeiro]

Yeah, unfortunately... I did rollback to 9+ for a particular project that use those packages in order to be unblocked...

[jbreuer]

I tried adding the following binding redirect, but it didn't work:

<dependentAssembly>
  <assemblyIdentity name="Microsoft.Rest.ClientRuntime.Azure" publicKeyToken="31bf3856ad364e35" />
  <bindingRedirect oldVersion="0.0.0.0-3.0.0.0" newVersion="3.0.0.0" />
</dependentAssembly>

@markcowl Any idea when a new version will be released which supports NewtonSoft.Json v10?

[bottomup]

@brjohnstmsft I tried your suggestion and upgraded to 3.0.4, but the same issue arises. Downgraded again to NewtonSoft.Json 9.0.1

[brjohnstmsft]

@bottomup Thanks for confirming. I wasn't sure if the issue with the Azure Search SDK is related to the issue affecting all Azure .NET SDKs, but it looks like it is related. @markcowl is working on a fix.

[isaacabraham]

Just adding to this - I see this, but only when I generate an invalid search query in the Azure Search SDK - then it comes back with this error (rather than the "real" error).

[hovsepm]

This is an issue in ClientRuntime.Azure and will be fixed shortly.

[brjohnstmsft]

@hovsepm Thanks Hovsep. The Azure Search SDK does some custom JSON serialization stuff as well, so once the issue in ClientRuntime.Azure is fixed, I still have to verify whether more work needs to be done to fix the issue in Microsoft.Azure.Search.

[galvesribeiro]

@hovsepm: Will #3304 solve the other nugets (like compute, networking, etc...) as well? As I mentioned, it is a general issue...

[markcowl]

We misunderstood the root cause for this, which is why we had the additional delay. Hovsep's fix will allow those using NetCore and those using binding redirects in net452 to use Newtonsoft.Json. Those who cannot use binding redirects will need us to republish the individual management libraries, which will happen next month.

@brjohnstmsft Code the depends on ClientRuntime exception handling should be fixed automatically with an upgrade. Code that directly uses properties deprecated in Version 10 will have to be updated.

[markcowl]

@galvesribeiro Those libraries are compatible with the new runtime, so you should be able to upgrade. Note that you will need to be using NetCore or assembly binding redirects to take this fix, because the assembly identity for Newtonsoft.Json has changed.

[galvesribeiro]

@markcowl thankfuly I'm free from binding redirect hell by using .Net Core :)

Thanks! Looking forward for the update.

[hovsepm]

The issue is resolved. Please use Microsoft.Rest.ClientRuntime.Azure v 3.3.8 or later.

[brjohnstmsft]

@hovsepm @markcowl Not sure this should be closed until the libraries depending on Microsoft.Rest.ClientRuntime.Azure have been republished, unless there is another issue tracking this work...? I'd love to republish the Search libraries, but I see that AzSdk.reference.props in the vs17Dev branch hasn't been updated with the new references to Microsoft.Rest.ClientRuntime.Azure and NewtonSoft.Json yet. What's the ETA for this?

[hovsepm]

@shahabhijeet should also have an idea when that could happen.
@brjohnstmsft you should not need to publish new library. Does Search has direct dependency on any version?

[brjohnstmsft]

@hovsepm No, but the minimum required version is 3.3.6, not 3.3.8. A few of our customers have already reported running into this issue as soon as they upgrade to JSON.NET 10. Referencing 3.3.8 resolves it, but in the meantime it creates extra support traffic for us.

[shahabhijeet]

@brjohnstmsft following [fix] (https://github.com/JamesNK/Newtonsoft.Json/pull/1341) fixes underlying error. New set of Clientruntime libraries are on track to be republished by next week.
In the mean time, upgrading to newtonsoft.json 10.0.3 will fix the issue whoever wants to upgrade to newtonsoft 10+

[brjohnstmsft]

@shahabhijeet Awesome, thanks for the update!