Hangfire: Intermittent JsonSerializationException when processing background tasks .net core 2.0

[Yoztastic]

After a transient job failure and subsequent re-queue we get one or more errors see stack trace below. This is followed by a random number of failed Retry Attempts 0 - 10 (10 is very rare 1 in 10000).

This can not be a malformed json as it eventually Deserializes (99.9999%). When it fails, It always fails at line 1, position 149 in our case.

Have seen similar issue raised here:
https://discuss.hangfire.io/t/could-not-cast-or-convert-from-system-string-to-system-type/4240

Failed
Can not change the state to 'Enqueued': target method was not found.

Newtonsoft.Json.JsonSerializationException
Error converting value "MarketingPreferences.Host.Events.Vortex.PrivacyServiceGetConsent, MarketingPreferences.Host, Version=1.8.3.0, Culture=neutral, PublicKeyToken=null" to type 'System.Type'. Path '[0]', line 1, position 149.

Newtonsoft.Json.JsonSerializationException: Error converting value "MarketingPreferences.Host.Events.Vortex.PrivacyServiceGetConsent, MarketingPreferences.Host, Version=1.8.3.0, Culture=neutral, PublicKeyToken=null" to type 'System.Type'. Path '[0]', line 1, position 149. ---> System.ArgumentException: Could not cast or convert from System.String to System.Type.
at Newtonsoft.Json.Utilities.ConvertUtils.EnsureTypeAssignable(Object value, Type initialType, Type targetType)
at Newtonsoft.Json.Utilities.ConvertUtils.ConvertOrCast(Object initialValue, CultureInfo culture, Type targetType)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)
--- End of inner exception stack trace ---
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EnsureType(JsonReader reader, Object value, CultureInfo culture, JsonContract contract, Type targetType)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObjectT
at Hangfire.Common.JobHelper.FromJsonT
at Hangfire.Storage.InvocationData.Deserialize()

[tsu1980]

I'm faced with the same problem.

This is the assembly name mismatch in cross framework problem.

For example. If the job method is

void Run(int id, string name);

If .NET Core client enqueued the job, ParameterTypes value is like this

["System.Int32, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","System.String, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"]

If .NET Framework client enqueued the job, ParameterTypes value is like this

["System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089","System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"]

When Hangfire server fetches and parses the ParameterTypes value. if assembly name is different in Hangfire server and Hangfire client, failed to parse job information and enter retrying.

My solution is here.

patch InvocationData.cs to remove assembly name if its another side.

        static bool isMscorlib = typeof(int).AssemblyQualifiedName.Contains("mscorlib");

        public Job Deserialize()
        {
            try
            {
                var type = System.Type.GetType(Type, throwOnError: true, ignoreCase: true);

                // Remove assembly name if its another side for cross framework compatibility
                var parameterTypesReplaced = ParameterTypes;
                if (isMscorlib)
                {
                    parameterTypesReplaced = parameterTypesReplaced.Replace(", System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e", "");
                }
                else
                {
                    parameterTypesReplaced = parameterTypesReplaced.Replace(", mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089", "");
                }
                var parameterTypes = JobHelper.FromJson<Type[]>(parameterTypesReplaced);

                var method = type.GetNonOpenMatchingMethod(Method, parameterTypes);
...

I'm not sure that would be enough, but its working so far.
Hope this help.

[odinserj]

These issues fixed with the two-step process in 1.6.22 and latest 1.7.0 betas. With the former release we started to replace System.Private.CoreLib to mscorlib during the de-serialization process, and with the latter it's possible to change the serialization process to serialize with the mscorlib assembly if type is forwarded from it to allow interoperability for even more classes.

You can enable the new serialization format in the following way, but ensure first all of your servers migrated to 1.7.0-*.

GlobalConfiguration.Configuration
    .UseSimpleAssemblyNameTypeSerializer()

[imuller]

@odinserj Still got this exception on the first time a job is executing. Second time the job runs fine. For every new task i got the exception once. I've upgraded all my projects to 1.6.22

Can not change the state to 'Processing': target method was not found.

Error converting value "Attributes, Business, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" to type 'System.Type'. Path '[0]', line 1, position 149.

[imuller]

@odinserj Is there any changes needed for 1.6.22 to get the new serialization format to work? The UseSimpleAssemblyNameTypeSerializer is not available for 1.6.22

[odinserj]

In 1.7.0 there're new type resolver and serializer that's plugged into both JSON converter and Hangfire itself. When using SimpleAssemblyNameTypeSerializer (via GlobalConfiguration.Configuration), it handles TypeForwardedFrom attributes in .NET Core to serialize the type name to the format .NET Framework understands. And when working with types, serialized in .NET Framework, .NET Core itself is handling them, thanks to the TypeForwardedFromAttribute.

New version is already released, please see the upgrade guide: https://docs.hangfire.io/en/latest/upgrade-guides/upgrading-to-hangfire-1.7.html.