Orleans: [question] Exception deserialization

[peter-perot]

Lately I got a strange exception in my logs that read TypeLoadException: Could not find EntityFramework.dll.

After some investigation it turned out that my data access layer (which is referenced by my Orleans server-side code) issued an EntityException which bubbled up to Orleans, travelled over the wire, but on the client it could not be deserialized since the client does not (and should not) reference EntityFramework.

I'm not interested in the concrete exception type – simply want to log the _unknown_ exception with ex.ToString() on my client.

What is the best way to accomplish this? I don't want to repackage each possible exception into a "client-side readable" well-known exception in (server-side) grain code like this:

try
{
  // ...
} catch (Exception e)
{
  // send a client-side known exception over the wire
  throw new ClientSideKnownException(e.ToString());
}

Any suggestion how to handle this?

[sergeybykov]

I can't think of a better way. This requirement goes contrary to the general desire of the programming model to deliver the exact exception to the caller as if it was thrown locally.

In theory, client could tell silos what assemblies are available, so that silos would detect exceptions that cannot be deserialized on the client side, and replace them with a generic one. However, the same limitation applies to any type returned by a grain - the type will fail to deserialize unless it is loaded on the client side. So even if we had a special treatment for exceptions, one would still be exposed to the more general issue.

[shayhatsor]

Adding to what @sergeybykov said, an Orleans client is more like a "proxy to an Orleans cluster" than just a simple client. Both client and silos must have the same DTOs, and in this case, exceptions. The problem here arises from the fact that everything is packed in EntityFramework.dll, if there was EntityFramework.Exceptions.dll you'd just reference that on the client side. My suggestion is to figure out which special exceptions can be thrown from EntityFramework.dll and repack them in an EntityFrameworkException, which will be referenced from both sides.

[sergeybykov]

@peter-perot What's the harm of having EntityFramework.dll on the client other than a bit unclean layout of the binaries?

[gabikliot]

Alternatively, you should be able to write your own serializer (we now support "External serializers"), where you simply delegate the serialization for all types to Orleans except for this one type from EntityFramework.dll which you serialize yourself into a simple Exception.

[peter-perot]

Adding the entity framework to the client would be possible, but I consider it a little hacky. I agree to what @shayhatsor said: exceptions are sort of a DTO - but then again there are some things I want to hide from the client: implementation details of (for example) the data access layer.

That said an EntityException should never bubble up to the client _as an EntityException_ since exception types which a consumer is not aware of cannot be handled in a satisfying manner. In this case I would prefer it to bubble up as some kind of _unknown exception_ which the client does not need to (and cannot) handle - but should log.

I think the approach @gabikliot suggested fits my needs best. So I'll have a look into the unit tests for _external serializers_ to get an idea how to implement this.

[peter-perot]

Can't get my "delegating" external serializer working. I registered the serializer in client and server config this way:

Client:

<?xml version="1.0" encoding="utf-8"?>
<!--
    This is a sample client configuration file.
    For a detailed reference, see "Orleans Configuration Reference.html".
-->
<ClientConfiguration xmlns="urn:orleans">
    <!-- [...] -->
    <Messaging ResponseTimeout="30s" ClientSenderBuckets="8192" MaxResendCount="0">
        <SerializationProviders>
            <Provider type="MyTemplate.Services.GrainInterfaces.MyExternalSerializer, MyTemplate.Services.GrainInterfaces" />
        </SerializationProviders>
    </Messaging>
    <!-- [...] -->
</ClientConfiguration>

Server:

<?xml version="1.0" encoding="utf-8"?>
<OrleansConfiguration xmlns="urn:orleans">
    <Globals>
    <!-- [...] -->
        <Messaging>
            <SerializationProviders>
                <Provider type="MyTemplate.Services.GrainInterfaces.MyExternalSerializer, MyTemplate.Services.GrainInterfaces" />
            </SerializationProviders>
        </Messaging>
    </Globals>
    <!-- [...] -->
</OrleansConfiguration>

Now I tried to write a simple "delegating" serializer, i.e. a serializer which does nothing than to intercept each call and delegate it to standard Orleans serialization. But I get a StackOverflowException in DeepCopy:

using System;
using Orleans.Runtime;
using Orleans.Serialization;

namespace MyTemplate.Services.GrainInterfaces
{
    public class MyExternalSerializer : IExternalSerializer
    {
        public MyExternalSerializer()
        {
        }

        protected TraceLogger Logger { get; private set; }

        public void Initialize(TraceLogger logger)
        {
            this.Logger = logger;
        }

        public bool IsSupportedType(Type itemType)
        {
            return true;
        }

        public object DeepCopy(object source)
        {
            return SerializationManager.DeepCopyInner(source);
        }

        public object Deserialize(Type expectedType, BinaryTokenStreamReader reader)
        {
            return SerializationManager.DeserializeInner(expectedType, reader);
        }

        public void Serialize(object item, BinaryTokenStreamWriter writer, Type expectedType)
        {
            SerializationManager.SerializeInner(item, writer, expectedType);
        }
    }
}

How can I intercept e.g. an EntityException and let the Orleans serializer treat it like an Exception?

[peter-perot]

Update: In the meantime I used the "hacky" solution by just letting my Orleans client (an ASP.NET REST API) reference both the grain interface and the implementation assembly. This way exceptions from EntityFramwork and some other 3rd party assemblies will not result in a senseless TypeLoadException so that the REST API can log properly.

I consider these additional references to be _runtime/deployment references_ – they can be deleted and the project will/should still compile, but during runtime they are required just for deserializing some "alien" exceptions for logging purposes.

Nevertheless am I interested in the _external serializer_ approach.

[sergeybykov]

If IsSupportedType returns true for all types, your serializer takes responsibility for serializing all of them. I think you want to return true only for the exception type you want to alter serialization of.

By calling SerializationManager from within your serializer's method you are creating an infinite loop because IsSupportedType returns true, hence tells SerializationManager to invoke your serializer for those types.

What I think you need to do is instantiate a different type that you want to use as a substitute, and then pass that instance to SerializationManager to deep-copy and serialize it.

[peter-perot]

Yeah, I also had the idea of using a surrogate object wrapping the one which needs special serialization. The problem I have is that the service actually doesn't know what exception types are unknown to the client, i.e. must be serialized as an unspecific exception.

[sergeybykov]

You could also try to look at assembly name where the exception type is defined, and skip those that are defined in the system assemblies that you for sure have on the client side.

[peter-perot]

You could also try to look at assembly name [...]

This could work, but AFAIK there are some NuGet packages which use System.* as namespace so it would be more accurate to check the assembly (if it is a "standard" .NET FX assembly).

The problem is: We have some additional assemblies which contain some custom exception types. Some clients reference these assemblies, some not.

I think the most flexible solution would be to wrap each exception in a wrapper exception (on server side) which contains the serialized original exception in a special property.

On client side it is tried to deserialize the wrapper exception's "special property". If this succeeds, the original exception is taken/thrown; otherwise, the wrapper exception (or some other custom exception which indicates that a non-deserializable exception has been caught) is (re-)thrown.

Do you think the Orleans serialization framework is capable of handling this? I'm not quite sure if my approach could be realized this way.

[peter-perot]

To be more specific:

Lets say on server side an exception occured and is wrapped in a WrapperException this way (I use JSON notation just for clarification):

{
  "Message": "An error occured.",
  "OriginalExceptionPayload": "VGhlIHF1aWNrIGJyb3duIGZveCBqdW1wcyBvdmVyIHRoZSBsYXp5IGRvZy4="
}

This exception will be filtered on client side. If the deserialization of the Payload succeeds, the resulting exception is thrown; otherwise the WrapperException is rethrown.

[sergeybykov]

You could try to configure a custom serializer that would ignore everything but the WrapperException type, and deserialize it conditionally based on the content into a WrapperException or the original exception type. I don't know off the top of my head if the substitution by a different type will work out of the box or some modifications of the SerializationManager code will be needed.

[peter-perot]

Hmm... There must be something wrong with the docs. The following code does not compile since some methods are internal:

int reference = SerializationContext.Current.CheckObjectWhileSerializing(item);

if (reference >= 0)
{
    writer.WriteReference(reference);
    return;
}

See http://dotnet.github.io/orleans/Advanced-Concepts/Serialization.

Has there been an API change?

[peter-perot]

Okay, here I've tried to write my own serializer (ignoring the internal methods which I cannot call) which tries to serialize any exception in a simple way (no exception wrapping as mentioned in this thread above - just wanted to understand how serialization works in Orleans).

public class MyExternalSerializer : IExternalSerializer
{
    public MyExternalSerializer()
    {
    }

    protected TraceLogger Logger { get; private set; }

    public object DeepCopy(object source)
    {
        using (var stream = new MemoryStream())
        {
            var formatter = new BinaryFormatter();
            formatter.Serialize(stream, source);

            stream.Position = 0;

            var result = formatter.Deserialize(stream);
            SerializationContext.Current.RecordObject(source, result);

            return result;
        }
    }

    public object Deserialize(Type expectedType, BinaryTokenStreamReader reader)
    {
        var count = reader.ReadInt();
        var data = reader.ReadBytes(count);

        using (var stream = new MemoryStream(data))
        {
            var result = new BinaryFormatter().Deserialize(stream);
            DeserializationContext.Current.RecordObject(result);

            return result;
        }
    }

    public void Initialize(TraceLogger logger)
    {
        this.Logger = logger;
    }

    public bool IsSupportedType(Type itemType)
    {
        return typeof(Exception).IsAssignableFrom(itemType);
    }

    public void Serialize(object item, BinaryTokenStreamWriter writer, Type expectedType)
    {
        using (var stream = new MemoryStream())
        {
            new BinaryFormatter().Serialize(stream, item);
            var data = stream.ToArray();

            writer.Write(data.Length);
            writer.Write(data);
        }
    }
}

Now I throw an exception from Orleans silo: throw new Exception("Something went wrong.")

On client side I get this exception:

System.TypeAccessException: Named type "Exception" is invalid: Type string "Exception" cannot be resolved.
   at Orleans.Serialization.BinaryTokenStreamReader.ReadSpecifiedTypeHeader()
   at Orleans.Serialization.SerializationManager.DeserializeInner(Type expected, BinaryTokenStreamReader stream)
   at Orleans.Serialization.BuiltInTypes.DeserializeOrleansResponse(Type expected, BinaryTokenStreamReader stream)
   at Orleans.Serialization.SerializationManager.DeserializeInner(Type expected, BinaryTokenStreamReader stream)
   at Orleans.Serialization.SerializationManager.Deserialize(Type t, BinaryTokenStreamReader stream)
   at Orleans.Runtime.Message.DeserializeBody(List`1 bytes)
   at Orleans.Runtime.Message.get_BodyObject()
   at Orleans.Runtime.GrainReference.ResponseCallback(Message message, TaskCompletionSource`1 context)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Orleans.Runtime.GrainReference.<InvokeMethodAsync>d__42`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at WebTemplate.AuthorizationService.Authorization.ApplicationOAuthServerProvider.<GrantResourceOwnerCredentials>d__8.MoveNext() in V:\WebTemplate\WebApi\AuthorizationService\Authorization\ApplicationOAuthServerProvider.cs:line 92

Why invalid type string "Exception"? Don't understand what's wrong here?

What is a type header? Is it written by Orleans automatically?

[peter-perot]

What is strange here: The code never runs into the Deserialize method. And yes: I registered the external serializer both in the server and in the client config file.

[peter-perot]

I digged a little bit deeper into the serialization topic. Now I wrote a serializer in a "standard" serializer class (which is not an _external_ serializer, and which must be registered with the RegisterSerializer attribute):

[RegisterSerializer]
public class MySerializer
{
    public static void Register()
    {
        SerializationManager.Register(typeof(Exception), DeepCopy, Serialize, Deserialize);
    }

    static MySerializer()
    {
        Register();
    }

    public static object DeepCopy(object source)
    {
        using (var stream = new MemoryStream())
        {
            var formatter = new BinaryFormatter();
            formatter.Serialize(stream, source);

            stream.Position = 0;

            var result = formatter.Deserialize(stream);
            SerializationContext.Current.RecordObject(source, result);

            return result;
        }
    }

    public static object Deserialize(Type expectedType, BinaryTokenStreamReader reader)
    {
        var count = reader.ReadInt();
        var data = reader.ReadBytes(count);

        using (var stream = new MemoryStream(data))
        {
            var result = new BinaryFormatter().Deserialize(stream);
            DeserializationContext.Current.RecordObject(result);

            return result;
        }
    }

    public static void Serialize(object item, BinaryTokenStreamWriter writer, Type expectedType)
    {
        using (var stream = new MemoryStream())
        {
            new BinaryFormatter().Serialize(stream, item);
            var data = stream.ToArray();

            writer.Write(data.Length);
            writer.Write(data);
        }
    }
}

The code in the methods is exactly the same as in the _external_ serializer.

The good news is: This serializer works!

The bad news: First, it only works for the type Exception, but not for derived types (which is by design I think). Second, IMHO there seems to be a bug in the external serialization framework which has a different behavior compared to the other serializer using the static methods and the registration attribute.

[peter-perot]

A solution could be to scan all assemblies for exception classes and register the same serialization callbacks for all of them. However I think there must be a solution taking the much more elegant approach using external serializers but there seems to be a quirk with the type headers.

[peter-perot]

@sergeybykov, @gabikliot: I did not come to a solution using external serializers; the docs concerning serialization seem to be outdated, since method CheckObjectWhileSerialzing does not exist (or is internal).

However, the problem was not prio one in our project, so we came around by copying all the assemblies of the Orleans service implementation to the web service folder.

Today I revisited the docs and read the "interceptor" chapter. My question is: What about putting a try-catch-block into a silo-level interception method and rethrowing those exceptions which are unknown from the client's perspective (e.g. EntityException)? If you don't veto, I would give it a try.

[peter-perot]

Good news: yesterday I tried the interceptor approach. Works like a charme. Simply remap all exceptions which are neither part of the basic .NET assemblies (typically in GAC) nor part of the project's DTO assemblies to Exception (or some special well-known exception type).

[sergeybykov]

Today I revisited the docs and read the "interceptor" chapter. My question is: What about putting a try-catch-block into a silo-level interception method and rethrowing those exceptions which are unknown from the client's perspective (e.g. EntityException)? If you don't veto, I would give it a try.

Seems like another use case for interceptors. Great that it already worked for you. If you could add a paragraph to the interceptors docs, that would make it a potentially reusable pattern for others.

[peter-perot]

If you could add a paragraph to the interceptors docs, [...]

Ok. Please give me a hint how to get the sources with mark down. When I clone my Orleans fork and go to the gh-pages branch I get the HTML files, but not the MD files. Are they in another repository?

[peter-perot]

Sorry, I found the MD files under ...\src\Documentation\.... I'll try to get along...

[peter-perot]

See pull request #2286.

[sergeybykov]

Merged. Thank you very much!

[peter-perot]

@sergeybykov:

There is a subtle caveat I have overlooked: my solution is unable to figure out whether the call comes from the grain client or from another grain.

Even the Orleans infrastructure makes grain calls, e.g. to the GrainBasedReminderTable grain, and I think we don't want to hide some special exceptions which the caller (another grain or the infrastructure) might handle.

I'll make another pull request. [Hint: I don't know if I can reopen an already merged PR.]

[peter-perot]

See PR #2296.