Orleans: Different de/serializers used for grain interface vs streams

[amccool]

given a silo and a client and a POCO project with type T. The silo and client communicate only by streams using T.

[Serializable]
public class T
    {
        public byte[] PhoneNumber { get; set; }
    }

The client does not include a reference to the grain interface project.

If a developer adds T as a argument on a grain method, then when the client goes to deserialize T from the stream they get

System.Runtime.Serialization.SerializationException: Unsupported type 'T' encountered. Perhaps you need to mark it [Serializable] or define a custom serializer for it?
at Orleans.Serialization.SerializationManager.DeserializeInner(Type expected, BinaryTokenStreamReader stream)
at Orleans.Serialization.BuiltInTypes.DeserializeImmutableT
at Orleans.Serialization.SerializationManager.DeserializeInner(Type expected, BinaryTokenStreamReader stream)
at Orleans.Serialization.BuiltInTypes.DeserializeInvokeMethodRequest(Type expected, BinaryTokenStreamReader stream)
at Orleans.Serialization.SerializationManager.DeserializeInner(Type expected, BinaryTokenStreamReader stream)
at Orleans.Serialization.SerializationManager.Deserialize(Type t, BinaryTokenStreamReader stream)
at Orleans.Runtime.Message.DeserializeBody(List`1 bytes)

It appears that because the silo is using the codegen version of T, and the client is using the POCO version of T, the two dont agree.

BTW - I could not find a way to create this problem in a unit test, because the unit test contained the grain interface, thus making both the silo and client use the same T

[sergeybykov]

The client does not include a reference to the grain interface project.

How does the client communicate with the grains then? Does the issue go away if you include the grain interface assembly on the client?

[amccool]

The client does not communicate with the grain at all.

We solved the issue two different ways. We are going with the 2nd solution so the T assembly is as pure as possible.

• One way - By adding the [Immutable] to T
• Second way - by removing T from the grain interface

At this point we are thinking that we will have an entirely separate assembly for stream messages, and that assembly cannot be referenced in the grain interface assemblies. (probably a good case for a roslyn analyzer to make sure no developer break the "rules")

[sergeybykov]

Interesting. So the client only communicates via streams? And you don't want to add the grain interface assembly on the client to keep it clean? Do I understand correctly?

Sounds like you are fighting the "over-eager" codegen that tries to generate serializers for all types that might go on the wire. Streaming will likely benefit from the generated serializer, performance wise and in case there is no fallback one (CoreCLR 1.0, type not marked as [Serializable]). But I guess you don't care? Would it help if you could generate a serializer for T outside of the grain interface assembly?

[amccool]

Thats correct. We are looking for extremely simple clients. Only using streams like a service bus.

How does one "make" the stream de/serializer smart enough to notice that its not the fallback(did I get that right?) deserializer being used, and its the codegen'd deserializer that needs to be used?

Seems like once T is touched by codegen - then the entire code base must use the codegen'd T.

[ReubenBond]

@amccool I'm trying to work out why this might be happening. Do you have an Assembly-level KnownType attribute for T in the client assembly or any assembly the client is referencing?

[amccool]

we tried the KnownType, but that did not help.

the client does have a reference to T, but it does not have a reference to the grain interface assembly which referenced T.

lets see if I can draw:

this is broken

but this worked (because T was not codegen'd)

[ReubenBond]

That's strange. Your client assembly references Orleans, right?
Is there any reason you can see for this code to return false?

c# return !assembly.IsDynamic && !CompiledAssemblies.ContainsKey(assembly.GetName().FullName) && TypeUtils.IsOrleansOrReferencesOrleans(assembly) && assembly.GetCustomAttribute<GeneratedCodeAttribute>() == null && assembly.GetCustomAttribute<SkipCodeGenerationAttribute>() == null;

Could you post the client logs (maybe with logging cranked up)?

[amccool]

client references orleans - yes, for streams

that code, I think should return true for assemblies in the client.

[ReubenBond]

@amccool it needs to return true :stuck_out_tongue: What do you think makes it return false?

[amccool]

@ReubenBond I edited the answer .... sorry. I think it does return true.

[sergeybykov]

@ReubenBond My understanding is that this happens because the grain interface assembly, which contains the generated serializer, is deployed with the silo but not with the client. Hence, silo uses the serializer to send T to the client, but the client doesn't have the serializer to deserialize it.

[amccool]

@sergeybykov I believe your understanding to be correct. For the long term fix it would seem that there needs to be synergy of serialization between the grain communication code and the stream communication code. which is independent of the codegen.

I am happy to table this issue for now, as we are creating a better enforcement of our contract class useage (one for grain comm, one for stream comm, and do not mix). However I am sure someone else will run into this.

[sergeybykov]

@amccool Thanks for confirming it. I see three possible options:

1) Add an option to suppress generation of a serializer for a type that otherwise would get generated;
2) Support generating serializers for types outside of grain interface/class assemblies;
3) Run time codegen.

. 1) seems ugly and error prone to me. 2) seems a like better path, but it would require a way to add codegen to an arbitrary project. 3) could be done for types pointed to with [GenerateSerializer].

Any other ideas?

[ReubenBond]

Ohh, @amccool are you not using runtime codegen at all? In that case this makes more sense. I'd enable runtime codegen even if you're also doing compile-time codegen just to act as a catch-all (eg, for assemblies which are loaded dynamically, or this case).

Even still, adding [KnownType(typeof(T))] in the client assembly should cause this to be generated when generating the client assembly.

Could you post a Verbose2-level log?

[gabikliot]

Would not a better way to handle this be to use custom serializer? He will be able to specify his own strategy to serialize this particular type, while delegating all other types to Orleans serializer. That makes the fact if Orleans does generate its serializer or not for that type irrelevant.

[ReubenBond]

@gabikliot I was under the impression that @amccool doesn't want to worry about writing serializers, but I think you're right that it would work as a workaround.

I've submitted PR #1888, which gives the codegen attributes more obvious semantics, but I still need to understand precisely why you're running into an issue, @amccool.

[gabikliot]

I don't consider that a workaround. There is a usual 95% case, and there is the exceptional special case.I think @amccool is definitely the exceptional case. We should allow that, for sure, but its reasonable to ask for a bit more work on the application side for exceptional case. We on our side should make writing custom serializers a breeze. If that is not the case now, as I suspect, we should improve it.

[ReubenBond]

@gabikliot I consider this a bug on our end - consumers should be able to communicate using streams alone and not have to reference a grain interface assembly.

The problem is that we cannot see what generic parameters will be passed through the stream during the codegen phase, so we simply cannot know what might get passed... We could make a Roslyn code analyzer for such a thing, but that's overkill and we're better off just letting them hint it to us.

[gabikliot]

Ok, then I agree that "adding [KnownType(typeof(T))] in the client assembly should cause this to be generated when generating the client assembly."
This should work, right?

[jeoffman]

We did try using [assembly: KnownType(...)] in the AssemblyInfo.cs but it did not help. I'm working on getting some logs from some of our various attempts to work around the issue.

[jeoffman]

Sample project at: https://github.com/jeoffman/OrleansSerializationExceptionTest

You can run that code as-is to see the error and comment-out the method "ThisMethodBreaksTheClient" to see it work.

I also have 4 log files ((client + silo) * (working + failing)) if you would prefer to see those. The client logs are pretty boring, but the server logs have a few entries of interest. I ran them through a DIFF and the failing silo has a few extra lines from the SerializationManager (below). I marked the lines which the "fail" silo has but the "success" silo does not with some asterisks (**)

[2016-06-30 13:00:46.770 GMT     9  INFO    100000  AssemblyLoader.Silo 127.0.0.1:11111]    Loading assembly D:\src\jkh\C#\SerializationExceptionTest\SiloHost\bin\Debug\GrainActor.dll...  
[2016-06-30 13:00:46.795 GMT     9  VERBOSE     100000  CodeGenerator   127.0.0.1:11111]    Generating code for assemblies: GrainActor, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null   
[2016-06-30 13:00:46.810 GMT     9  VERBOSE     100000  CodeGenerator   127.0.0.1:11111]    Generating code for assemblies: GrainInterface, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null   
[2016-06-30 13:00:46.817 GMT     9  VERBOSE-2   100000  CodeGenerator   127.0.0.1:11111]    Will generate code for: global::GrainInterface.IMyInterface 
[2016-06-30 13:00:46.820 GMT     9  VERBOSE-2   100000  CodeGenerator   127.0.0.1:11111]    Generating code for: global::GrainInterface.IMyInterface    
[2016-06-30 13:00:46.820 GMT     9  VERBOSE-2   100000  CodeGenerator   127.0.0.1:11111]    Generating GrainReference and MethodInvoker for global::GrainInterface.IMyInterface 
**[2016-06-30 13:00:46.827 GMT     9    VERBOSE-2   100000  SerializationManager    127.0.0.1:11111]    Scanning assembly D:\src\jkh\C#\SerializationExceptionTest\SiloHost\bin\Debug\MessageContract.dll for serialization info**  
**[2016-06-30 13:00:47.012 GMT     9    VERBOSE-2   100000  CodeGenerator   127.0.0.1:11111]    Generating & Registering Serializer for Type global::MessageContract.SimpleMessage**    
[2016-06-30 13:00:48.182 GMT     9  VERBOSE     103802  CodeGenerator   127.0.0.1:11111]    Compilation of assembly OrleansCodeGen.dll succeeded.   
[2016-06-30 13:00:48.186 GMT     9  VERBOSE-2   100000  SerializationManager    127.0.0.1:11111]    Scanning assembly  for serialization info   
[2016-06-30 13:00:48.187 GMT     9  VERBOSE-2   100000  SerializationManager    127.0.0.1:11111]    Scanning assembly  for serialization info   
**[2016-06-30 13:00:48.187 GMT     9    VERBOSE-2   100000  SerializationManager    127.0.0.1:11111]    Scanning assembly  for serialization info** 
[2016-06-30 13:00:48.188 GMT     9  VERBOSE-2   103802  CodeGenerator   127.0.0.1:11111]    Generated code for 1 assembly in 1377ms 
[2016-06-30 13:00:48.188 GMT     9  VERBOSE-2   100000  SerializationManager    127.0.0.1:11111]    Scanning assembly D:\src\jkh\C#\SerializationExceptionTest\SiloHost\bin\Debug\GrainInterface.dll for serialization info 

[jeoffman]

UPDATE: I have been tinkering with [assembly: KnownType(typeof(SimpleMessage))]. It ends up this does solve the problem when added to the TestClient AssemblyInfo.cs. I believe that before I was using it in the MessageContract project.

[jeoffman]

Another note - within the client's AssemblyInfo.cs I added a "using Orleans.CodeGeneration;" for the KnownType reference. This is NOT the "KnownType" from "using System.Runtime.Serialization;". Maybe if that attribute was "KnownOrleansType" then it would have been clear to me.

[ReubenBond]

@jeoffman Thank you for that! I agree that it's confusing. Maybe we should obsolete it. In the next release of Orleans you will be able to use [ConsiderForCodeGeneration] which is more explicit & unique.

[sergeybykov]

I like the idea to obsolete [KnownType] and replace it with [ConsiderForCodeGeneration].

[sergeybykov]

@ReubenBond, isn't this already resolved with the new IL-based fallback serializer?

[ReubenBond]

Correct, @sergeybykov, this should be solved with the new IL-based fallback serializer (which is _not_ the default on full .NET).

Closing because I believe this to be resolved in the latest release. We can reopen if it's still troublesome.

[jeoffman]

Sadly this issue came back up for us this week = except this time it was a POCO class that was used on a stream and in a GrainState.

[sergeybykov]

@jeoffman Can you provide more details?

[jeoffman]

The failure has been intermittent for the past few weeks - sometimes it would "fix itself" on my development machine if I delete the bin+obj folders and reboot my machine. The test has never failed on our build server. Yesterday I was not able to "fix" the test with a reboot, and eventually tracked the problem down to having a POCO that was used as a stream message and also saved in a Grain's state.

Here is a github repo that reproduces the failure (on my machine, at least):
https://github.com/jeoffman/OrleansSerializationExceptionTesting_1881_3405

The failing test is named "SerializationExceptionFromClassUsedInStreamsAndAlsoStateTestTest" (the ONLY test). There is a #define in the grain .cs file that removes the POCO from the state and then the test will pass. See "CAUSE_THE_TEST_TO_FAIL_WITH_SERIALIZATION_EXCEPTION_SOMETIMES" in GrainWithStateUsingContract.cs

[ReubenBond]

This should not be an issue in 2.0:

• Assemblies with generated code are added to the client/silo builder by the developer explicitly.
• There are attributes customizing code generation so that a serializer is generated for your type, for example: [KnownAssembly(asm)], [GenerateSerializer(type)], and [KnownType(type)].
• Runtime code generation is now explicit and opt-in, not opt-out

@jeoffman apologies for the late reply - I can help you through any troubles you're encountering.

EDIT: I'll close this again for now