Wcf: Consider shipping System.Private.ServiceModel crossgened

[KKhurin]

This will shave off roughly 800ms off the first cold request (or multiple requests if many requests are issued while the code is still being jitted).

We also need to investigate why manually crossgen-ing already shipped binaries didn't improve the timing.

[gmaggiodev]

Any news about this bug? We are still experiencing low performance in multiple requests scenario.
Thanks

[KKhurin]

Stephen, did you have a chance to take a look at the build tools to see how its done for other assemblies?

[StephenBonikowsky]

Not yet, this is going to have to wait until I can get a vanilla build going with the new Arcade.

[gmaggiodev]

I'm sorry to bore you but we are really in trouble. In production environment our customers are experiencing random system blocking when stressing the system with multiple requests. During the blocking period (which lasts about 5 minutes) server CPU is not used but the no request is served anymore. We are so in trouble that we are thinking to abandon .net core in favor of .net standard but, for me, this is a defeat. I don't want to do this step backward.
Please help us!

[gmaggiodev]

Sorry, With .net standard I mean .net full framework (4.7.2).

[KKhurin]

@GianlucaMaggio, 5 minute long blocking periods are not related to jitting .NET core code. Jitting would take 800ms on average and will happen only once per process.

Is this Windows or Linux? Did you use the exact same app on full framework?

Do you believe its the client code that stops sending the requests or the server part that stops sending the responses? Do you see any errors or exceptions on either side? The easiest way is to attach windbg to the process and enable managed exceptions (sxe clr or sxe clr "k; g").

If you monitor the production machines can you take dumps when the blocking occurs? You can use procdump to semi-automate this dump collection. Once you take the dump - open it in windbg and look at all the call stacks (~*k99), check native locks (!locks), check managed locks (!syncblk). In many cases this will already reveal the culprit. If you know the structure of the code - find (via !dumpheap - type) and inspect the state of channels: are they open, closed, faulted? Are there any timeouts? Are there any exceptions in the managed heap? Is the server requested by the clients healthy (e.g. its heap does not contain lots of unreplied requests)?

Once you collect more details and conclude that this is a problem on the cliend side, could you open a new bug?

[StephenBonikowsky]

@GianlucaMaggio do follow @KKhurin 's guidance, your problem does not appear to be related to whether our assemblies are crossgened or not.

Regarding crossgening, we have not yet decided if it is something we will do since we would have to do it for each platform, architecture and distro which would mean we have to create new packages or greatly increase the size of our existing packages.

[gmaggiodev]

@StephenBonikowsky, yes you are right. Like @KKhurin said 5 minutes blocking is another story. By setting up the serviceThrottling in this way:

<serviceThrottling maxConcurrentCalls="192"
                   maxConcurrentInstances="192"
                   maxConcurrentSessions="1200" />

seems the problem disappear.

Anyway, the jitting problem (initial delay) is still annoying. Please consider other solutions to address it.

Thanks

[cspital]

@StephenBonikowsky

Hi, I'm curious how progress on this is going? I've been looking into this issue lately, as we have to integrate with a large (180k lines) auto-generated soap client for a vendor, and it seems like the JIT cost is correlated with the size of the Reference.cs file generated by the wsdl. In our current situation we have a few different class library projects, auto-generated via dotnet-svcutil and running in dotnet core 3.1.

Library A

• 180k lines of auto-generated code in Reference.cs.
• System.ServiceModel dependencies are at 4.7.*.
• First call takes between 15-20 seconds.
  
  
  
  • All of this time is spent pre-network IO.
  
  
  • Network IO takes about .5 seconds.
  
  

Library B

• 26k lines of auto-generated code in Reference.cs
• System.ServiceModel dependencies are at 4.7.*.
• First call takes between 1 - 2 seconds.
  
  
  
  • All of this time is spent pre-network IO.
  
  
  • Network IO takes about .25 seconds.
  
  

Have you seen this kind of variability?

Thanks!

[StephenBonikowsky]

@cspital I'm on a different project now but @HongGit and @mconnew should be able to answer.

[mconnew]

Shipping ready2run pre-jitted images wouldn't help in this scenario. The delay is caused by reflecting over your interfaces and creating a Contract description for the service at runtime. I had a customer support issue for .NET Framework where they had over 100,000 operations on their service and it took nearly 2 minutes to do the reflection to create the Service description object at startup. I put together a workaround and got it down to I think 2 seconds by pre-reflecting the contract and serializing the description to binary XML and saving that to disk. I don't think the same approach is possible for the client as there isn't an extensibility point to allow injecting the contract description. I've seen this issue come up multiple times from multiple people recently. Some mechanism to pre-generate the ContractDescription for the client is definitely something which would be a good feature to add. I'll take another look and see if I missed some way to do this without a product change otherwise this would need to be a feature request.

[cspital]

Thanks @mconnew this is really helpful.

I appreciate you taking a look, if you need some outside user to put the feature request together please do let me know, I am happy to do that. Also for what it's worth, we're running on dotnet core 3.1.

Thanks again.

[mconnew]

I think I've found a way to do it. ChannelFactroy\

[cspital]

Yeah @mconnew I'm happy to review what you have for framework and see what I can do for core although I haven't dealt with the concepts you mentioned before. Unsurprisingly I'm also crazy busy, so I can't guarantee any timeframe but I'd like to at least look at it and see if it's feasible. I assume this work is not related to this issue, should we create a new issue to track this work?

[mconnew]

I had to clean the code up a bit to remove pieces I couldn't release. You can find it here. I plan to write a blog post for it at some point which is why I put it in my Blog samples repo. I wasn't suggesting you implement a feature for WCF to ship on this, just that this would be a starting point to solve your own problem. If you get this working for ChannelFactory, feel free to provide a sample. This would probably be best to roll up into dotnet-svcutil and have an option to pre-cache the ContractDescription.
As to your question, yes this is a separate issue. Feel free to open a feature request issue for pre-caching the ContractDescription.