Runtime: Remove Artificial Object Graph Limit in BinaryFormatter

[Alois-xx]

Since issue dotnet/runtime#20569 is solved we now can deserialize bigger object graphs with up to 12 million objects.
As I have found since a long time ago: https://aloiskraus.wordpress.com/2017/04/23/the-definitive-serialization-performance-guide/ there are issues with the max object count BinaryFormatter can cope with because it contains a hard coded prime number table which ends at 6584983 objects.

Please remove this artifical limit and do the same thing like e.g. in Dictionary which also needs prime numbers.

[joperezr]

cc @ViktorHofer can you take a look?

[ViktorHofer]

will take a look this week.

[danmosemsft]

BTW I notice the HashHelpers in CoreFX precomputes up to 2B (highest to fit in int) whereas the one in Corelib precomputes only up to 7M (and then generates primes). This means Dictionary (which uses the Corelib one) is generating a new prime after 7M entries. I suggest replacing the CoreCLR one with the CoreFX one.

Has for ObjectIDGenerator (for BinarySerializer) ideally it would use Hashhelpers but it chooses to start at 5 instead of 3 for some reason. maybe that could change.

[danmosemsft]

@Alois-xx I enjoyed your blog post! Thanks for pointing out these issues.

[Alois-xx]

@danmosemsft thanks for the kind words. I always do a root cause analysis of interesting issues. This involves quite a bit of explorartory boundary testing which often leads to nice profiling graphs or Excel charts or both.

[danmosemsft]

@alois-xx what sort of problems are interesting to you in general? I could look out for an interesting perf optimization opportunity.

[Alois-xx]

@danmosemsft I work on a very large application which uses pretty much every .NET Framework (desktop) technology. Startup time is always an issue. Windows Forms/WPF interop is a source of pain with many temporary bitmaps beeing generated. The WPF measure calls to layout the UI controls are very costly if you have a large nested UI hierarchy.
The workflow foundation VB compile times and memory consumption are ludicrous. This is costing hundreds of MB for a 5 MB xaml file which is surprising to say the least. Despite having several long lasting calls for this issue nothing has happened since years.

From a .NET Core perspectivethe the costs of getting live data for processes (System.Diagnostics.Process ... ) seems to be overly costly which could be improved if I am e.g. only interesting in the memory footprint of other processes.

[danmosemsft]

Would .NET Core improvements benefit you? Or is your goal that changes are stepping stone to port to .NET Framework?

For sure we are open to perf improvement suggestions or PRs for Process.

[Alois-xx]

@danmosemsft I hope to get improvements done in .NET Core to light up in the regular .NET Framework as well which makes it very attractive to invest time right now since you are moving incredibly fast.
Switching over to .NET Core is currently due to the many dependencies to WPF/WinForms/WF ... not an option for some time. But for new projects .NET Core is definitely an attractive option if the web is the new UI.

[danmosemsft]

@ViktorHofer I see you took this one... are you planning to do this for 2.1 (ie by Friday)?

[ViktorHofer]

done

[danmosemsft]

Nice work

[Alois-xx]

@ViktorHofer, @danmosemsft , @jkotas : Awesome! The amount of dedication to arrive at the best solution is much appreciated.

[danmosemsft]

@Alois-xx you are welcome, I hope you do end up basing new projects on .NET Core as you mention, and share other feedback with us as it comes up for you.