Pnp-sites-core: ExecuteQueryRetry() - What's the point???

[SchauDK]

We've all learned to use ExecuteQueryRetry() to back off in case of throttling responses from SharePoint. Which btw. is a HUGE issue for ISVs. See https://github.com/SharePoint/sp-dev-docs/issues/1247

But what is the point of retrying if the request isn't returning what you requested after the retry?
This is what we're seeing more and more.

Eg.
Applying a provisioning template where the Field object handler gets throttled.
```Information: 0 : 2018-08-30 09:47:54.2958 [Fields] [1] [Debug] Updating field {EF74E985-EBF0-4C48-B722-28EC6266FFA4} in site 109796ms 7b2b82a1-83df-4f46-96fd-b175089a4c20
Warning: 0 : 2018-08-30 09:47:57.1248 [OfficeDevPnP.Core] [0] [Information] CSOM request frequency exceeded usage limits. Sleeping for 500 milliseconds before retrying. 0ms
Error: 0 : 2018-08-30 09:47:57.9501 [Fields] [1] [Error] Updating field {EF74E985-EBF0-4C48-B722-28EC6266FFA4} failed: The property or field 'SchemaXml' has not been initialized. It has not been requested or the request has not been executed. It may need to be explicitly requested. : at Microsoft.SharePoint.Client.ClientObject.CheckUninitializedProperty(String propName)
at Microsoft.SharePoint.Client.Field.get_SchemaXml()
at OfficeDevPnP.Core.Framework.Provisioning.ObjectHandlers.ObjectField.UpdateField(Web web, String fieldId, XElement templateFieldElement, PnPMonitoredScope scope, TokenParser parser, String originalFieldXml, Boolean updateFieldIfTypeChanged)
at OfficeDevPnP.Core.Framework.Provisioning.ObjectHandlers.ObjectField.ProvisionObjects(Web web, ProvisioningTemplate template, TokenParser parser, ProvisioningTemplateApplyingInformation applyingInformation) 113450ms 7b2b82a1-83df-4f46-96fd-b175089a4c20
Exception thrown: 'Microsoft.SharePoint.Client.PropertyOrFieldNotInitializedException' in OfficeDevPnP.Core.dll

This is the code from UpdateField
```C#
            var existingField = web.Fields.GetById(Guid.Parse(fieldId));
            web.Context.Load(existingField, f => f.SchemaXml);
            web.Context.ExecuteQueryRetry();

So, it is retrying once waiting 500ms and the request completes, but the SchemaXml property isn't loaded.

Any comments from the OfficeDevPnP Team about this?

[wobba]

Hi @SchauDK,
I've noticed the same thing, and we have some ideas we want to try out to resolve this. Basically trying to clone the context before execution, and then re-use the same exact context on every re-try.

[jansenbe]

We've managed to repro the issue and are working on a fix. Fix is not so straightforward as the clientcontext object (which already can hold lots of loaded data) is impacted due to the server side 429. Did try with a deep clone as possible "quick" fix but that did not work for me, so now working with the devs to see how we can fix this.

[SchauDK]

Sounds great. We're also trying to push some buttons through the ASfP channel.

[phawrylak]

I've investigated this issue a little and tried several methods of deep cloning ClientContext (serialization/deserialization with BinaryFormatter, mapping with AutoMapper, cloning with several libraries) and the only library/method that worked for me is DeepCloner. I can make a PR, but I don't know if using this library is acceptable. Looking on benchmarks it seems to be pretty good. @jansenbe, what do you think?

[wobba]

@phawrylak be patient 🙂 a proper fix is in the works, so a PR is not needed.

[phawrylak]

@wobba, ok, great to hear that. Just wanted to help/suggest solution in case this issue is not the top priority. 😅

[jansenbe]

Thanks for looking into this @phawrylak. As mentioned by @wobba we'll have a fix inside CSOM that will make it easy to retry a failed request. We hope to have this in the next CSOM drop, so stay tuned for that.

[jansenbe]

@SchauDK, @phawrylak : a new throttling implementation has just been committed to the dev branch. Will be part of the December release...and this implementation now does work. The CSOM engine has been updated to make this possible. Does require latest CSOM version (16.1.8316.1200). Will close this issue now, but feel free to re-open if the new implementation does not work for you

[SchauDK]

That's fantastic. Thanks a lot.
I'll test and let you know if it doesn't work.

[NinjaSciurus]

What about hybrid scenarios? Why are those fixes only introduced into the logic that targets SPO? We're building a large hybrid solution where an on-premise client app (C#) needs to work with SP2019 and SPO concurrently and we're constantly tripping over stuff like this. We're stuck with the Nuget Package Mircosoft.SharePoint2019.CSOM 16.0.10337.12109 which targets SP2019 RTM (Oct 2018). Is there a reliable (and supported) workaround for this CSOM version? All other ideas imply a heavy refactoring overhead which is really inconvenient.