Sp-dev-docs: Intermittent error: "Connection to SPO DS failed."

[YannickRe]

Category

• [ ] Question
• [ ] Typo
• [X] Bug
• [ ] Additional article idea

Expected or Desired Behavior

We use SharePoint CSOM and PnP Core C# client side libraries to provision new site collection and configure them. I expect these calls to work.

Observed Behavior

Since a week or so, some of our calls fail sometimes with the error message "Connection to SPO DS failed". This may run for hours without a problem, then fail a couple of times and then start working again.

Example code:

                Tenant tenant = new Tenant(sharePointHandler.ClientContext);
                var site = tenant.GetSitePropertiesByUrl(siteUrl, false);
                sharePointHandler.ClientContext.Load(site, s => s.Status);
                sharePointHandler.ClientContext.ExecuteQuery();

Error information:

ServerErrorTypeName: Microsoft.Online.SharePoint.Common.SpoException
ServerErrorTraceCorrelationId: 51ac079f-b073-1000-6613-d2b2ddbfe58a
Source: Microsoft.SharePoint.Client.Runtime
Message: Connection to SPO DS failed
StackTrace:    at Microsoft.SharePoint.Client.ClientRequest.ProcessResponseStream(Stream responseStream)
   at Microsoft.SharePoint.Client.ClientRequest.ProcessResponse()
   at Microsoft.SharePoint.Client.ClientRequest.ExecuteQueryToServer(ChunkStringBuilder sb)
   at Microsoft.SharePoint.Client.ClientContext.ExecuteQuery()
   at <stripped>

Steps to Reproduce

I have none, can't find a reason why the same code mostly does work and sometimes fails.

[msft-github-bot]

Thank you for reporting this issue. We will be triaging your incoming issue as soon as possible.

[VesaJuvonen]

We are looking into this now. If someone else is experiencing this, please do let us know immediately so that we can analyze the worldwide impact. Thx.

[advdberg]

@VesaJuvonen @YannickRe we see the same error for a few weeks now, but then on creating new sites with either New-PnPTenantSite or New-SPOSite PowerShell cmdlets. Already have a MS service call active on that (16467790). We've added a, sort of, reproducable situation (in about half of the times we run the command on that specific tenant).

For now we (luckily) only see it happening on one specific tenant. We will try to reproduce it on our 'internal' testing tenants.

Because we suspected throttling we already have a 10 times retry with 10 seconds interval in our automation scripts, but in some cases even that is not enough and we get the same error 10 times.

[YannickRe]

@advdberg We were already suspecting more people to be affected, and I've got the bug confirmed already. Probably Vesa can give a more detailed update (not sure what I can share now).

[advdberg]

FYI: we traced it down to the following request/response:

Request:

<Request AddExpandoFieldTypeSuffix="true" SchemaVersion="15.0.0.0" LibraryVersion="16.0.0.0" ApplicationName="SharePoint Online PowerShell (16.0.7317.0)" xmlns="http://schemas.microsoft.com/sharepoint/clientquery/2009">
    <Actions>
        <ObjectPath Id="4" ObjectPathId="3" />
        <ObjectPath Id="6" ObjectPathId="5" />
        <Query Id="7" ObjectPathId="5">
            <Query SelectAllProperties="true">
                <Properties />
            </Query>
        </Query>
    </Actions>
    <ObjectPaths>
        <Constructor Id="3" TypeId="{268004ae-ef6b-4e9b-8425-127220d84719}" />
        <Method Id="5" ParentId="3" Name="CreateSite">
            <Parameters>
                <Parameter TypeId="{11f84fff-b8cf-47b6-8b50-34e692656606}">
                    <Property Name="CompatibilityLevel" Type="Int32">0</Property>
                    <Property Name="Lcid" Type="UInt32">1043</Property>
                    <Property Name="Owner" Type="String">[email protected]</Property>
                    <Property Name="StorageMaximumLevel" Type="Int64">0</Property>
                    <Property Name="StorageWarningLevel" Type="Int64">0</Property>
                    <Property Name="Template" Type="String">SITEPAGEPUBLISHING#0</Property>
                    <Property Name="TimeZoneId" Type="Int32">4</Property>
                    <Property Name="Title" Type="String">CTBTST 32</Property>
                    <Property Name="Url" Type="String">https://customer.sharepoint.com/sites/ctbtst32</Property>
                    <Property Name="UserCodeMaximumLevel" Type="Double">0</Property>
                    <Property Name="UserCodeWarningLevel" Type="Double">0</Property>
                </Parameter>
            </Parameters>
        </Method>
    </ObjectPaths>
</Request>

Reponse:
[
{
"SchemaVersion":"15.0.0.0","LibraryVersion":"16.0.19304.12019","ErrorInfo":{
"ErrorMessage":"Connection to SPO DS failed.","ErrorValue":null,"TraceCorrelationId":"f26f059f-902b-1000-ca6c-a44681c55587","ErrorCode":-1,"ErrorTypeName":"Microsoft.Online.SharePoint.Common.SpoException"
},"TraceCorrelationId":"f26f059f-902b-1000-ca6c-a44681c55587"
}
]

[advdberg]

@YannickRe glad to see that you have your bug confirmed already after 2 days, we're struggling now for 15 days with MS support to get it investigated. Should have created an issue overhere earlier, but because we saw it only happening on one tenant until now we decided to walk the MS support road.

[VesaJuvonen]

Issue should be now resolved with @YannickRe's case, but if @advdberg you are still seeing this, we should investigate it by getting the correlation ID catched in your code and then sharing that to Microsoft support.

@YannickRe wrote a blog post explaining this process - https://blog.yannickreekmans.be/grab-the-correlation-id-from-sharepoint-online-csom-exceptions/

that then helps on MS people to check the ULS logs for detailed analyses of the situation.

[advdberg]

Great! Already provided correlation id's before, and this morning informed our MS case manager about this thread. Thanks for the quick response!

[advdberg]

It looks like our issue is now also solved. Did some testing last weekend and this morning and didn't encounter the error message anymore :-)

@VesaJuvonen My conclusion is that both issues had the same root cause?

[VesaJuvonen]

Thanks for following up @advdberg. I do not have insights on your support case or your correlation ID, so can't confirm, but if you hare seeing exactly the same exception, then probability is quite high that the root cause was the same issue. Let's still follow up on this for a while during this week and close it if needed on Wed/Thu time frame.

[VesaJuvonen]

Closing this as got also numerous confirmations internally that reported issue is gone. Thanks everyone for sharing your input on this and we do apologize any inconvenience it might have caused.

[DaytaMOD]

Still getting the error via SharePoint Online Admin in browser, abd via PowerShell. New-PnPTenantSite intermittently fails.

[msft-github-bot]

Issues that have been closed & had no follow-up activity for at least 7 days are automatically locked. Please refer to our wiki for more details, including how to remediate this action if you feel this was done prematurely or in error: Issue List: Our approach to locked issues