Pnp-sites-core: Implement handling of multiple pnp:ProvisioningTemplateFile elements

[ruslanurban]

Category

[x] Bug
[ ] Enhancement

Environment

[ ] Office 365 / SharePoint Online
[x] SharePoint 2016 (December 2016 CU)
[ ] SharePoint 2013

Using latest version of PnP-PowerShell module SharePointPnPPowerShell2016.

Expected or Desired Behavior

Our Global Configuration site template contains tens of thousands of lines. To simplify maintenance of the template files, we split them into a few smaller files: Content Type Hub site definition, Taxonomy (MMS), Site Fields and Content Types.

Schema documentation and sample files contain element pnp:ProvisioningTemplateFile, which implies that the partial provisioning templates can be loaded from external files.

Issue

Import of the template below fails with the following error:

Apply-SPOProvisioningTemplate : Value cannot be null.
Parameter name: source

Steps to Reproduce

Provisioning template: _Global.xml_

<?xml version="1.0"?>
<pnp:Provisioning xmlns:pnp="http://schemas.dev.office.com/PnP/2016/05/ProvisioningSchema">
    <pnp:Preferences Author="[Me]" Version="2017.1.0.0" Generator="GlobalExport">
        <pnp:Parameters>
            <pnp:Parameter Key="SiteTitle">Global Configuration</pnp:Parameter>
        </pnp:Parameters>
    </pnp:Preferences>
    <pnp:Templates ID="GLOBAL-TEMPLATES">
        <pnp:ProvisioningTemplateFile ID="GLOBAL-CTHUB-SITE" File="Global-CTHub-Site.partial.xml" />
        <pnp:ProvisioningTemplateFile ID="GLOBAL-TAXONOMY" File="Global-Taxonomy.partial.xml" />
        <pnp:ProvisioningTemplateFile ID="GLOBAL-FIELDS" File="Global-Fields.partial.xml" />
        <pnp:ProvisioningTemplateFile ID="GLOBAL-CTYPES" File="Global-CTypes.partial.xml" />
    </pnp:Templates>
</pnp:Provisioning>

PowerShell script:

$templateFile = Get-Item 'Global.xml';
$importParameters = @{ SiteTitle='Global Configuration'};
Connect-SPOnline 'http://cthub';
Apply-PnPProvisioningTemplate -Path $templateFile.FullName -ResourceFolder $templateFile.Directory.FullName -Parameters $importParameters;

Analysis

The following code fragment in XMLPnPSchemaV201605Formatter.cs (lines 1236-1264) appears to try loading only the first pnp:ProvisioningTemplateFile element. And, that is the only place in code that queries for the pnp:ProvisioningTemplateFile elements that I found.

```c#
foreach (var templates in wrappedResult.Templates)
{
// Let's see if we have an in-place template with the provided ID or if we don't have a provided ID at all
source = templates.ProvisioningTemplate.FirstOrDefault(spt => spt.ID == identifier || String.IsNullOrEmpty(identifier));

// If we don't have a template, but there are external file references
if (source == null && templates.ProvisioningTemplateFile.Length > 0)
{
    // Otherwise let's see if we have an external file for the template
    var externalSource = templates.ProvisioningTemplateFile.FirstOrDefault(sptf => sptf.ID == identifier);

    Stream externalFileStream = this._provider.Connector.GetFileStream(externalSource.File);
    xml = XDocument.Load(externalFileStream);

    if (xml.Root.Name != pnp + "ProvisioningTemplate")
    {
        throw new ApplicationException("Invalid external file format. Expected a ProvisioningTemplate file!");
    }
    else
    {
        source = XMLSerializer.Deserialize<V201605.ProvisioningTemplate>(xml);
    }
}

if (source != null)
{
    break;
}

}

And, there may be an issue with deserialization of the template in the following line:
```c#
source = XMLSerializer.Deserialize<V201605.ProvisioningTemplate>(xml);

Root element in the referenced template file is expected to be pnp:ProvisioningTemplate whereas properly formatted template should have non-nullable XML root xml:Provisioning.
See ProvisioningSchema-2016-05.cs
```c#//
namespace OfficeDevPnP.Core.Framework.Provisioning.Providers.Xml.V201605 {
using System.Xml.Serialization;

/// <remarks/>
[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd", "4.6.81.0")]
[System.SerializableAttribute()]
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.ComponentModel.DesignerCategoryAttribute("code")]
[System.Xml.Serialization.XmlTypeAttribute(Namespace="http://schemas.dev.office.com/PnP/2016/05/ProvisioningSchema")]
[System.Xml.Serialization.XmlRootAttribute(Namespace="http://schemas.dev.office.com/PnP/2016/05/ProvisioningSchema", IsNullable=false)]
public partial class Provisioning {
...
}

...


[System.CodeDom.Compiler.GeneratedCodeAttribute("xsd", "4.6.81.0")]
[System.SerializableAttribute()]
[System.Diagnostics.DebuggerStepThroughAttribute()]
[System.ComponentModel.DesignerCategoryAttribute("code")]
[System.Xml.Serialization.XmlTypeAttribute(Namespace="http://schemas.dev.office.com/PnP/2016/05/ProvisioningSchema")]
public partial class ProvisioningTemplateFile {

    private string fileField;

    private string idField;

    /// <remarks/>
    [System.Xml.Serialization.XmlAttributeAttribute()]
    public string File {
        get {
            return this.fileField;
        }
        set {
            this.fileField = value;
        }
    }

    /// <remarks/>
    [System.Xml.Serialization.XmlAttributeAttribute(DataType="ID")]
    public string ID {
        get {
            return this.idField;
        }
        set {
            this.idField = value;
        }
    }
}
...

}
```

Thus, it appears that the implementation may be errant, and the source variable may not get initialized due to deserialization error which can be causing the original issue described in this request.
Though, I may be wrong, and the error happens somewhere else.

Recommendations

1. Fix the errant code and allow loading of multiple referenced partial template files
2. If the fix cannot be implemented:
   
   
   
   • Throw NotImplementedException indicating that handling of pnp:ProvisioningTemplate has not been implemented.
   
   
   • Remove line 24 from PnP-Provisioning-Schema/Samples/ProvisioningSchema-2016-05-FullSample-02.xml
     
     <pnp:ProvisioningTemplateFile ID="WORKFLOWSITE" File="ProvisioningSchema-2015-05-ReferenceSample-01.xml" />
   
   
   • Add a note in the schema documentation PnP-Provisioning-Schema/ProvisioningSchema-2016-05.md stating that pnp:ProvisioningTemplateFile has been reserved for future implementation.
   
   

Note

There is issue #611 with the same symptom, which was closed but proper resolution was never implemented. I spent a couple of hours trying to figure out what the issue was, and now I need to implement a workaround because of the misleading documentation and schema samples. Hope, the functionality will be implemented or the documentation will be updated so that others are not going to spend their time unproductively and would account for unsupported and not implemented functionality in the solution architecture.

[wobba]

You can work around this by using xml includes:

<xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Objects/TermGroups.xml" />

Except the includes only have their own nodes, not the full header schema. The above line links in a fragment for terms from a subfolder Objects.

[ruslanurban]

Thank you Mikael. I will give it a try.

[jbranders]

Hi, How must I implement the

Thank you in advance.

[wobba]

You need to extract the parts you want to new .xml files which you then include. Below is a sample. The TermGroup.xml file could then be re-used in multiple templates.

template.xml
<?xml version="1.0"?> <pnp:Provisioning xmlns:pnp="http://schemas.dev.office.com/PnP/2016/05/ProvisioningSchema"> <pnp:Templates ID="CONTAINER-TEMPLATE"> <pnp:ProvisioningTemplate ID="TEMPLATE" Version="1"> <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="TermGroups.xml" /> </pnp:ProvisioningTemplate> </pnp:Templates> </pnp:Provisioning>

TermGroups.xml
<pnp:TermGroups xmlns:pnp="http://schemas.dev.office.com/PnP/2016/05/ProvisioningSchema"> <pnp:TermGroup Name="Common" ID="3176a966-a019-402d-87fe-d24b2f62386c" Description=""> <pnp:TermSets> <pnp:TermSet Name="Region" ID="0536a796-e07f-4e5e-b7e3-9c6cc342000a" /> </pnp:TermSets> <pnp:Contributors /> <pnp:Managers /> </pnp:TermGroup> </pnp:TermGroups>

[jbranders]

But i don't use mutliple templates. What i did is Get-SPOProvisioningTemplate of my site and than i run Apply-SPOProvisioningTemplate to my destination site and he gives me the error message? What can i do to resolve this?

[jansenbe]

@jbranders : think you've another issue resulting in the same error (Value cannot be null) as that error is generic.

@ruslanurban : thanks for reporting, we'll investigate this. Did you in the meanwhile get unblocked with @wobba 's approach?

[ruslanurban]

Thank you for following up @jansenbe.
Yes, the XInclude approach works. I actually found how it is implemented in the code, and it is not out-of-the box implementation for loading of XML documents. It would be nice to update samples to showcase this approach and to have it documented.

[jansenbe]

@ruslanurban : thx for the feedback!

[wobba]

@olemp or @okms should do a small sample as they are the ones who showed me it was possible :)

[vman]

@jansenbe is the XInclude approach that @wobba mentioned officially recommended in this case?

[MartinHatch]

@jansenbe any update on this? I'm also keen to know if XInclude is the officially recommended approach for "partial XML files" while we wait for ProvisioningTemplateFile elements to be fixed

[jansenbe]

Yes, XInclude is something that you can safely use. We reproduced the issue internally, so marked it as a bug.

[georgebonney]

The FirstOrDefault call when processing multiple templates is a big problem for me. The schema implies multiple templates are supported but the code only takes the first one. The seems like a high priority bug to me. Any thoughts from anyone on this?

[wobba]

My thought is that since XInclude works, it might not be a priority - but would be nice. @georgebonney you are free to do a PR :)

[joshtransient]

I'd like to push a little further on this. XInclude might work for one-offs, like the <pnp:TermGroups> example above, but deploying additional elements _into_ container nodes is a non-starter.

Example: My base template has:

<pnp:ProvisioningTemplate ...>
  <pnp:Lists>
    <pnp:ListInstance ... />
  </pnp:Lists>
</pnp:ProvisioningTemplate>

I can't XInclude additional <pnp:ListInstance>s in any fashion because:

• If I start my included XML with <pnp:Lists>, template validation fails due to multiple <pnp:Lists> elements
• I can't place multiple <pnp:ListInstance> elements into an XML without a parent node, and <pnp:ListInstance> won't validate under any other node

At which point, I might as well do this:

<pnp:ProvisioningTemplate ...>
  <pnp:Lists>
    <pnp:ListInstance ... />
    <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="template.list-1.xml" />
    <xi:include href="template.list-2.xml" />
    ...

XInclude works, but only in limited scenarios. At this point, it makes considerably more sense for me to apply my base template first, then apply a second template to add additional items.

Really hope <pnp:ProvisioningTemplate> is re-evaluated soon.