Powershell: DSC resource platform changes to use JSON instead of MOF

[SteveL-MSFT]

Problem Statement

As we continue to make DSC more viable cross-platform, we have a serious problem with MOF. MOF parsing is provided by MMI.dll (there is both native code and managed code). MMI itself has dependency on OpenSSL 1.0 for the WSMan features. The non-Windows build of MMI is no longer supported and has been deprecated.

Proposal

To enable DSC resources to work cross-platform and on new distros that have moved to OpenSSL 1.1 (which is not backwards compatible with 1.0), we need to move away from MOF schema. This also means we would only support DSC resources written as PowerShell script (Get, Set, Test functions) or PowerShell classes. PowerShell classes already don't require a MOF schema, but many DSC Resources are script resources so we need to move to something other than MOF to describe the schema of that resource.

The current proposal is to move to JSON instead of MOF.

Engine Changes

System.Management.Automation.dll needs to be updated to understand JSON schema instead of MOF schema for PowerShell script DSC resources.

[NOTE!] Compilation of DSC configuration to MOF will not change for now, but in the future we also want to have configuration compile to a JSON format instead of MOF.

PSDesiredStateConfiguration module

This module exists outside of this repo. Plan is to eventually make it Open Source. For now, changes will need to be made to this module to work with JSON schema instead of MOF. In addition, it should expose a new cmdlet to help convert existing MOF schema to the new JSON schema format for existing DSC PowerShell script resources.

With this change, embedded objects passed to Get/Set/Test will be PSObjects instead of CimInstances. Unless the script is explicitly checking that it is a CimInstance, existing code should work with PSObject.

Also, a new Test-DscResourceSchema cmdlet will be added to validate the script resource matches the JSON schema.

What about LCM (Local Configuration Manager) and WindowsPowerShell?

Building upon the work in PS7.0 to enable use of Invoke-DscResource, the current plan is to not integrate with WindowsPowerShell LCM. This means that new DSC Resources based on JSON will not be compatible with WindowsPowerShell. This also means that existing DSC Resources that are WMI/OMI based or only have MOF schema will not be supported.

[iSazonov]

Do you consider YAML instead of Json? It looks more user-friendly.

Do you plan use new .Net Json API for the project?

Should we have some RFC-s for the project?

With this change, embedded objects passed to Get/Set/Test will be PSObjects instead of CimInstances.

I feel strong-typed objects would be more convenient.
Especially if there is a scheme.

the current plan is to not integrate with WindowsPowerShell LCM.

Why? MSFT team invested in Windows Compatibility feature - call Windows PowerShell from PowerShell Core. They could do the same - call PS7 from Windows PS.

This means that new DSC Resources based on JSON will not be compatible with WindowsPowerShell.

Does this mean that Json schema is not compatible with MOF schema and it is inpossible to implement a simple converter?

This also means that existing DSC Resources that are WMI/OMI based or only have MOF schema will not be supported.

Where will not be supported in? In PS7? If so the same question about a schema converter.

[bergmeister]

Do you consider YAML instead of Json? It looks more user-friendly.

Since YAML is a superset of JSON, users will be able to convert to/from YAML at any time if they wish. The lack of an established standard library for YAML and PowerShell not shipping YAML support out of the box (and the lack of a community agreed module) is a strong argument against YAML IMHO

[cblackuk]

This is FANTASTIC!!! Thank you for doing this! JSON/YAML FTW!!!

[bigoals365]

I prefer YAMs over Mofs :}

Making YAML the "official" schema would be cleaner and more readable. JSON is format spaghetti.

Help make 2020 better with some common sense. Unless you really enjoy obfuscation :}

I now want to make a TiK Tok video about this... NOT

[johlju]

Could it be made possible to have both MOF and JSON in the same DSC resource to make it compatible for both Windows PowerShell and PowerShell 7.1?

[Monte-Hazboun]

I second johlju's request. For projects that use DSC and use both Win PowerShell and 7.1, having to maintain two separate versions of a resource module is a chore, even with a conversion tool. The ability to detect which version is being used and then load the correct format schema would be a huge boon.

I also second Bergmeister's comment and prefer JSON over YAML.

[SteveL-MSFT]

Do you consider YAML instead of Json? It looks more user-friendly.

The JSON is not intended to be used by users. It's a serialization format only.

I feel strong-typed objects would be more convenient.

The current is just a transformation from CimInstance to PSObject. Both are not strongly typed to the original object.

Why? MSFT team invested in Windows Compatibility feature - call Windows PowerShell from PowerShell Core. They could do the same - call PS7 from Windows PS.

The WindowsPowerShell LCM hosts .NET Framework. It would need lots of work to also host .NET Core. Our direction is "bring your own agent" where it could simply be a PowerShell script serving the same capabilities as LCM.

Existing WMI/OMI based DSC Resources are tied to legacy LCM which is not supported. Therefore, those specific type of resources are also not supported. We want to move towards a PowerShell script/classes based DSC resource model to enable faster development and also encourage cross platform instead of compiled code.

Could it be made possible to have both MOF and JSON in the same DSC resource to make it compatible for both Windows PowerShell and PowerShell 7.1?

Yes, the MOF will be ignored by the new DSC and the old DSC will ignore the JSON file.

We can discuss more of this at next DSC Community Call.

[SteveL-MSFT]

Do you plan use new .Net Json API for the project?

The changes to use JSON instead of MOF simply uses the ConvertTo-Json and ConvertFrom-Json cmdlets.

[iSazonov]

The changes to use JSON instead of MOF simply uses the ConvertTo-Json and ConvertFrom-Json cmdlets.

If we want to move the cmdlets to .Net 5.0 Json API there will be many small breaking changes and makes sense to make this work earlier then later. Otherwise the new DSC project will catch the breaking changes after release and it is very bad.

[kvprasoon]

Our direction is "bring your own agent" where it could simply be a PowerShell script serving the same capabilities as LCM.

@SteveL-MSFT This is an awesome thought 👍

[iSazonov]

Our direction is "bring your own agent" where it could simply be a PowerShell script serving the same capabilities as LCM.

It makes us think about some common module.

[Jeff-Jerousek]

If the main argument for MOF was to support non-Windows builds and that support has been deprecated it makes sense to move it.

I hope the goal is the next 7 LTS release. Using the same configs irregardless of OS and letting the agent determine the details would be a strong feature to support moving to 7 in production environments.

[SteveL-MSFT]

I would love to see some LCM replacements as modules on PSGallery. Also, with "bring your own agent", it means integration with existing tools like Ansible, Puppet, and Chef should be easier as they can invoke DSC resources directly instead of going through LCM.

Goal is to have something people can start using in 7.1 and have a more complete story by 7.2 which is our next LTS.

[kgraves83]

So if we are no longer using the built in LCM, would we use a model similar to Ansible, where we run a script from a workstation which connects to a server and runs a DSC module to apply the configuration? Would we still need to install the DSC modules on the server? Or could it be agentless and everything is installed on your workstation (like you can do with Ansible).

[SteveL-MSFT]

@kgraves83 with the current design, DSC modules will just be installed using PowerShellGet. This means they can be on a local fileshare or a remote gallery. So from that perspective, you could have orchestration initiated from a server, but everything else happens locally on the target (workstation).

[kvprasoon]

@SteveL-MSFT Feels like remotely triggering Test/Set/Get methods... How about the Pull server and LCM properties like ConfigurationMode etc... ? are all these left to people to design in the way anyone need ?

[gaelcolas]

Given the challenges and the need to have a breaking change on all DSC resources using an embedded type (need to replace [CimInstance] in PSM1s by [PSObject], breaking the resource for PS5.1), I suggest dropping completely the MOF or JSON based DSC Resource, and focus on Class based resource, superior in many ways anyway.

See full comment here: https://github.com/PowerShell/PowerShell/pull/13399#issuecomment-690930913

[wsmelton]

Requiring a Class to be used would cut down the number of contributors you may have to DSC Resources. Writing PowerShell Classes has a harder entry point for many because it is not common (in beginner world at least) and most are used to writing function-style scripts and native PowerShell. Many have learned to code around JSON because it is commonly used with REST APIs.

[michaeltlombardi]

From an integrating tool developer standpoint, JSON is _much_ preferable to MOF for us. Right now, parsing resources for their API surface is less than pleasant; JSON would substantially ease that problem but not solve it (since it wouldn't aid with class-based resources).

[SteveL-MSFT]

Closing this in favor of https://github.com/PowerShell/PowerShell/issues/13731