Powershell: Including open source libraries to support YAML cmdlets

[dfinke]

Need guidance on creating YAML cmdlets by including this library.
https://github.com/aaubry/YamlDotNet

[ArieHein]

@dfinke : Try looking at the Ansible repos from https://github.com/trondhindenes
to see how he handled it as Ansible is YAML

[ArieHein]

We discussed having a DSC resource to handle some common text files. YAML would fit there too.
https://github.com/PowerShell/DscResources/issues/155

[dfinke]

@ArieHein Could you say more about your comment? I've wrapped the YAMLDotNet library in PowerShell so I can do ConvertFrom-Yaml. Works fine. I was thinking it'd be good to add this to PowerShell directly so these cmdlets are available out of the box.

I was thinking of pulling YAMLDotNet into the PowerShell code, and create a cmdlet for it. I wanted guidance on adding other oss into the core product. If v6.0 ships with Windows, are there legal restrictions? Is there a mechanism to add these types of cmdlets so it ships with the OSS version and not the Windows release? etc

[ArieHein]

The starting point was adding a DSC resource to handle various text formats. We didnt mention YAML as an option as implementing it would take longer time. Naturally it would be based on the appropriate .net classes and types (were not inventing the wheel).

Trons' repo was something I found along the way when I was learning Ansible and he has created DSC resources to handle YAML, so I figured you might want to take a look if anything looks reasonable to you. Perhaps the core code is already inside PowerShell without bringing additional libraries.

[kilasuit]

I agree with @dfinke that it would be good to get some guidance on whether it is possible to be able to add this directly in the PowerShell 6 release as this would also help where there are other possible options that could also be included from other OSS projects too

[RamblingCookieMonster]

This would be quite valuable. Yaml is a common format in the wider Microsoft ecosystem, and beyond. Some common use cases:

• Introduce more opportunity for tooling and automation for systems using yaml for configuration
• Provide an friendlier (IMHO - at the very least, "another") way to serialize and deserialize data for our own PowerShell tools

Cheers!

[vors]

This raise the question "What should be available in the core distro and what should be downloadable from the gallery". Historically (before Windows PowerShell v5), it was hard to relay on any modules that are not part of the distro. With https://www.powershellgallery.com/ it became much easier to install and depend on 3rd party modules.

[kilasuit]

@vors - personally it makes sense for as much to be pushed to gallery as possible and "decoupled" from the core of PowerShell where I've rationalised why in #2136 & #1979

But saying that I feel that any ConvertTo/ConvertFrom cmdlets like the proposed YAML ones should really be bundled into Microsoft.PowerShell.Utility.

But then again that's just my thoughts :wink:

[RamblingCookieMonster]

I'd be for decoupling. Presumably, if it goes into PowerShell itself, it's locked to that and higher versions of PowerShell (and difficult to update outside of that cadence, as you mentioned @kilasuit).

On top of that, presumably an external module could target down-level PowerShell versions. IMHO this alone, assuming you target PS3 or PS4, would justify a separate module - many of us on the operations side don't have a choice, or aren't ready to move to PS5+ across the board - not everyone runs a farm of cattle unfortunately : )

Cheers!

[kilasuit]

This is where module metadata in the PowerShell Gallery plays a very important role. Even modules released to the Gallery can be downloaded without having PackageManagement installed using something like this

$file = 'https://www.powershellgallery.com/api/v2/package/ISESteroids/2.6.0.3'
$Request = [System.Net.WebRequest]::Create($file)
$Request.Timeout = '1000000000'
$URL = $Request.GetResponse()
$Filename = $URL.ResponseUri.OriginalString.Split('/')[-1]
$WC = New-Object -TypeName System.Net.WebClient
$WC.DownloadFile($file,('C:\Files\{0}' -f $Filename))
$WC.Dispose()
$URL.Close()

So things like PowerShellVersion & PowerShellHostName etc should be filled in where possible - else why bother having the metadata available to us in the first place?

[vors]

Based on the discussion, our resolution is: we will keep YAML parsing capabilities outside of the scope for PowerShell core (inbox modules) in the observable future.
We are encouraging you to use PowerShell gallery as a delivery vehicle for the module.

[dfinke]

Just another data point.

AWS CloudFormation Introduces YAML Template Support

https://aws.amazon.com/about-aws/whats-new/2016/09/aws-cloudformation-introduces-yaml-template-support-and-cross-stack-references/

[bgshacklett]

I understand that every feature needs a business case. Currently, YAML is supported by pretty much every language out there, except for PowerShell. When I tell my coworkers that they should be looking into PowerShell on Linux or Mac OS, and then I say they're going to have to add another external library to handle YAML, it's going to seriously hurt PowerShell's position relative to other languages.

[iSazonov]

If this language is so important and needful (is it really true?), I see three direction to add support it:

1. Module as external package
2. New System.Yaml namespace in CoreCLR
3. Native Yaml cmdlets and native yaml

[ArieHein]

@bgshacklett hmm what language supports YAML as a core type ? even python requires 'another external library'. So I cant see how this will 'hurt' PowerShell position.

@iSazonov I dare say there wont be a System.Yaml for the same reason there's no System.JSON (not counting Silverlight) though you have some abilities via Serialization but this is left for the 3rd party to
create their own implementations and not be bound. MS are good at making frameworks with enough hooks for the programmers to build upon and extend and thus create a partner ecosystem.

As @RamblingCookieMonster said, decoupling is a good thing as you're not bounding the language to a specific implementation.