Powershell: Add ConvertTo-Base64String and ConvertFrom-Base64 String cmdlets

[steviecoaster]

Summary of the new feature/enhancement

With the help of @vexx32 I've put together a C# module that includes both ConvertTo-Base64String and ConvertFrom-Base64String which supports using the standard encodings available from System.Text.Encoding. I would like to see this included as part of PS Core, but am unsure of the process. I've heard rumblings the ideally the process moving forward would be to maintain it as module, and then add it in PS Core that way, but am fuzzy on the details

Proposed technical implementation details (optional)

I would like to see these two cmdlets make it into core, and am looking for direction as the best way to handle this moving foward. Should I PR the codebase in Core proper? Or should I PR the proposal to add the module as a dependency?

For your review the codebase is housed here: https://github.com/steviecoaster/PSBase64

NOTE: I still need to look at the documentation for adding good comment-based help to this module. It's my first foray into C#!

[SteveL-MSFT]

@steviecoaster seems like a good enhancement. To add new features to PSCore6, you should create a RFC in PowerShell-RFC repo which allows the community to provide feedback before spending time coding and the adjusting/fixing later based on feedback. Since you've already written the code, perhaps we can expedite this by having a discussion in this issue on the cmdlet parameters and you can contribute the code as an Experimental Feature which would allow users to try it out and provide design feedback before it's committed and becomes breaking changes.

[steviecoaster]

Absolutely. The parameters are relatively straightforward. An InputObject (string), and the Encoding type you want to use.

What's the best method to attach the code to this issue aside from linking to it from my repository? I suppose a PR marked Experimental would be prudent, with continuing discussion being followed up here?

[vexx32]

I'm tempted to recommend the InputObject be typed as PSObject instead of string, so that it can handle both $stringarray | ConvertTo-Base64String as well as ConvertTo-Base64String -InputObject $stringarray

This does require some additional handling and detection of whether you're in a pipeline context, however; see tail end of discussion in #4242 for some possible recommendations. 😄
You'll also need to work with InputObject.BaseObject when passing the values about, but that's not a huge deal. 🙂

@SteveL-MSFT do we have a drawn-up document about how to apply things like this as experimental features, or should we be looking to the PR that initially added the ability to create experimental features still?

[SteveL-MSFT]

@vexx32 not exactly sure what you're asking. The Experimental Feature doc I linked above shows how to create an Experimental Feature.

@steviecoaster you can submit a PR with title prefix "WIP:" to indicate it is a work-in-progress and get feedback that way. Make sure to include tests as that's an easy way to see intended usage.

[vexx32]

@SteveL-MSFT Ah, you're right, I must have overlooked the link. Thanks! I need to read that one myself. 😄

[markekraus]

Some thoughts on these cmdlets:

How would they handle multiple inputs? Convert each individually? Collect them and convert them as a whole? Both? Toggled how? Which is default?

As an example:

Get-Content -Path $MyTextFile | ConvertTo-Base64String 

Since GC returns those as an array of strings.
It may be prudent to support both (this is a lesson I have learned form my involvement with the JSON cmdlets). Users have wildly different expectation about how this works because the example above is different than, say, this:

"a", "b", "c" | ConvertTo-Base64String

Also, strings are not the only thing needed to be converted to base64 (and in my use-cases, they are often the least common thing I convert to base64). It would be prudent to also support bytes and byte arrays

Get-Content -AsByteStream -Path $MyBinaryFile | ConvertTo-Base64String

The reverse may also be truly needed:

$Base64String | ConvertFrom-Base64String -AsByteStream | Set-Content -AsByteStream -Path $MyBinaryFile

I would also like to see support for System.IO.Stream objects... I still think those are a huge missed opportunity in pwsh.

[msftrncs]

A couple of comments:

• Not sure if this is a good thing or not. cmdlets are nice and all, but having the full power of .NET with a simple method call (okay, it takes two method calls), makes this particular function seem overkill for C#, except maybe for handling io.streams. Calling cmdlets take time and resources, directly calling the methods is much faster and uses fewer resources.

Case in point. This line quickly decodes, alters, and then re-encodes and formats the string back to original, and it processes multi megabyte files in a blink of an eye. (this manipulates a text file that has been encoded within a <data> element inside a .NET RESX file, returning it to the RESX file in an altered state.) https://github.com/msftrncs/PwshIFMResXTools/blob/master/IFM%20MPC%20RESX%20Reducer.ps1#L102-L108

$datablock.value = ([Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes(
            # convert from Base64String
            [Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($datablock.value)
                # remove effectively empty S3, S2, or S1 DATA records
            ) -replace 'S(?:(?:3.{4})|(?:2..)|(?:1)).{6}(?:FF)+..\r*\n')
        # take the new Base64String and break it into 80 character lines formatted as assumed for the RESX file
    ) -replace '.{1,80}', "`n        `$&") + "`n"

• On the other hand, wouldn't the PowerShell Gallery be an ideal place to showcase something like this?
• Would there not be some use in providing other parameters from the .NET functions, such as the start character and length of characters, formatting options, or even the ability to input/output char[] (Base64CharArray())?
• The other possibility I have seen used is to create custom types to alias these method calls, so again, a cmdlet may be overkill. See https://tfl09.blogspot.com/2013/02/working-with-base64-strings-in.html. (this is an example of adding a custom method to the string type to handle both encoding (fixed to Unicode) and conversion in a single virtual method.) I could see this being improved to add additional methods to byte[] for the encoding, and that would make both functions more simple to use. $newString = $origString.toUTF8ByteArray().toBase64String(), $decoded = $encoded.ConvertFromBase64().toUTF8DecodedString()

[SteveL-MSFT]

Working on a TextUtility module that will include this

[GHRoss]

Hi @SteveL-MSFT,

This has been quiet for a while, are you still working on this?

If not, can @steviecoaster's additions be added?

[lahell]

ConvertTo-Base64String would be very useful, especially if it had parameters LineLength with ValidateRange(64, 76) and EndOfLine with ValidateSet('CRLF', 'LF').

[TBBle]

For reference, the Microsoft.PowerShell.TextUtility module @SteveL-MSFT mentioned does appear to have Base64 support, see ConvertBase64Command.cs. It doesn't appear that it was ever published to PowerShellGallery.com though.

[SteveL-MSFT]

Apologies on not getting around to publishing the module. Was working on a new cmdlet to add to it, got sidetracked by other important things, and then simply forgot about it. I'll see about getting what I have published as a preview the next few weeks. There will be a separate repo for it, so we'll carry this conversation over there once it's created.