Powershell: Named format template parameters

[rjmholt]

Summary of the new feature/enhancement

Something that would be really useful in PowerShell is a readable, safe, hygienic way to format a template string given a set of key/value pairs.

Today in PowerShell, there are many ways to achieve string formatting, such as:

• "Hello my name is {0}" -f "Henry"
• "Hello my name is $Name"
• [string]::Format("Hello my name is {0}", $Name)
• Even "Hello my name is " + $Name

However, a scenario I see crop up from time to time is needing to read a template string from some file and wanting to insert values into it in a parameter-like way.

Today this can be done in a few ways, but none is quite ideal:

1. Use positional format templates. For example:

   $str = '{0} likes the colour {1}'
   $str -f 'Molly','red'
   

   This is both safe (there's no risk that the template string will have a side-effect) and hygienic (there's no risk the formatting will replace something not intended to be a parameter), but is not readable; it's not clear what the author has in mind in the template from {0} and {1}, and when the template is instantiated, the values "Molly", and "red" are again decontextualised -- we have no idea how they're being used. So the greater the distance/abstraction between the template and its parameters, the harder the script is to reason about. Two more points detract here:

   • Multiple uses of {0} are unclear from both the template and instantiation perspective (hard to keep track of parameters)
   • Heterogeneous templates, for example where some use all parameters and others don't, are hard to manage like this

2. Use string replacement. For example:

   $str = 'VAR_NAME likes the colour VAR_COLOR'
   ($str -replace 'VAR_NAME','Molly') -replace 'VAR_COLOR','red'
   

   This is more readable, and still safe, but:

   • It's not hygienic, consider if the variable was just called COLOUR or if VAR_NAME were replaced with VAR_COLOR. In general there's no system of syntax at work to ensure that parts of the string aren't intentionally replaced.
   • It's not ergonomic; for each variable we must use a new -replace expression
   • It's inefficient; each variable causes a new string to be allocated, when we could really do all of this in one pass

3. Use PowerShell variables. For example:

   $str = '$name likes the colour $color'
   & { $name = 'Molly'; $color = 'red'; Invoke-Expression $str }
   

   This is both hygienic and readable, since it reuses PowerShell's own variable-driven string expansion concepts to drive the template, but:

   • It's not at all safe, since it executes the template string as given. This may cause arbitrary code execution (from within a $(...) subexpression). So if the template is not trusted, then this cannot be used.
   • It's not efficient; we must execute a PowerShell pipeline to do a simple string template instantiation
   • It's not ergonomic to do in a clean way; we are forced to instantiate variables in the calling scope, which is why I invoke it in a new scriptblock in my example, so our parameters don't leak into the wider context

Proposed technical implementation details (optional)

Instead of these options, I think Python sets an excellent example for string formatting. In particular:

params = {
    'purpose': 'find the Holy Grail!'
}

"My quest is to {purpose}".format(**params)

Here the template string can be specified in a way that makes it easily understood in the absence of concrete parameters, while the parameters can listed in a convenient and readable order. There's also a simple correspondence here between the general concept of splatting and named parameters in template strings.

In PowerShell today, we support template strings with positional parameters supplied as an array:

"My quest is to {0}" -f "find the Holy Grail!"

I think it's a logical extension for -f to accept a hashtable:

"My quest is to {purpose}" -f @{
    purpose = "find the Holy Grail!"
}

Just to motivate this a bit further, the reason I opened this issue is that I was confronted with a heterogeneous list of templates stored in JSON, of which some accept different parameters to others, or accept the same parameters in different order. Moreover I could imagine new entries being added that need a different format, which would make things less convenient again with positional parameters:

[
    {
        // Other fields...
        "template": "{packageName}_{release}-1.debian.9_amd64.deb"
    },
    {
        "template": "{packageName}_lts_{release}_{sku}..."
    }
]

In such a scenario, I would love to simply parameterise the strings as above so I can do something like:

Get-Content -Raw ./packages.json |
    ConvertFrom-Json |
    ForEach-Object { $_.template -f @{ packageName = "powershell"; release = "7.2"; sku = "normal"; ... } }

[SeeminglyScience]

Yeah this is something that gets asked about pretty frequently in support channels. There's probably a few issues for it already that could be closed in favor of this one.

Like #12374, #11693, #11412

[rjmholt]

Ah I wondered if there was already an issue, but couldn't find one...

I'll leave this one here for now, and hopefully we can decide on a canonical issue down the track. I notice that @mklement0's issue has the most 👍s, but that's not quite what I'd be looking for. That API is not safe either:

> $ExecutionContext.InvokeCommand.ExpandString('Banana $(Write-Host "Hello")')
Hello
Banana

[mklement0]

@rjmholt, #11693 isn't about surfacing $ExecutionContext.InvokeCommand.ExpandString() _as-is_ - it's just a _starting point_ - the "Security considerations" paragraph in #11693 addresses your security concern.

[iSazonov]

I'd prefer we implement #11693 before we consider the new language enhancement because it seems we would complicate Parser without needs and I don't think we would get a perf win.

[jhoneill]

A couple of observations.

1. -f is AIUI a wrapper for [string]::format so the idea of allowing it to take a hash table, while a good one, is probably better done with a new operator.
2. {placeholder} already comes unstuck if you are trying to insert things into, say, JSON text `

@"
{
   "what": "Some Json",
    "why": {0}
}
"@ -f $reason 

Will fail with input-string was not in the correct format.

@"
{
   "what": "Some Json",
    "why": $reason
}
"@

Is better in code, but it can't be saved to a file and read at run time - either $reason was evaluated when the file was written or it is literal text. And invoking the literal text is bad/unsafe.

Even limited invoke-expression where only $ expressions in the string are evaluated still gives a file a chance to contain "$(Invoke-Evil)" so the hygiene rule says the code reading the string must specify what goes into placeholders.

And so one comes back round to saying today '{0} likes the colour {1}' is using array indices and later $str -f '@(Molly','red') doesn't link nicely. If also gets nasty when the code evolves. '{0} {2} likes the colour {1}' / $str -f '@(Molly','red','really') feels wrong. But if we don't want switch parameters round everywhere we can't change the meaning.
A hash table would be nicer.

[ThomasNieto]

Does .NET have an issue open adding support for named placeholders?

[vexx32]

@jhoneill yeah, if you want to use a string that contains { / } characters you have to escape them by doubling up any of them that aren't intended to be format tokens.

IMO there's nothing inherently wrong with adding this to -f, it follows that pattern reasonably well.

[rjmholt]

@jhoneill good summary.

-f is AIUI a wrapper for [string]::format so the idea of allowing it to take a hash table, while a good one, is probably better done with a new operator.

In an ideal world, I don't think anyone using PowerShell should care about the implementation details for an operator, and I think a hashtable does map nicely, conceptually speaking. With that said, it probably introduces an opportunity for us to break something or otherwise clobber an underlying .NET functionality, which I think PowerShell has done somewhat badly in the past.

{placeholder} already comes unstuck if you are trying to insert things into, say, JSON text

Yeah, any templating scheme is going to have some kind of input that collides with its syntax, which is why it must have an escaping mechanism. The nice thing about the existing -f positional template syntax is that it already covers both the template syntax and how to escape it, so it's only one extra step to introduce naming, rather than teaching people a new operator and possibly a new syntax/mini-language underneath it.

[rjmholt]

@ThomasNieto I was able to find some discussion in https://github.com/dotnet/runtime/issues/20317, but I believe it's been closed. It would be nice to reuse something already well tested and understood, but I'm not sure that's on offer here.

[jhoneill]

@vexx32 / @rjmholt I was thinking it was better to be which has a .NET equivalent and which is PowerShell specific . The more I turn this over the more I think I was wrong - off the top of my head I can't think of two operators which do the same operation but on different types which is what this would be (string -f array . string -g hashtable ) . It would be better done in .NET and then the question doesn't arise.

is the doubling of {} documented ? For more than a decade I've written those as "[[ should be braced ]]" and then replaced [[ and ]] with { and } afterwards. But it seems to have always been there (powershell -version 2 seems to support it)

[SeeminglyScience]

@jhoneill it's in the docs for string.Format.

[vexx32]

But yes, would be good to either link or directly mention the doubling of curly braces in the docs for -f if it's not already there.

C# _natively_ supports things like $"Expand token {name}" which will find an object with the identifier name accessible from the current scope and insert it into the string. So while it's not exposed as an API in .NET directly, the compiler supports it.

[mklement0]

@vexx32, there is a link to the .NET API documentation in the -f documentation, but I agree that mentioning the escaping in-topic for the -f operator would be helpful - see https://github.com/MicrosoftDocs/PowerShell-Docs/issues/6667#issue-707565138

[mklement0]

The C# 6+ interpolated strings (e.g, $"Expand token {name}") are the direct equivalent of PowerShell's _expandable strings_, only with different syntax.

The proposal in https://github.com/dotnet/runtime/issues/20317, which was rejected ("String.Format is already a huge source of bugs and performance issues and adding more layers is not something we really want to do. I like the general idea of a basic templating system but I think it would be better served as a higher level concept not directly on string.") would have been the equivalent of _what we already have_, namely in $ExecutionContext.InvokeCommand.ExpandString(), which #11693 proposes surfacing in a friendlier - and safer - manner _as a cmdlet_: that is, you craft a _verbatim_ string as you normally would an _expandable_ one, and later expand it on demand, with the then-current values of the variables referenced and / or output from the embedded expressions.

Of course, just like -f and expandable strings happily coexist currently, with different syntax forms, there's no reason not to implement _both_ #11693 and the hashtable extension to -f proposed here.

Given that the proposed -f extension would be a strict _superset_ of the .NET String.Format() method, I think it is conceptually unproblematic, as long as the relationship is clearly documented.

[SeeminglyScience]

@mklement0 that seems less like a template and more like slightly different string interpolation. The actual tokens that get replaced are less parameters to the template and more bits of state it pulls out?

So for example, I have a template in email_message.txt that looks like this:

Hello $to,

I'm here to sell you car insurance at `$$price!

In order to use that as a template you'd have to do this?

$to = 'Person'
$price =  '10.00'
Expand-String $myTemplate

[mklement0]

Correct, @SeeminglyScience.
It is string interpolation _on demand_ that isn't tied to _instantly_ interpolated string _literals_.
I think it's fair to call a non-literal string that contains placeholders to be expanded via the caller's state a _template_ - the only difference to this proposal is the _source_ of the expansion (implicit interpolation, via the caller's state vs. explicit interpolation, via a hash table of placeholder replacement values). Note that @BrucePay even favors the name Expand-Template over Expand-String.

However, as my previous comment hopefully made clear: _both_ approaches are valuable and worth implementing.

[SeeminglyScience]

I guess the problem I have with that is that it reminds me of when someone is just starting PowerShell they use variables instead of parameters, like:

function DoThing {
    Get-ChildItem | Export-Csv $global:OutputPath
}

$global:OutputPath = 'C:\'
DoThing

In every scenario I can think of it either ends up looking like that or expandable strings would just be a better fit. That very well might be my lack of imagination though.

However, as my previous comment hopefully made clear: _both_ approaches are valuable and worth implementing.

Fair enough, but realistically this thread is most likely about which of these get implemented. If the consensus ends up being that both are good so do both, we're probably not getting either of them.

[mklement0]

It may remind you of that, but there's no inherent connection. The OP in #11693 shows a better example, and https://stackoverflow.com/search?q=%5Bpowershell%5D+%24ExecutionContext.InvokeCommand.ExpandString should give you a sense that there's a demand for this feature and that the use cases are legitimate.
Just like expandable strings and the current -f operator are complementary, so would Expand-String and -f with named placeholders be, for conceptually related but distinct use cases.

My intent was to compare and contrast the two approaches to show how they differ and relate to C#, so we have conceptual clarity, and to make the case that both are useful. If you think that #11693 isn't a good idea in and of itself, please discuss there.

Fair enough, but realistically this thread is about which of these get implemented.

Please don't prejudice the discussion this way.

If you think this is a question of not having the time/resources to do both, I'd say that #11693 is fairly easy to implement - but, to be clear: I don't think we should have to choose.

Stack Overflow

Human verification

Stack Overflow | The World’s Largest Online Community for Developers

[SeeminglyScience]

The OP in #11693 shows a better example,

I guess that's part of my confusion, all of those examples could be made more clear by using expandable strings:

# Define the template string, with *single quotes*, to avoid instant expansion.
$template = 'Variable `$foo contains ''$foo'' and contains $($foo.Length) character(s).'
# Give $foo different values in sequence, and expand the template with each.
foreach ($foo in 'bar', 'none') {
  $ExecutionContext.InvokeCommand.ExpandString($template) 
} 

# vs

# Give $foo different values in sequence, and expand the template with each.
foreach ($foo in 'bar', 'none') {
  "Variable `$foo contains '$foo' and contains $($foo.Length) character(s)."
} 

'bar', 'none' | Expand-String 'Variable `$_ contains ''$_'' and contains $($_.Length) character(s).'

# vs

'bar', 'none' | ForEach-Object { "Variable `$_ contains '$_' and contains $($_.Length) character(s)." }

and https://stackoverflow.com/search?q=%5Bpowershell%5D+%24ExecutionContext.InvokeCommand.ExpandString should give you a sense that there's a demand for this feature and that the use cases are legitimate.

There's definitely a demand for template functionality, and the ExpandString method is definitely what some folks have often tried to use in lieu of proper template support.

Fair enough, but realistically this thread is about which of these get implemented.

Please don't prejudice the discussion this way.

It's unlikely that multiple proposals for string templating will be accepted and imo that's a good thing. I would argue against having separate competing ways to solve this particular problem.

If you think this is a question of not having the time/resources to do both, I'd say that #11693 is fairly easy to implement

Anything security sensitive like arbitrary code execution would likely take same time to verify that it's secure. Even if the implementation ended up being fairly simple.

Stack Overflow

Human verification

Stack Overflow | The World’s Largest Online Community for Developers

[jhoneill]

@SeeminglyScience I think the difference between expandable strings and what @mklement0 is talking about was your mailshot example.
I write a script

Function new-output {
Param ($to, $price)
@"
Hello $to,

I'm here to sell you car insurance at `$$price!
"
}

And its works just fine.

Now someone comes along and says "OK but we want to be able to send different forms of the mail shot with different text, can you read it from a file"

And that works today IF marketing understand they have to write

Hello {0},

I'm here to tell you we are offering everyone in {1} {2} insurance at a {3}% discount on normal premiums. 

But do they know numbers start at zero and the type of insurance is 2 etc. Would it be easier if they could put something else in the braced numbers ? They could ,and we could use invoke-expression so

Hello $to

works - but so does

I'm here to $(invoke-malware)

So how does one get a file where place holders are easy without a risk of getting things one should not be able to get ?

[SeeminglyScience]

@jhoneill Yeah I definitely understand the value of string templates. In fact, one of the first modules I put out was PSStringTemplate.

What I'm getting at is defining arbitrary bits of state like variables as template parameters isn't good UX.

e.g.

$template = 'Hello $to'
$to = 'Person'
Expand-String $template

# vs

$template = 'Hello {to}'
$template -f @{ to = 'Person' }

# or even

$template = 'Hello $To'
Expand-String $template -To Person

[mklement0]

Thanks, @jhoneill - that is indeed why Expand-String would be useful: it works when you're _given_ a string (template) _from the outside_ - such as having read it from a file - which is therefore impossible to express as an expandable string _literal_ - I've clarified the OP of #11693 accordingly.

@SeeminglyScience:

It's unlikely that multiple proposals for string templating will be accepted and imo that's a good thing

By that logic you should oppose that PowerShell supports both expandable strings and the -f operator - two features that ultimately do the same thing. The two approaches we're discussing here are _natural extensions_ to both.

Anything security sensitive like arbitrary code execution would likely take same time to verify that it's secure

I don't think so:
If the processing is AST-based, it should be easy to rule out non-variables or to limit command use to a predefined safe subset, as @rkeithhill suggests.

For someone who wants to roll their own, variables-only stopgap version (just to show a simplistic, but effective approach to ruling out commands):

function Expand-String {
  param(
    [Parameter(Mandatory, ValueFromPipeline)] [string] $Template
  )
  process {
    $sanitized = $Template -replace '\$\(', "`0"
    $ExecutionContext.InvokeCommand.ExpandString($sanitized) -replace "`0", '$$('
  }
}
'$HOME is where the heart is. $(Write-Host "Oh noes!")' | Expand-String
# -> '/Users/jdoe is where the heart is. $(Write-Host "Oh noes!")'

arbitrary bits of state like variables as template parameters isn't good UX.

With _both_ approaches there's a tight coupling between the template placeholder and the calling code that instantiates it, and it's actually tighter in the -f case - and whether that's preferable depends on the use case.

It's a tradeoff between being the convenience between direct use of elements of the caller's state you can rely on - e.g., automatic variables such as $HOME in an Expand-String call - and having to always explicitly pass _all_ replacement values in an -f expression.

With _both_ approaches you can make the operation fail if a bit of caller state is missing / if a replacement value isn't passed (analogous to '{0}{1}' -f 'one' failing).

[SeeminglyScience]

By that logic you should oppose that PowerShell supports both expandable strings and the -f operator - two features that ultimately do the same thing. The two approaches we're discussing here are _natural extensions_ to both.

In the next sentence I say: "I would argue against having separate competing ways to solve this particular problem."

Plus I dunno man, I wasn't there when they decided to add it. Maybe I would have argued against it but that bell can't be unrung.

If the processing is AST-based, it should be easy to rule out non-variables or to limit command use to a predefined safe subset, as @rkeithhill suggests.

Like I said, even if the implementation was simple that doesn't mean it wouldn't take some time to verify.

With _both_ approaches there's a tight coupling between the template placeholder and the calling code that instantiates it, and it's actually tighter in the -f case - and whether that's preferable depends on the use case.

What I'm saying is that we don't recommend that folks do this:

function DoThing {
    "Hello $global:To!"
}

$global:To = 'Person'
DoThing

and instead we recommend:

function DoThing {
    param($To)
    "Hello $To!"
}

DoThing -To 'Person'

So for the same reasons we recommend the latter, we shouldn't introduce a feature that requires the former.

[jhoneill]

>

What I'm getting at is defining arbitrary bits of state like variables as template parameters isn't good UX.

I agree. But how is this

$template = 'Hello {to}'
 $template -f @{ to = 'Person' }

Worse than

$template = 'Hello {0}'
$template -f @{ 0 = 'Person' }

OK we don't create collections with keys of 0,1,2,3... as hash tables we use arrays but that's what -f has today, a collection where we assign things by numeric index within the collection.

At some point if you use a template the person using it has to know what the place holders are. Text is easier than numbers for both parties.

When you see this away from the template, you need to go somewhere else to see what it means.
$template -f $datarow.colx , $datarow.coly , $datarow.colz

but this tells you where each is going into the template
$template -f @{to=$datarow.colx ; place= $datarow.coly ; discount= $datarow.colz

As a general principle supporting parameters-by-name is good, and only allowing parameters-by-position not-so good from a UX point of view.

[SeeminglyScience]

@jhoneill Yeah that's the syntax I'm advocating for.

[mklement0]

@SeeminglyScience:

"I would argue against having separate competing ways to solve this particular problem."
but that bell can't be unrung.

The particular problem being solved is simply a generalization of existing, well-established functionality in two existing incarnations, both of which users have embraced.

it wouldn't take some time to verify.

Why? It seems like a straightforward problem to solve, verifiable with straightforward tests.

The advantage of the Expand-String approach is that a template may be (mostly) _self-contained_, if it references only automatic variables and commands declared as safe (or even unsafe ones with a -Force override).
Also, if in a given scenario the replacement values are already stored in variables (e.g., passed as arguments to a function that encapsulates expansion of a template), you're spared the awkwardness of 'This is {us}' -f @{ us = $us } (vs.
Expand-String 'This is $us')

[mklement0]

On a meta note, @SeeminglyScience, quoting in chronological order:

(a)

realistically this thread is about which of these get implemented.

(b)

I would argue against having separate competing ways to solve this particular problem.

Notwithstanding the fact that both statements, in context, are couched in _conjecture_ about the _likelihood_ of an associated outcome, I ask you not to conflate (b), an expression of _personal preference_, with (a), an assertion of _objective reality_.

[SeeminglyScience]

Notwithstanding the fact that both statements, in context, are couched in _conjecture_ about the _likelihood_ of an associated outcome, I ask you not to conflate (b), an expression of _personal preference_, with (a), an assertion of _objective reality_.

Fair enough, the language is too definitive. I've changed it to:

realistically this thread is most likely about which of these get implemented.