Powershell: Feature Request: Import-PowershellDataFile should accept string input

[jazzdelightsme]

Summary of the new feature/enhancement

Currently Import-PowershellDataFile takes a path to a file on disk. But what if I have the .psd1 content in memory? (Such as in a string, as downloaded from AzDO or github or such.)

Proposed technical implementation details (optional)

Another parameter set that accepts a string.

[vexx32]

If you have it as a string, you can use Invoke-Expression since its just a hash literal.

If there were to be another cmdlet for it, I'd imagine it'd follow the CSV pattern and have a ConvertFrom/ConvertTo cmdlet for in-memory conversions.

[jazzdelightsme]

If you have it as a string, you can use Invoke-Expression

I thought the point of Import-PowershellDataFile is that it lets you import data in a safe way, without executing code in the file.

(In other words, if what you suggest were an acceptable solution, then we would not need the Import-PowershellDataFile in the first place.)

[vexx32]

Aye, just mentioning something that works for some use cases for the time being. You can manually construct a script block and verify no commands are being used in it if you prefer as well, but I agree it would be better handled by a cmdlet.

[mklement0]

As an alternative to a new cmdlet, we could think about generalizing @JamesWTruher's idea proposed in the context of https://github.com/PowerShell/PowerShell/issues/4332#issuecomment-643486358: To allow targeting a variable via a provider path; e.g.:

$dataFileContent = ... # read *.psd1 file into memory

Import-PowerShellData -LiteralPath variable:dataFileContent

All file-processing cmdlets could benefit from this, notably Import-CliXml / Export-CliXml, which would obviate the need to implement #3898 (separate ConvertTo/From-CliXml cmdlets).

I've always felt that the current dichotomy - one cmdlet for in-memory operations, another for file operations - is clumsy, both in terms of UX and implementation.

[vexx32]

Hmm. If the cmdlet is using provider paths correctly I would imagine they should already support that. Is there a specific reason they don't?

[SeeminglyScience]

@vexx32 iirc that command just does something like this:

Parser.ParseFile(path, out _, out _).Find(ast => ast is HashtableAst, false).SafeGetValue();

So it's quite a bit cheaper to use that directly rather than calling the content provider API's. Side note, you can do something similar to that but with ParseInput as a workaround.

[mklement0]

While Get-Content supports it, here's how the file-processing Import-* cmdlets fail: all but Import-Alias do the same thing as Import-PowerShellDataFile, as presumably explained by @SeeminglyScience:

$foo = 'hi there'
gcm import-* | where name -notmatch 'pssession|packageprovider|module|localizeddata' | % {
  $null = try { & $_.Name -LiteralPath variable:foo -ea SilentlyContinue -ev e } catch { $e = $_ }
  [pscustomobject] @{
    Cmdlet = $_.Name
    Error = "$e"
  }
}

Cmdlet                    Error
------                    -----
Import-Alias              Cannot import the alias because the specified path 'variable:foo' referred to a 'Microsoft.PowerShell.Core\Variable' provider path. Change the value of t…
Import-Clixml             Cannot open file because the current provider (Microsoft.PowerShell.Core\Variable) cannot open a file.
Import-Csv                Cannot open file because the current provider (Microsoft.PowerShell.Core\Variable) cannot open a file.
Import-PowerShellDataFile Cannot open file because the current provider (Microsoft.PowerShell.Core\Variable) cannot open a file.

[vexx32]

Yeah looks like they explicitly check for other providers and prevent them from being used.

IMO they should just... not, really. 😂

New-WordCloud from my PSWordCloud module allows this kind of functionality (writing to arbitrary PSProviders). The only real issue I ran into was that not all providers (the variable provider is one of these) support the Content.Clear() operation, so you do have to catch that and (probably) swap it for an Item.Remove() operation instead (if the provider supports that) if you want to be able to do Set-Content style things which clear data before writing.

[SeeminglyScience]

Yeah looks like they explicitly check for other providers and prevent them from being used.

IMO they should just... not, really. 😂

I'd be willing to bet none of these commands are written in a way that make this that simple. Nor should they be really, the content API's are super slow and allocate way too much. It would be nice to have a fallback of sorts though:

// sort of psuedo code that doesn't account for a lot of things
public static string ReadProviderItemAsString(this PSCmdlet cmdlet, string path)
{
    string providerPath = cmdlet.SessionState.Path.GetUnresolvedProviderPathFromPSPath(
        path,
        out ProviderInfo provider,
        out _);

    if (provider.Name is FileSystemProvider.ProviderName)
    {
        return File.ReadAllText(providerPath);
    }

    using IContentReader reader = cmdlet.SessionState.InvokeProvider.Content.GetReader(
        new[] { path },
        force: true,
        literalPath: true)[0];

    var text = new StringBuilder();
    bool first = true;
    for (IList items = reader.Read(1); items is { Count: > 0 }; items = reader.Read(1))
    {
        foreach (object item in items)
        {
            if (first)
            {
                first = false;
            }
            else
            {
                text.AppendLine();
            }

            if (item is null)
            {
                continue;
            }

            text.Append(item);
        }
    }

    return text.ToString();
}

That said, there are theoretically providers that don't necessarily deal with string content. So that might end up being a bunch of default Object.ToString() calls.

That also doesn't account for commands whose implementation might be able to work directly with a FileStream, or might call a native API that takes a file system path.

[vexx32]

Sounds like we need a provider Content API that doesn't just unilaterally assume strings, but that's a different conversation. ^^