FlashExtact is MSR project.
@LeeHolmes originally took code of the project instead of nuget dependencies, because of the Windows build requirements.
We should replace FlashExtract sources by Nuget packages and bring the code for ConvertFrom-String
.
We also need to figure out licensing for this component.
Home page: https://microsoft.github.io/prose/
Nuget packages: https://www.nuget.org/packages/Microsoft.ProgramSynthesis/
To be clear, this is what we need for ConvertFrom-String
Moving it to Future.
We are probably do it as a separate module and ship thru the gallery.
Home page: https://microsoft.github.io/prose/
Nuget packages: https://www.nuget.org/packages/Microsoft.ProgramSynthesis/
It seems this project for a long time had no updates. Is it frozen?
I see now ProgramSynthesis lives! Current version is 4.0.0.
Can we get any progress with the Issue?
Don't think we'll get to this for 6.0.0, putting it in 6.1.0 for now
This is a really important feature, I am sad to see it moved to 6.2
@AceHack we need an OSS implementation of the FlashExtract engine before we can port this to PSCore6
Looks like https://www.nuget.org/packages/Microsoft.ProgramSynthesis/ has been getting regular updates, so this is viable.
@SteveL-MSFT Can MSFT share again the code of Convert-String and Convert-From-String that was removed from the repo?
@iSazonov Let me see if I can add it back, the problem is making sure to remove any use of private APIs...
@iSazonov I took a look at this. The Windows PowerShell code for ConvertFrom-String
and Convert-String
is 90% proprietary code with 10% being the cmdlet code that calls a wrapper class to do the actual work. It may be easier to just create a new set of cmdlets for similar functionality.
I'm not sure that's really worth the effort. In every case where I've tried to use that cmdlet for anything more than the simplest tasks, I've ended up giving up after a good hour or so and just using regex instead.
A simple cmdlet to generate PSCustomObjects from strings via a regex pattern and match groups would be far more useful (if pretty straightforward) in my opinion.
@SteveL-MSFT Please clarify - can we re-implement the cmdlets (with same names) or we need create new cmdlets (with new names)? For first case we need you share Pester tests to ensure compatibility.
@iSazonov It's not clear to me how many people actually used those cmdlets that we would need to retain syntax. I've always found them difficult to use with inconsistent results, so I didn't use them much myself. My suggestion would be to start with understanding the use cases and optimizing for that rather than focusing on compatibility. Looking in the old code base, there isn't much tests besides some basic ones:
Describe "DelimitedText" -tags 'DRT' {
It "verifies automatic property generation" {
$result = "Hello 9", "Hello 10", "Hello 90" | ConvertFrom-String
## Verify first properties got extracted
$result[0].P1 | Should be 'Hello'
$result[1].P1 | Should be 'Hello'
$result[2].P1 | Should be 'Hello'
## Verify second properties got extracted
$result[0].P2 | Should be 9
$result[1].P2 | Should be 10
$result[2].P2 | Should be 90
}
It "verifies property overflow generation" {
$result = "Hello 9" | ConvertFrom-String -PropertyNames A
$result.A | Should be 'Hello'
$result.P2 | Should be 9
}
It "verifies property renaming" {
$result = "Hello 9" | ConvertFrom-String -PN B,C
$result.B | Should be 'Hello'
$result.C | Should be '9'
}
It "verifies property typing of numbers" {
$result = "Hello 9" | ConvertFrom-String -Property B,C
$result.C.GetType().FullName | Should be 'System.Byte'
}
It "verifies property typing of TimeSpan" {
$result = "Hello 1:00" | ConvertFrom-String -Property B,C
$result.C.GetType().FullName | Should be 'System.TimeSpan'
}
It "verifies property typing of DateTime" {
$result = "Hello 1/1/2012" | ConvertFrom-String -Property B,C
$result.C.GetType().FullName | Should be 'System.DateTime'
}
It "verifies property typing of Char" {
$result = "Hello A" | ConvertFrom-String -Property B,C
$result.C.GetType().FullName | Should be 'System.Char'
}
It "verifies empty strings don't turn into INTs" {
$result = "Hello" | ConvertFrom-String -Delimiter 'l'
$result.P2.GetType().FullName | Should be 'System.String'
}
It "verifies property typing of String" {
$result = "Hello World" | ConvertFrom-String -Property B,C
$result.C.GetType().FullName | Should be 'System.String'
}
It "verifies the ability to change the delimiter" {
$result = "Hello-World" | ConvertFrom-String -Delimiter '-'
$result.P1 | Should be 'Hello'
$result.P2 | Should be 'World'
}
It "verifies that only matching text gets parsed" {
$result = "Foo1","Hello1 World1","Hello-World" | ConvertFrom-String -Delimiter '-'
$result.P1 | Should be 'Hello'
$result.P2 | Should be 'World'
@($result).Count | Should be 1
}
It "verifies that a good error message is returned from an invalid regular expression" {
try
{
$result = "Hello World" | ConvertFrom-String -Delimiter '['
}
catch
{
$errorRecord = $_
}
$errorRecord.FullyQualifiedErrorId | Should be "InvalidRegularExpression,Microsoft.PowerShell.Commands.StringManipulation.ConvertFromStringCommand"
}
}
$here = Split-Path -Parent $MyInvocation.MyCommand.Path
$data = Join-Path $here "StringManipulationData\ConvertString"
Describe "Convert-String test cases" -tags 'DRT' {
It "Changes first and last name with one Example" {
$result = "Gustavo Soares" | Convert-String -Example "camilla araujo=araujo, c."
$result | Should be "Soares, G."
}
It "Changes first and last name with one Example, and three inputs" {
$result = "Lee Holmes", "Gustavo Soares", "Sumit Gulwani", "Vu Le" |
Convert-String -Example "camilla araujo=araujo, c."
$result[0] | Should be "Holmes, L."
$result[1] | Should be "Soares, G."
$result[2] | Should be "Gulwani, S."
$result[3] | Should be "Le, V."
}
It "Changes first and last name with two Examples" {
$examples = [PSCustomObject] @{ Before = 'camilla araujo'; After = 'araujo, c.' },
[PSCustomObject] @{ Before = 'lee holmes'; After = 'holmes, l.' }
$result = "Gustavo Soares" | Convert-String -Example $examples
$result | Should be "Soares, G."
}
It "Changes first and last name with one dictionary example" {
$result = "Gustavo Soares" | Convert-String -Example @{ Before = "camilla araujo"; After = "araujo, c." }
}
It "Changes first and last name with two dictionary example" {
$result = "Gustavo Soares" | Convert-String -Example @(@{ Before = "camilla araujo"; After = "araujo, c." },@{ Before = "vu le"; After = "le, v." })
}
It "Check invalid text example" {
{ "Gustavo Soares" | Convert-String -Example "camilla araujo" } | Should Throw
}
It "Check invalid psobject examples" {
$examples = Import-Csv $data\incorrect-examples.csv
{ "Gustavo Soares" | Convert-String -Example $examples } | Should Throw
}
It "Replace by empty" {
$examples = Import-Csv $data\replace-name-by-empty.csv
$result = "Gustavo Soares" | Convert-String -Example $examples
$result.length -eq 0 | Should be true
}
}
@SteveL-MSFT Thanks!
My suggestion would be to start with understanding the use cases
I agree. That the two cmdlets look like a small demo. PROSE home site has more samples. Parsing log, trace, tabular files, JSON, YAML.
PROSE implements still many great APIs.
I've always found them difficult to use with inconsistent results
Unfortunately, that has been my experience as well.
What do you have in mind? Remove or improve?
I would really like to see this or similar functionality in core.
The question is how to do it more conveniently.
Am I missing something or don't ConvertTo and ConvertFrom essentially solve the same problem?
e.g.
$csv = ConvertFrom-String $stringValue -Format CSV
versus
$csv = ConvertTo-CSV $stringValue
I'm also not really that keen about a "ConvertFrom-String" given how common a string input can be as it ends up being a swiss-army-knife function .. it has to support a potentially massive number of potential output formats / types.
My initial thoughts (although I really haven't analysed this in detail) from a code management/quality perspective to have separate parsing functions from each object.. so it is clear which objects support parsing from a string (and other formats), and which ones do not.
Just my $0.02 - but would be genuinely interested why it is thought "ConvertFrom" is a better approach?
Hi there, i wonder if there is a new update on this? will we have it in recent powershell version? I am porting some code into Azure function app and would be great to be able to continue to use this...thanks.
@htwashere You could use a proxing with https://github.com/PowerShell/WindowsCompatibility
GitHubModule that allows Windows PowerShell Modules to be used from PSCore6 - PowerShell/WindowsCompatibility
As a proxy solves this problem, but only for Windows. I just came looking for this because I wanted to look at the source code, since it could use some improvements.
I agree that this cmdlet can be difficult to use; however, it is also of significant value when you are parsing string data, which you end up doing a lot with native executables, of which there are plenty these days on Windows, Linux, and macOS. It is much easier than using regex
to do the same.
Anyhow, I wanted to voice my opinion that I think this functionality should be brought forward in an open source command so that we can enhance it without reinventing the wheel.
Also for reference, places where I am using this cmdlet right now in Windows PowerShell:
.sln
and .csproj
files -- tons of content here, all rich metadata about a solution, and I just want to pull it out and identify what exactly is in the solution so that I can automate the doc generation for that rather than do it manually and have to keep it currentI wanted to look at the source code
It is in source-deport
git branch.
I agree that this cmdlet can be difficult to use
Perhaps we could design some helper cmdlets.
It is in
source-deport
git branch.
Link? I browsed to that branch, the only thing a search for ConvertFromStringCommand
turns up is this issue.
:-) You could look history before https://github.com/PowerShell/PowerShell/commit/0db07f84354ef661f2c1a33676e6346248a40e16
I should say that this will help little because the PROSE engine has been completely rewritten.
Most helpful comment
:-) You could look history before https://github.com/PowerShell/PowerShell/commit/0db07f84354ef661f2c1a33676e6346248a40e16
I should say that this will help little because the PROSE engine has been completely rewritten.