Powershell: Question: not getting expected results for get-formatdata

[JoeSalmeri]

This is a followup to issue #8063 which appears closed but which did not fix the originally reported problem.

Steps to reproduce

Hi Steve,

I have been anxiously awaiting this fix to test out the issues we discussed last year.

Unfortunately the problems do not appear to be fixed and things appear to be seriously broken still to the point that the results are unusable.

Run these commands

pwsh -noprofile
Get-FormatData -PowerShellVersion 6.2 -TypeName System.IO.DirectoryInfo | Export-FormatData -LiteralPath DirectoryInfo.ps1xml
Get-FormatData -PowerShellVersion 6.2 -TypeName System.IO.FileInfo | Export-FormatData -LiteralPath FileInfo.ps1xml
If you compare DirectoryInfo.ps1xml and FileInfo.ps1xml you will find the are exactly the same, however, the output for a directory item versus a file item should not be the same as a directory does not contain a length.

Ignoring that issue, the results returned are missing lots of XML code that was previously in FileSystem.format.ps1xml.

I won't detail them all here as you can compare the results produced of Get-FormatData with the old FileSystem.format.ps1xml file to see them all, however, I will point out a few.

FileSystem.format.ps1xml created a SelectionSet named FileSystemTypes and then used that and a CustomControlName FileSystemTypes-GroupingFormat for the view definitions. Those are all missing from the "Get-FormatData" results for System.IO.DirectoryInfo / System.IO.FileInfo.

Ignore those issues, I attempted to merge my changes into the results returned from Get-FormatData and I found that my changes were not displaying so I did this simple test.

pwsh -noprofile
Get-FormatData -PowerShellVersion 6.2 -TypeName System.IO.DirectoryInfo | Export-FormatData -LiteralPath DirectoryInfo.ps1xml

Edit the DirectoryInfo.ps1xml file and make the following change:

Look for <Label>LastWriteTime</Label> and change it to <Label>XXXXWriteTime</Label>

Now run

Update-FormatData -PrependPath .\DirectoryInfo.ps1xml

Now run Get-ChildItem and you will see that the label still says LastWriteTime instead of XXXXWriteTime

Now run this to get what the updated formatting info looks like

Get-FormatData -PowerShellVersion 6.2 -TypeName System.IO.DirectoryInfo | Export-FormatData -LiteralPath DirectoryInfo2.ps1xml

You will find that it does show the change, however. there is another section after it that still has the label as "LastWriteTime".
I believe all of these issues are because Get-FormatData is not returning the full and correct format data for these types.

If I run "Update-FormatData -Prepend My,FileSystem.format.ps1" which uses the modified version of the old FileSystem.format.ps1xml file that used to be included in PowerShell, then everything works correctly and it has my changes.

So we are back to the original problem.

There needs to be a way for me to obtain the format data being used (either via a file like it previously did or with Get-FormatData returning the correct formatting that is really being used).

Since PowerShell is no longer including the *.format.ps1xml files and Get-FormatData is broken and does not return the correct format data I have no way to determine what changes were made in new versions of PowerShell so that I can integrate them into my modified version.

I hope that this can be made a priority and get fixed soon.

Holler if you want any more details.

Thanks!

Joe

[TravisEz13]

@JoeSalmeri
We actually do you one format definition to display both objects. The scripts embedded in the object take are of the issues of displaying or not displaying the length:

$format = get-formatdata -TypeName System.IO.FileInfo -PowerShellVersion 6.2
$format.FormatViewDefinition[0].Control.Rows[0].Columns[2].DisplayEntry.Value

results:

if ($_ -is [System.IO.DirectoryInfo]) { return '' }
if ($_.Attributes -band [System.IO.FileAttributes]::Offline)
{
    return '({0})' -f $_.Length
}
return $_.Length

[JoeSalmeri]

@TravisEz13

That explains why Get-FormatData for System.IO.DirectoryInfo and Get-FormatData for System.IO.FileInfo return the same results.

I figured that you might be doing something like as part of the cleanup when the format.ps1xml files were integrated into the code, however, that does not address the main issue I reported that was supposed to be fixed.

Either Get-FormatData is not returning the data properly or Export-FormatData is not properly formatting it into XML (or I'm making a mistake but I don't believe that is the case).

First some quick background:

I have some customization that I have made to the old FileSystem.format.ps1xml file. With each new version of PS I have taken the newest version and integrated my changes into it to make sure that I picked up any changes that MS made.

PS Core removed the format files and integrated them into the C# code and the solution I was given was to use Get-FormatData and Export-FormatData to create the XML which I could then tweak and then apply using Update-FormatData -Prepend.

Get-FormatData had bugs which were to be fixed in PS Core 6.2 (by Steve)

With PS Core 6.2 available now I am testing out that functionality and the ability for me to re-integrate my changes.

Ok, now that you have the history here is a simple example that is not working properly:

pwsh -noprofile
Get-FormatData -PowerShellVersion 6.2 -TypeName System.IO.DirectoryInfo | Export-FormatData -LiteralPath DirectoryInfo.ps1xml

Edit the DirectoryInfo.ps1xml file and make the following change:

Look for LastWriteTime and change it to XXXXWriteTime
Now run

Update-FormatData -PrependPath .\DirectoryInfo.ps1xml

Now run Get-ChildItem and see that it is not displaying the XXXXWriteTime label it is displaying LastWriteTime still.

Clearly there appears to be something wrong with roundtrip between Get-FormatData, Export-FormatData, Update-FormatData as a simple change like that should have not had any issues.

Note: This is a simple change to illustrate that either Get-FormatData or Export-FormatData has a problem, my actual customization does a bunch of other things.

When I compare the DirectoryInfo.ps1xml file created from Get-FormatData and Export-FormatData there are many differences from the old FileSystem.Format.ps1xml file (I listed some in my earlier reply).

To me it seems like the issue is still with Get-FormatData not properly returning the correct format data for DirectoryInfo so when I modify it and then try to reapply it does not work.

If I am making a mistake, could you please point out to me how to properly do the roundtrip to get the format data for DirectoryInfo as XML that I can modify and then reapply?

Thanks very much!

Regards,

Joe

[JoeSalmeri]

@TravisEz13

Hi Travis, never heard anything back on this.

With the removal of the format files, there really needs to be a way to get the format data, export it, modify the exported format data and then re-apply the update.

The proposed solution I was given does not seem to work because EITHER Get-FormatData is still not fixed to get ALL the format data OR more likely Export-FormatData is not properly exporting all the format data retrieved.

Here are the commands I was told to use:

Get-FormatData -PowerShellVersion 6.2 -TypeName System.IO.DirectoryInfo | Export-FormatData -LiteralPath DirectoryInfo.ps1xml

Modify DirectoryInfo.ps1xml using my editor/tools.

Update-FormatData -PrependPath .\DirectoryInfo.ps1xml

It is quite disappointing to see functionality removed when the replacement functionality does not appear to work properly

If I am incorrect and there are no more problems with Get-FormatData and Export-FormatData could you PLEASE provide some documentation or address my issue in that a roundtrip does not appear to be working?

This is on PowerShell Core 6.2 where it was all supposed to be fixed.

Thanks for your help.

Joe

[vexx32]

To be honest we really need to have a cmdlet or something to _create_ format data that hooks into the C# APIs so that it's a lot easier to work with. Would be great to have a DSL-style module to make it really simple to work with. A lot of the C# APIs do a lot of method chaining to get things done, might be interesting to try to expose that as a series of cmdlets for use in PS, and improve Update-FormatData to accept this kind of input.

[JoeSalmeri]

@vexx32

I agree that would be a welcome change.

My issue is that the files are gone now and the methods that would allow me to modify still seem to be broken. I am still using my modified FileSystem.format.ps1xml file from PS 5. In the past I would check whenever that file was updated for any changes to make sure I kept my file updated with future PS changes.

Looking at my changes they really come down to changing the format for numbers (length attribute) and dates. The other changes are to adjust the width in output to handle the change in size for the number and date formats.

In Windows the formatting of Dates and Numbers is configured via Control Panel / Region and in Linux and also in other programming languages like Python they are configurable via locale.

Since those are user defined settings for the formatting we are discussing it seems like PS should be using those settings instead of what it currently does or maybe we just need something like

 SET-DEFAULT-NUMBER-FORMAT
 SET-DEFAULT-DATE-FORMAT
 SET-DEFAULT-DATETIME-FORMAT
 SET-DEFAULT-TIME-FORMAT

If those existed then I suspect that I would not need the width changes I make because PS would adjust it's output based on the widths defined in those defaults.

Is this even on the radar of any of the developers since we lost functionality and the suggestion methods are broken?

Thanks!

[vexx32]

Number and datetime formats are typically controlled by culture settings, if I'm not mistaken? You might be able to adjust those from the objects Get/Set-Culture work with.

[JoeSalmeri]

Hi Vexx32,

Sorry just saw this as I did not receive the normal email notification.

(Get-Culture).NumberFormat has the desired settings but they are not being used by the built in formats.

[vexx32]

I would try modifying the settings with the objects you get from Get-Culture, and then pass them back into Set-Culture and/or $PSCulture/$PSUICulture

$Culture = Get-Culture
$Culture.NumberFormat = <# set number format #>
$Culture | Set-Culture
$PSUICulture = $PSCulture = $Culture

[JoeSalmeri]

@vexx32,

(Get-Culture).NumberFormat has

NumberGroupSizes : {3}
NumberGroupSeparator : ,

but the views defined for System.IO.DirectoryInfo and System.IO.FileInfo do not use the culture information when displaying the file sizes (length property).

In my custom format.sp1xml I just format them the way I want but it seems to me that the real issue is that the views defined do not use the culture settings. If the internal views were changed to use the culture settings then everyone would get the values displayed in their native format (or could change the culture settings as you suggested) and I would not need my custom formats.

[marckassay]

I believe all of these issues are because Get-FormatData is not returning the full and correct format data for these types.

@JoeSalmeri, just dropping in real quick as I haven't read this thread fully but did a search for 'scriptblock' and found nothing. You suspected it may be Get-FormatData causing issues. But that is being piped into Export-FormatData. That format type, DirectoryInfo, contains scripts, which if needed, would have to be explicitly requested by using the IncludeScriptBlock switch parameter for Export-FormatData.

[JoeSalmeri]

@marckassay

Thanks for chiming in regarding this issue.

Just to summarize for you.

PowerShell no longer includes the *.ps1xml files that previously shipped with it.

Previously whenever a new PS version came out, I copied the ps1xml file and reapplied my customizations and then used Update-FormatData to load my changes. I did this so that my customizations always included any changes that MSFT made to the ps1xml files.

Once the files were removed I have been trying to use Get-FormatData to get the full source for DirectoryInfo so that I can apply my customizations to it.

Originally there was a bug (which is supposed to be fixed now) but I have tried the recommendations and still do not receive the full format data.

Here are the commands recommended that I am using (with your suggestion now included):

Get-FormatData -TypeName System.IO.DirectoryInfo -PowerShellVersion 7.0 |
Export-FormatData -LiteralPath DirectoryInfo.ps1xml -IncludeScriptBlock

If I issue that command AFTER I load my customizations then the DirectionInfo.ps1mxl file DOES contain the ScriptBlocks that I had defined in my ps1xml file.

If I start powershell without my profile (which loads my customizations) using

pwsh -noprofile

and then issue the above Get-FormatData command then the DirectoryInfo.ps1xml file DOES NOT contain any ScriptBlocks.

After initially reporting the issue about not being able to get the full format data for DirectoryInfo, I also asked how the "length" field information was handled using the new FormatData (where DirectoryInfo and FileFileInfo were merged) because only files have a length.

Above you can see @TravisEz13 reply where he showed the following:

$format = get-formatdata -TypeName System.IO.FileInfo -PowerShellVersion 6.2
$format.FormatViewDefinition[0].Control.Rows[0].Columns[2].DisplayEntry.Value

Which produced

if ($_ -is [System.IO.DirectoryInfo]) { return '' }
if ($_.Attributes -band [System.IO.FileAttributes]::Offline)
{
return '({0})' -f $_.Length
}
return $_.Length

However, using the Get-FormatData command above, that information does not appear in the ps1xml file.

To summarize, I am trying to get the full format data for DirectoryFileInfo so that I can export it customize to my liking and then Update-FormatData to load my customizations.

I use this approach so that if Microsoft ever makes changes to the format data then I will not lose those changes in my customizations.

If you issue the pwsh -noprofile command followed by the Get-FormatData command above and look at the resulting DirectoryInfo.ps1xml file you will see that no Script blocks exist and it does NOT include the ScriptBlock shown by @TravisEz13

Interestingly the resulting XML contains the following:

<PropertyName>LengthString</PropertyName>

however, if you issue

dir | gm *

the resulting object does not contain a LengthString property.

If you issue the following commands from @TravisEz13 now on PS 7 rc 2

$format = get-formatdata -TypeName System.IO.FileInfo -PowerShellVersion 6.2
$format.FormatViewDefinition[0].Control.Rows[0].Columns[2].DisplayEntry.Value

you get
LengthString

So something else has changed again since @TravisEz13 reply

One of the formating changes that I make in my customizations is to add commas to the length property. Above you will see a recommendation to use Get-Culture to configure my desired format, however, that does now work because DirectoryInfo does not honor the culture settings.

So the end goal is to use Get-FormatData and Export-FormatData to produce the complete format data that is being used by DirectoryInfo now that the ps1xml files are removed.

It appears that this is still broken and not possible to do.

It is frustrating that the ps1xml files were removed before the replacement code to extract the format data was working properly.

This is not an intermittent problem and therefore it should be easy to debug why we can no longer get the complete formatdata for DirectoryInfo.

Please try the above commands and you will see what I am talking about.

Hopefully someone will investigate and either fix the problem or provide the correct commands to get the full format data.

Holler if you need more details.

Thanks again!

[marckassay]

@JoeSalmeri thanks for the thorough explanation! I didn't realize that ps1xml are excluded in shipment until you mentioned, as it is still mentioned in the docs, and obviously still can be used in PowerShell. So it's just PowerShell today doesn't return all or correct ps1xml data, in the scenarios you mentioned.

[JoeSalmeri]

@marckassay sure no problem!

You are correct (at least last time I had checked) that the documentation still references the ps1xml files.

It is really frustrating that the files were removed before the complete replacement solution was in place.

Unfortunately the issue does not seem to get any priority.