Powershell: Changes to FileSystem.Format.ps1xml

[JoeSalmeri]

PowerShell 6.1.0 is missing the format.ps1xml files.

I have some custom modifications which I have made to the FileSystem.Format.ps1xml file and have updated/merged them into new PS versions of the file whenever new PS versions were released.

This was discussed in issue #7973 with Steve-L from MSFT as it was they are simple changes which would be beneficial to all users and he suggested I submit those changes for inclusion in the product.

Attached please find the My.FileSystem.format.ps1.zip file contains My.FileSystem.format.ps1 which includes the changes.

This file is a copy of the latest FileSystem.format.ps1xml file and ALL changes are clearly documented in a summary at the top of the file and then each change is clearly documented where the change is made making it very easy to see the differences.

My.FileSystem.format.ps1xml.zip

Thanks for your consideration!

Steps to reproduce

Expected behavior

Actual behavior

Environment data

> $PSVersionTable

[iSazonov]

You could use Update-FormatData cmdlet.

[JoeSalmeri]

@iSazonov

Yes, I understand that, it is what I have done since PS was originally released many years ago.

Did you read issue #7973 that I previously submitted about PS 6.1 missing the *.format.ps1xml files?

In that issue, I explained the changes I made and Steve (from MSFT) asked me to submit my changes as a new issue for inclusion in the default files.

This issue was created to do as I explained pretty clearly in my original message.

[KirkMunro]

@JoeSalmeri I appreciate you sharing your extensions, but I feel I should point out a few reasons why they shouldn't be applied to PowerShell as the default format that ships with the product:

1. The date format that PowerShell uses adheres to the international ISO8601 standard. Your proposed format is US-specific, and therefore only easier to read for someone in the United States. As someone not living in the US, I would vote strongly against the proposed date format.
2. The number format that you proposed, to add commas to file lengths, which may be nice in a UI, but I'm not sure I agree with it in PowerShell output. I much prefer showing the value without any commas, so that regardless of what computer I'm working on and what culture it uses, I see the data in the same format (not all cultures use commas as their thousands separator).

My recommendation would be to leave the existing date/time/length formats as is, and if you want special formatting in your environment, use Update-FormatData.

[JoeSalmeri]

@KirkMunro

Thanks very much for your feedback.

Did you read issue #7973 which was the original one that I reported?
If not, I would recommend reading it, however I will summarize.

Since the Windows PowerShell days, I have tweaked the FileSystem.Format.ps1xml file and then used Update-FormatData to apply my changes.

When each new version of the PS was released I would grab the new FileSystem.Format.ps1xml file and then merge in my changes.

This was done to ensure that any NEW changes that were made to PS would not be lost using my older version of the FileSystem.Format.ps1xml file while preserving my desired changes.

PS Core does NOT include the *.format.ps1xml files (although the documentation still says it does) therefore other than reviewing the PS source code I don't have a way to compare/find any changes easily that are made to the *.format.ps1xml files.

In issue #7973, Microsoft ASKED me to submit a new issue with the changes that I am making so that they could review for possible inclusion.

It would really save a LOT of time if people would have read the original issue first :-)

As for the number format, I can understand how a different culture might want a different format however, even the later DOS years had commas in the file sizes :-)

It would seem that the PS team could adjust the formatting I proposed to use the locale/region settings for the installed environment as a default which is cross platform and would address my desired settings as well as those located in other regions.

At a minimum, the *.format.ps1xml files should be included in the distribution as they previously were so that end users (like me) can customize them without have to dig into the PS source code files to determine if changes have been made.

That seems more than reasonable to me

[KirkMunro]

@JoeSalmeri I read your issue original issue, including @SteveL-MSFT's comment where he proposed you share your changes for consideration. I also read your source file changes. My points rest the same -- I made them because I don't believe the changes proposed should be included in PowerShell Core, for the reasons identified above.

I agree it's unfortunate that format ps1xml files are not in PowerShell Core, because they did serve as a good example and foundation for creating extensions. At this point, however, I would simply recommend that they provide adequate documentation with examples, but that would be in response to your original issue. For this issue, I would simply recommend you use Update-FormatData so that you can get the output that is meaningful to you (as you have been doing), and perhaps share it on a blog or in a module published on the PowerShell Gallery if you want to provide it as optional for others who want to opt-in to that experience as well.

[KirkMunro]

@JoeSalmeri I feel I should also make one point of clarification, because you seemed a little defensive in your last reply. In #7973 Microsoft proposed that you submit an issue with your changes so that they could be reviewed for possible inclusion. That review is happening right now. It is not exclusive to Microsoft. It includes the PowerShell community. This is the nature of open source. I didn't indicate anywhere that you shouldn't have made the proposal. I simply reviewed the proposal and shared why I don't think it is a good idea. I just wanted to call that out because I'm not knocking you at all for anything you did here -- you should always propose changes you feel would be beneficial to the core product. It's the right thing to do. In this case, however, I don't believe the proposed changes should be made.

[JoeSalmeri]

@KirkMunro

No need to defend your points, I am perfectly fine if they chose not to include my changes for the reasons you stated or for any other reasons which might arise that neither of us considered.

However, "including a few examples" is not a satisfactory solution to the original issue I reported because those examples might not include the areas of formatting which I am currently modifying or which I may modify in the future. I cannot predict which future formatting changes I might make any more than MSFT could predict which formatting changes they might make :-)

That's why I went through the effort of re-applying my changes every time a new version of PS came out because I did not want to inadvertently lose some new formatting or features that MSFT included because my starting point for my changes was based on an older version of the format files.

Here is a simple real world example that actually occurred at some point during of the PS releases.

The FileSystem.Format.ps1xml file changed the display of the "Mode" attribute to include a "l" at the end of the "Mode" attribute for a item which represented a "link".

If I did not have the full source to the "new" format files (as is the case now since they are not included), then I would not have been aware of that change since my files were based on the older version.

Since the full source for the format files was provided at that time, when I started with the new version of the files and then applied my changes I did not lose any of the new features or changes that MSFT made.

Over the years, there have been other changes to the files.

Hopefully my simple example explains why providing some examples of formatting would not be sufficient to address the original issue.

I would be perfectly fine doing what I have done for all these years, I just need the source files when new versions come out.

Hope that makes sense!

[mklement0]

If I did not have the full source to the "new" format files (as is the case now since they are not included)

There is no need for the *.ps1xml _files_, given that you can export the built-in definitions to *.ps1xml files on demand with Export-FormatData; e.g.:

Get-FormatData | Export-FormatData -LiteralPath all.ps1xml

For a specific type; e.g., [System.Timespan] (note that type accelerator names are _not_ supported):

Get-FormatData System.Timespan | Export-FormatData -LiteralPath timespan.ps1xml

That said, Get-FormatData is currently buggy: #4237

Also, Export-FormatData doesn't accept the output file name as a _positional_ argument: #6680

[SteveL-MSFT]

@KirkMunro is correct that my ask is to submit changes @JoeSalmeri makes that he finds useful to see if the broader community found them useful and if so, it would make sense to include in the product. Based on the discussion, thus far, it seems there is a concern that it is very en-US locale specific. Given that analysis, it doesn't make sense to change the default formatter. However, it may make sense to introduce a way to see dates/numbers in local culture, but need a proposal on how to handle this if there is sufficient user demand for it.

Regarding the ps1xml, the default formatters were converted to C# code and thus the ps1xml is no longer used. As @mklement0 pointed out, one can export the C# format code to a ps1xml file that one can customize so that seems like a fair solution.

[JoeSalmeri]

@SteveL-MSFT

Regarding the ps1xml you pointed out what @mklement0 said regarding exporting using Get-FormatData, however, you seemed to ignore the part where he said that Get-FormatData is current buggy...

I ran the "Get-FormatData | Export-FormatData -LiteralPath all.ps1xml" that @mklement0 provided and it is indeed quite buggy as it appears that NONE of the items in FileSystem.format.ps1xml are exposed via Get-FormatData's output therefore I'm not sure how you could say it is a fair solution when it doesn't provide ANY of the information that is currently being used for the output we are discussing.

It will also be much more time consuming (on my side) to go through that output when there are only a handful of places that I am currently tweaking.

If the bugs did not exist, then I could probably use the -TypeName parameter to get the ones that I am CURRENTLY modifying, however, that does not do much to help me in the future if new types are added to what is currently in FileSystem.format.ps1xml as I would not know what the new names were whereas when the FileSystem.format.ps1xml files existed it was trivial and took 10 minutes to integrate my changes into the new versions of the files.

I would suggest that it is only proper to remove the format files AFTER the replacement options actually work are are documented. It is no wonder that customers get frustrated when changes are made without through testing and updated documentation which reflects the changes.

[iSazonov]

The our fundamental policy is that formatting and error reporting is not a _public contract_. I hope and sure that it will be not changed.
My understanding that @JoeSalmeri want to implement a scenario where formatting is a public contract.
First question is - is it business or personal preference scenario? Do you know applications there formatting should/must be preserved?
Second question - what do we fix/enhance in Get/Update-FormatData cmdlets to address the requested scenario?

[SteveL-MSFT]

@JoeSalmeri I submitted https://github.com/PowerShell/PowerShell/pull/8063 which fixes the problem with Get-FormatData and validated you can export it to a ps1xml file. This should resolve your ability to customize formatting.

[JoeSalmeri]

@SteveL-MSFT

Wow, that was a quick turnaround!

A few questions for you....

1) Will the results returned match what is now being used internally so that if that internal formatting is
changed in the future then the results will reflect those changes? Do we have something in place to
ensure that Get-FormatData output is not missing formatting data as it was in this case?

2) Is there a targeted version for PowerShell Core as to when this will be released, such as 6.1.1?

After reviewing #8063 I am confused by one of your comments in that issue:

It "Should return nothing for format data requiring '-PowerShellVersion 5.1' and not provided" {
Get-FormatData System.IO.FileInfo | Should -BeNullOrEmpty

Why would "Get-FormatData System.IO.FileInfo" be null or empty?

I thought that this fix was to get it to return that information (as well as System.IO.DirectoryInfo) which it is now currently missing?

If your comment is correct that it would be null or empty, how exactly does that fix the issue of it not returning that data?

If your comment is correct that seems to imply that something else is being used for DirectoryInfo and FileInfo.

I assume I have misunderstood something in your comment, could you please clarify for me and answer those questions?

Thanks!

[SteveL-MSFT]

@JoeSalmeri yes, Get-FormatData returns the current state, not the default so you can rely on it. This would be in 6.2 (Preview.2 at the earliest). Regarding my comment, basically if a format requires enhancements made in 5.1 and 5.1 wasn't requested, then you wouldn't get that formatter. This is to allow Windows PowerShell v2, v3, v4, and v5 to work against a Windows PowerShell 5.1+ remoting endpoint. FileInfo and DirectoryInfo in 5.1+ has these enhancements. So without specifying -PowerShellVersion 5.1 (or higher), you get no format data (which results in default formatting).

The requirement of -PowerShellVersion 5.1 is an unfortunate requirement currently. Need someone to think through how to make this parameter compatible with remoting (backwards compatible), but not necessary when run locally.

[iSazonov]

Get-FormatData returns the current state, not the default so you can rely on it

Make sense to enhance the cmdlet to get default state?

[SteveL-MSFT]

@iSazonov if there's a need. However, you can work around that by starting a new PowerShell

pwsh -c get-formatdata ...

[JoeSalmeri]

@SteveL-MSFT

Thanks for the clarification and also rough timeline as to when I can expect to the see the change to Get-FormatData.

Regarding your clarification about it getting the current state and not the default state and your tip to get the default state by starting a new pwsh, I believe that would not actually work because launching the new pwsh would by default load the users profile which would normally be where the user used the Set-FormatData to change the layouts (at least that is how things seem to work today).

To get the default I believe the correct method would be to use

pwsh -NoProfile -Get-FormatData.....

Agree?

[SteveL-MSFT]

@JoeSalmeri yes, there is a risk that a user profile may make changes, so using -NoProfile is needed to ensure it's "clean"

The PR was merged, so you can try it out with tomorrow's nightly build (or just build master branch yourself).

[iSazonov]

@JoeSalmeri You could make additions for the scenario in PowerShell-Docs repo.

[JoeSalmeri]

@SteveL-MSFT

Thanks, I'd prefer to just download a nightly branch to test.

Do you have a link for the night branch builds?

I found the following but was not sure if it was the nightly builds you are referring too:

https://github.com/PowerShell/PowerShell
Then scroll down to the section which says
You can also download the PowerShell binary archives for Windows, macOS and Linux.
Then download
https://github.com/PowerShell/PowerShell/releases/download/v6.2.0-preview.1/PowerShell-6.2.0-preview.1-win-x64.zip

If that is not the nightly build can you please provide the URL?

Thanks!

[vexx32]

Hmm. Scrolling a little further down there lists the nightly/daily build release build status. ~However, there doesn't appear to be any published artifacts for Windows pipeline -- the Mac/Linux are on Azure CI and do appear to have published artifacts, but no artifacts are published on the linked Appveyor nightly/daily build page. :(~

EDIT: Found them!!

@JoeSalmeri in the above linked section, click the Appveyor build badge to be taken to the Appeveyor test results. Select the first test in the list to be taken here and display further information.

Then, select the Artifacts tab to be taken to a display of all published build artifacts.

Bit of an unintuitive navigation for those of us unfamiliar with Appveyor's layout, but anyhow. I believe the TestPackage.zip should contain the build itself. 😄

[JoeSalmeri]

@vexx32

Yeah that's intuitive :-)

Actually I had wandered down and clicked on the Appveyor build badge but once there nothing seemed like what I was looking for.

Followed your instructions and selected the first item and downloaded TestPackage.zip and unzipped however no pswsh.exe was in that zip. That item appears to be for something related to the PesterTests.

If you select the 2nd item (vs the 1st) and then click on Artifacts, that has the various install files and zip files that I was looking for.

From there I downloaded PowerShell-6.2.0-preview.848-win-x64.zip.

After unzipping that does indeed seem to be the what I was looking for and at first glance does appear to have the fix that @SteveL-MSFT and I have been discussing.

Thanks very much for chiming in! Although TestPackage.zip did not contain the build you helped me find the correct location.

FYI, I think the build was not done yet when you looked and when I started to look because, the build badge is now gone and replaced with "av-nightly-image".

Thanks again!