Powershell: Some bits in the file attributes are masked out

[sba923]

Files in the OneDrive sync folder with the "Files on Demand" feature make use of a number of specific attribute bits, such as:

0x00000200 | SparseFile
0x00000400 | ReparsePoint
0x00001000 | Offline
0x00080000 | Pinned
0x00100000 | Unpinned

(non-exhaustive list)

In PowerShell 6.1, some of the bits are masked out, so code that runs in Windows PowerShell 5.1 will not run correctly under PowerShell 6.1.

(this might be related to underlying .NET Core behavior, but the PS6 user… doesn't care ;-) )

Steps to reproduce

Run the following commands under PowerShell 6.1:

"'{0:x}' -f (gi 'C:\Users\steph\OneDrive\Always keep on this device.txt').Attributes
"'{0:x}' -f (gi 'C:\Users\steph\OneDrive\Free up space.txt').Attributes

Expected behavior

The output should be:

00401620

like in Windows PowerShell 5.1.

Actual behavior

Instead, PowerShell 6.1 outputs:

00400020

Environment data

Name                           Value
----                           -----
PSVersion                      6.1.0
PSEdition                      Core
GitCommitId                    6.1.0
OS                             Microsoft Windows 10.0.17763
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

[BrucePay]

It appears to be a .NET limitation. Calling the API directly returns the same result as PowerShell.

'{0:x}' -f ([system.io.file]::GetAttributes((resolve-path ~/onedrive/documents/mydoc.txt)))

[SteveL-MSFT]

@sba923 would you mind opening an issue in corefx repo?

[sba923]

Done, it's at https://github.com/dotnet/corefx/issues/33644

[SteveL-MSFT]

Reopening as it appears the Windows APIs want this to be an application decision

[SteveL-MSFT]

We probably need to call: https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/content/ntifs/nf-ntifs-rtlsetprocessplaceholdercompatibilitymode

[sba923]

I think so too. But this looks like underdocumented to me, I would have no idea as to how to use that API...

[sba923]

After all, this wasn't too hard ;-)

````
namespace TestFileAttributes
{
class Program
{

    [DllImport("ntdll.dll")]
    static extern sbyte RtlQueryProcessPlaceholderCompatibilityMode();
    [DllImport("ntdll.dll")]
    static extern sbyte RtlSetProcessPlaceholderCompatibilityMode(sbyte pcm);

    const sbyte PHCM_APPLICATION_DEFAULT = 0;
    const sbyte PHCM_DISGUISE_PLACEHOLDER = 1;
    const sbyte PHCM_EXPOSE_PLACEHOLDERS = 2;
    const sbyte PHCM_MAX = 2;
    const sbyte PHCM_ERROR_INVALID_PARAMETER = -1;
    const sbyte PHCM_ERROR_NO_TEB = -2;

    class PlaceholderCompatibilityMode
    {
        public static string ToString(sbyte pcm)
        {
            switch(pcm)
            {
                case 0: return "PHCM_APPLICATION_DEFAULT";
                case 1: return "PHCM_DISGUISE_PLACEHOLDER";
                case 2: return "PHCM_EXPOSE_PLACEHOLDERS";
                case -1: return "PHCM_ERROR_INVALID_PARAMETER";
                case -2: return "PHCM_ERROR_NO_TEB";
                default: return String.Format("(??? unsupported PlaceholderCompatibilityMode value: {0} ???)", pcm);
            }
        }
    }

    static void Main(string[] args)
    {
        var filepath = args[0];
        sbyte pcm = RtlQueryProcessPlaceholderCompatibilityMode();
        Console.WriteLine("Current placeholder compatibility mode: {0} ({1})", pcm, PlaceholderCompatibilityMode.ToString(pcm));
        var attr = System.IO.File.GetAttributes(filepath);
        Console.WriteLine("Attributes for '{0}': 0x{1:x}", filepath, attr);
        sbyte newpcm = PHCM_EXPOSE_PLACEHOLDERS;
        Console.WriteLine("Setting placeholder compatibility mode to: {0} ({1})", newpcm, PlaceholderCompatibilityMode.ToString(newpcm));
        RtlSetProcessPlaceholderCompatibilityMode(newpcm);
        attr = System.IO.File.GetAttributes(filepath);
        Console.WriteLine("Attributes for '{0}': 0x{1:x}", filepath, attr);
    }
}

}
````

Here's the output:

Current placeholder compatibility mode: 1 (PHCM_DISGUISE_PLACEHOLDER) Attributes for 'c:\users\steph\OneDrive\Free up space.txt': 0x00500020 Setting placeholder compatibility mode to: 2 (PHCM_EXPOSE_PLACEHOLDERS) Attributes for 'c:\users\steph\OneDrive\Free up space.txt': 0x00501620

[vexx32]

Would this fall under things that would need to have some of the code put in https://github.com/PowerShell/PowerShell-Native perhaps?

[SteveL-MSFT]

@vexx32 PowerShell-Native is for specific native code such as integrating with WinRM and some native Unix APIs. (Over time, PowerShell-Native should get smaller and smaller if we can just use .NET Core)

This code should be in this repo.

[sba923]

What are the next steps? I don't think the code change itself is that big (could even do it myself) but where should it end up?

[vexx32]

I guess you'd need to place the code that enables the flags to be returned somewhere in the Start() method for the FileSystemProvider, perhaps? It will need to be wrapped in #if !UNIX flags since it's a Windows-specific API or something along those lines.

That way the .NET Core built in methods for pulling attributes should then return the proper values, if I understand this correctly.

[SteveL-MSFT]

@sba923 are you able to work on a PR for this? I agree with @vexx32 that putting it in Start() makes sense

[sba923]

I probably can. Where can I find information about the process, testing methodology, coding rules etc.?

Next to that I need to determine how to:

• make that code Windows-only in the PowerShell source
• make that code conditional to running on Windows versions that do support those (poorly documented) APIs which seem to have evolved with the latest Redstone versions if I compare the SDK header contents

[vexx32]

make that code Windows-only in the PowerShell source

This is pretty straightforward, I think. If you poke about in the code one you'll see compiler directives like #if !UNIX/#endif for things that shouldn't be compiled on Unix platforms; I think using that should be sufficient. If you wrap both the necessary fields and the code that triggers the setting to enable the feature in those you should be good. More on those here: https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/preprocessor-directives/preprocessor-if

As for the latter, we'll need to figure out which Windows versions support the API, if there are any versions where it's safe to leave in where it will just no-op or something, and if there are versions we need to exclude it from.

[sba923]

I'm of course familiar with preprocessor directives ;-) but for some reason I didn't want to go fishing around in the code to _guess_ what mechanism(s) is (are) used to distinguish Windows and Unix builds... #if !UNIX / #endif it is...

I've inserted the p/Invoke declarations just below #region CmdletProvider members and the code at the end of the Start() method just before the return statement. This works like a charm.

Now I'm looking for documentation that describes how to deal in p/Invoke for "optionally available methods." IOW, what happens if you do [DllImport("foo.dll")] if the function isn't exported by foo.dll?

[vexx32]

... Hmm. /cc @indented-automation as he's really the only person I know with pretty extensive p/invoke experience. Any idea, Chris?

[SteveL-MSFT]

Take a look at https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/namespaces/FileSystemProvider.cs#L2450 it does a runtime check as that flag is only availble on Win10 and it appears this api is only available on Win10. Doing the DllImport itself shouldn't be an issue if you don't invoke that method.

[sba923]

Checking Environment.OSVersion.Version definitely works. What I was remembering reading is https://blogs.windows.com/buildingapps/2015/09/15/dynamically-detecting-features-with-api-contracts-10-by-10/#3HubB7h03hufW5g4.97 but after giving it some thought that's something for UWP apps, which PowerShell Core isn't.

For now I'm checking that we're running at least Windows 10 version 1803 ('cos that's what https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/content/ntifs/nf-ntifs-rtlsetprocessplaceholdercompatibilitymode mentions as prerequisite, but I'm wondering whether the code should just trust that documentation...).

I presume I ought to write a test case that will get added to the global PowerShell Core CI test suite?

And I need to get familiar with the PR creation / contribution process / workflow. Will start by reading the docs... ;-)

[sba923]

PR ready to merge, awaiting review by code owners @BrucePay and @anmenaga