Projectreunion: Improved file-access APIs for UWPs

[jonwis]

Proposal: Improved file-access APIs for UWP and AppContainer

Summary

Provide an updated version of StorageFile that fixes performance and usability issues in the current Windows.Storage.Storage* set of types. Support apps that think in terms of file paths & handles instead of StorageFile objects.

Rationale

The current Windows.Storage.Storage* types are designed to give programmatic access to regular files, shell namespace items, and other Windows entities backed by files. A single abstraction over all those spaces is interesting but comes with high costs - namely performance and impedence-mismatch with existing application code. Project Reunion should clearly explain why StorageFile is important and - where possible - provide lower-level access to the user's files.

• The BroadFileAccess capability is "scary" to customers but required even for limited UWP scenarios.
• Many libraries and helpers take file paths instead of objects (even instead of handles)
• Apps observe serious performance costs related to using StorageFile's built in functionality
• Low-level code can access the ...FromApp API variants, but higher-level code (JS, C#) interop for them is very limited.

Scope

| Capability | Priority |
| :---------- | :------- |
| App has direct access to any files the user has given it using low-level APIs | Must |
| Feature exposes an object-oriented view of ...FromApp platform APIs | Must |
| APIs in this feature are _synchronous_ after user permission is acquired | Must |
| Feature provides standard IStream/ISequentialStream APIs for these files | Should |
| App can convert between existing W.S.Storage* and types in this functionality | Should |
| UWPs can access Shell Namespace types directly with user approval | Should |
| APIs taking or returning StorageFile should return/accept these objects instead | Should |
| App can prompt the user for broader access without requiring an app restart | Could |
| App can access user data without user intervention | Won't |

Open Questions

Should UWPs be given access to the IShell* family of APIs?

As these APIs are not truly universal on all editions of Windows, if an app were to program against them directly the app would not work on editions like HoloLens or Xbox.

Should Reunion warn the user about apps using this capability?

Some platforms periodically remind users that "App X is using feature Y, are you OK with that?" How should Reunion make the user aware that a "low rights" application is using capabilities?

[MarkIngramUK]

For applications that are already compatible with sandboxing (i.e. only read from %APPDATA% or %PROGRAMDATA%, and only access files explicitly chosen by a user) I don't think a continuous reminder is warranted. This could perhaps be controlled via some sandbox capabilities options that are chosen at development time (similar to the macOS model).

[eklipse2k8]

A good test of this feature would be, I can create an SQLite database in the user's documents folder. I shouldn't need permission to do this if it's a new file. This is such a common scenario that the sandbox and the politics around it have never tried to solve for. An app should never need "broadFileAccess" but they should also make it so users can easily open documents and see their saved file right there.

[soumyamahunt]

A good test of this feature would be, I can create an SQLite database in the user's documents folder.

Why not create in AppData folder itself? A lot of users have complained apps creating random folders in their library folders like Documents, Music etc. introduces clutter to their workspaces and they have more trouble in finding their Work folders. Also, this kind of behavior from developers is what prompted MS to limit storage access in first place.

[benstevens48]

It’s really great that this is being looked into, and that it’s the first issue here. UWP file performance and permissions is the thing I’ve most wanted to see an improvement in since 2015! I imagine that any permissions changes will be difficult to deliver via NuGet, so hopefully something can happen on this as soon as possible in order to ensure we don’t have to wait too long before the users have the necessary OS updates.

Below I’ve put together a collection of scenarios related to file system performance and permissions that are problematic in UWP. These are things I have encountered while building my photo-viewing app. (Maybe it’s a bit selfish of me to concentrate on these problems, but hopefully they cover a wide range of use cases). If this post is too long, feel free to move it to a separate issue. Also, I am very happy to discuss any of these issues/ideas further. The scenarios themselves come first, followed by the problems with them in UWP and some possible solutions. I hope that this project will make time to think more creatively about how to deal with some of the file permissions issues. Some of my suggestions below are just initial ideas, so don’t judge them too harshly. It’s such an important subject that I think it’s worth spending a lot of time thinking about.

The scenarios

0. The user does not necessarily keep all their images in their Pictures library

I put this in as point 0 as it influences all the others. A key principle of my app is to allow users to organise their photos how they like and not to force them into any patterns. Therefore the app cannot assume that all the user’s images are in their Pictures library, and indeed it’s designed for viewing all images, even for web designers etc who are probably not keeping images in there. Also, I’m going to assume that broadFileSystem access is not used, because that is perhaps too unrestricted and also the current user experience for it is poor.

1. User selects a folder containing a large number of files (can include sub-folders) to view in the app

My app has quite a simple concept of allowing users to select files they want it to display. They can do this by selecting a folder, optionally including subfolders, or selecting individual files etc. They can make multiple selections. It is not uncommon for a folder selected to contain 10,000+ image files. The app needs to maintain a list of the selected files (including the order they were selected in etc), therefore it needs to be able to list files in folders, and the quicker and less resource-intensive this is, the better. Obviously the app needs to be able to access the contents of these files alter on, but at this stage a path and file name is all that’s needed.

2. The app needs to be able to watch for files added/removed/modified in the folders that the user has selected

My app has the option to auto-refresh the selected files to match the contents of the folders that the user selected to view. This can include modifications to files as well as files added/removed/renamed.

3. The app makes heavy use of StorageFile.GetBasicPropertiesAsync StorageItemContentProperties. RetrievePropertiesAsync to get properties such as DateTaken and Orientation

This should ideally be fast. Also, it expects to be able to read properties simultaneously.

4. The user can select files/folders via drag and drop, and expects to be able to edit these files

This is self-explanatory.

5. The app offers a ‘Duplicate file’ function

This is self-explanatory. There is a button in the app that allows the user to duplicate the file they are currently viewing in the app. (The file is one of those selected in scenario 1). This should place a copy of the file in the same directory as that file, with the name suffixed by ‘ Copy’ or similar. It should be a one-click operation as it’s a convenience function for the user (e.g. to make a copy to edit).

6. The app offers a button to load the other files in the same folder as the file that the user is currently viewing

The user may have selected an individual file to view in scenario 1. The app would like to show a button that would load the remaining files from the same folder in this case, ideally via a 1-click operation.

7. The app maintains a ‘quick access’ or ‘favourites’ list of files. Generally, clicking on a file in this list should also load the neighbors, as launching from File Explorer does.

Users who make heavy use of this list may add hundreds or even thousands of files to it.

8. The app would like to offer a feature that allows users to save their session to a file, e.g. to create a slideshow that they can bring up straight away

Such as slideshow may consist of thousands of images, which were originally selected in multiple different ways. The saved data file would store the file paths of these images, and when the data file is loaded it needs to be able to load the associated images.

9. The user can load files by double-clicking on a file in file explorer. This should load the neighboring files.

The user needs to be able to do the same things with these files that they can do with files selected another way. An important feature of this is that the neighboring files are ordered in the same way as in File Explorer. The neighboring files list should be resilient to problems with the file system indexer.

The challenges faced in a UWP app

Scenario 1

The challenge here is simply that listing the StorageItems in a StorageFolder is incredibly slow and resource intensive compared to using the standard .Net/Win32 API to list the file paths. A quick test showed that in .Net it takes about 0.003ms per file, whereas with UWP StorageItems it takes about 2ms per file, which is around 700 times slower. Also, looking at the Runtime Broker, UWP uses about 70kB memory per file, which is a huge cost when loading 10,000+ files, whereas .Net uses around 0.1kB per file (that’s a very rough estimate).

Scenario 2

UWP has the StorageFileQueryResult.ContentsChanged method for watching for file system changes. However, it has absolutely no configuration options for what sort of changes to watch for, so it will fire on any change, even something like last access time I think, and include subfolders. (I agree that if you have no configuration options that firing on any change is the best option). It also does not tell you what changed. Combined with the speed issues in scenario 2, this makes for a doubly bad experience. If you look at the .Net docs of file system watching, they give an example of watching for changes in “C:\”, which made me laugh. In UWP, assuming that you managed to get permission for this folder, in order to determine what changed, you would first have to enumerate all the contents, which would literally take several days, then on a change you would have to enumerate all the contents again, taking several days, and in fact several weeks if you also want to check the date modified to detect file modifications, and look at the difference. Also, at one stage, I tried adding a separate watching for each of 600 sub-folders that had been added to the app (in order to avoid having to query all the files for changes) and basically it ground to a halt – it seems to create a new thread for each watcher which is not necessarily very efficient.

Scenario 3

I added this because I’m not aware of a way to do this easily in .Net without writing interop code. Also, for any alternative to StorageFile in UWP, it needs to be possible to get these properties ideally without the overhead of creating a StorageFile. Also, there are some issues with these in UWP. Firstly, if you call any of the Get..PropertiesAsync methods apart from RetrievePropertiesAsync simultaneously on the same StorageFile instance, they will fail, even though they are read-only as far as I can see so have no reason to. Secondly, fetching System.Size and System.DateModified using RetrievePropertiesAsync after the first time does not return an up-to-date result. Thirdly, there can be issues with scenario 9, which I’ll mention later. Finally, using SorageFile.GetBasicPropertiesAsync to get the file modified data is incredibly slow compared to .Net. in .Net it takes about 0.05ms per file. In UWP, it takes about 5ms per file, which is 100 times slower, and also adds about an extra 60kB per file (if you keep the StorageFile), compared to about 0.05kB per file in .Net.

Scenario 4

Currently in UWP if a user drags and drop a file/folder from File Explorer into the app, the file/folder is read-only, so can’t be edited, unless the app has access to the file path via some other means.

Scenario 5

This duplicate button is not possible in UWP if the user opened the file individually or by double-clicking in File Explorer, unless the file is in the Pictures library (assuming the Pictures library capability). Worse, the app cannot even open the save file dialog to the current folder, since there is no way to set the starting folder of the save file dialog.

Scenario 6

This is a similar problem to scenario 5. If the user opened the file individually this is impossible, unless the file is in the Pictures library.

Scenario 7

The only general way to save access to files is via the FutureAccessList. However, this is beset with problems. It has quite a small limit of 1000 files which is easily exceeded. The developer has to write code to handle the case when this happens, which is hampered by the fact that any of the APIs surround the FutureAccessList are quite slow, typically taking several milliseconds, and hence such a function has to be carefully debounced so as not to impact performance of the app. Furthermore, the app has to maintain a list of the tokens it is using and ensure this is in sync with the list, which is awkward. It would be easier if the list could be accessed via file path, with a use count. This could be stored in the metadata currently but for some reason in order to obtain the metadata for a token you have to enumerate the whole list. I find that whenever I add a feature to my app that requires this list, it takes me a couple of days to get the logic right, and even then it’s full of annoying compromises. Compare this with a .Net app where you can simply forget about the whole thing and spend zero days on it.

Scenario 8

Due to the fact that the slideshow list would be stored in an external file which could then be deleted without the knowledge of the app, and also because this could exceed 1000 files, the FutureAccessList cannot be used for this purpose, so it is impossible unless the files are in the Pictures library.

Scenario 9

There are currently various issues with the neighboring files query, which I believe is related to the way it uses the file system indexer. Firstly, it is quite common for the file system indexer to stop working properly. On my old PC this happened after every major Windows update. In this case, the neighboring files query would contain no files or be missing some files. I found disabling and re-enabling file content indexing fixed the problem. Also, if you use RetrievePropertiesAsync to retrieve properties for files loaded in this way, you won’t be able to access some properties. Also, some properties like title will be limited to a certain number of characters (e.g. around 250 for System.Title). Another problem is that the allowed file extensions does not always include all media files .e.g. for some reason .webm video files are not included in the neighboring files list (as of 1909).

A few suggestions for solutions

Scenarios 1-3 are performance-related, and in the case of scenario 2 the APIs are simply incomplete for UWP and need to be completed. The performance issues should be covered by the suggestions in your proposal. In my opinion in this case it is important that all APIs involving StorageFile/Folder have versions which accept the lightweight alternative. This should include an alternative for RetrievePropertiesAsync. Also, once you have permission to access a file for the current app session, you should be able to access it via any means, unlike currently where StorageFile.GetFileFromPathAsync(myAlreadySelectedStorageFile.Path) will fail. (Memory usage should be taken into account if it becomes a problem, but hopefully it shouldn’t).

Scenario 4 is a simple matter of change the default behavior to read-write, and I think most people agree with this.

Scenario 9 could be covert by enabling the app to indicate that it doesn’t want the neighboring files query to use the file system indexer, or something like that. It could also be solved by just allowing access to the folder, as long as it was possible to get the current sort order. I know this is an increase in access rights, but it would also solve some other problems which is why I mention it. The issue of some media files being omitted could be solved by Windows having a list of such allowed extensions that could be updated independently of a feature update so we do not have to wait a year for such a simple change (assuming the list was then updated to add files such as .webm).

Scenarios 5-8 are more tricky. Basically all of these require more permissive file access or additional capabilities. I have sketched out a few ideas, some of which you might consider outrageous, but I think this is a very important area that needs exploring.

• Suggestion 1 – If the user double-clicks on a file in File Explorer which launches the app, allow the app full access to that folder. This would solve some of the issues with 5 and 7 and 9 (it is very common for the user to open files in this way). Alternatively, if technically possible, allow read-write access to that folder including creating new files but only for Pictures file extensions (if the app has the Pictures library capability).
• Suggestion 2 – A bit out on a limb. Add an ‘exendedPicturesLibrary’ (and videos etc) capability which would allow read-write access to all standard folders for the users, but would be limited to picture files. This would solve most of the issues for my particular app but perhaps is not so generally applicable. Also, the app may save some settings files etc in a non-image format, so maybe it should include other file types specific to the app – it could even be required that these extensions should be specially named (e.g. start with x- or something, to ensure the app does not gain access to other files).
• Suggestion 3 – Allow the app to prompt for access to files on a file-by file basis. The problem with broadFileSystemAccess, in my opinion, is that at the moment the app prompts for it, the user will assume the app is trying to access all their files, rather than just the ones they want it to, and that will put them off. It makes sense for the app for be able to tell the system ‘I want to access this list of file and folder paths’ and then the system can prompt the user (if the app does not already have access), and say ‘Do you want to give the app access to these files’. This dialog should ensure the list is displayed in an easy to read way that is not overwhelming even if the app prompts for multiple files. There could also be a checkbox ‘Always allow this app to access all my files’ which would be the same as broad file system access, except that now it would not seem like the app was unjustifiably tying to access all files. Ideally also the system should then automatically remember that the app was given access to those files so it does not have to prompt the next time. It might be difficult to maintain a large list however, so the system could simply maintain the most recent few thousand or something. This would address most of the file system issues and remove a lot of the need for the FutureAccessList, although it would still be necessary to add files to this to guarantee future access. It should not be necessary for the user to go into settings and restart the app to achieve any of this.
• Suggestion 4 – I’m not really sure this would work. In order to support scenarios like 8, I had this idea that once the app has access to a file, it can ask the system for a future access token, and the file access permission is then stored in the token itself. For example, this could be achieved with signing using the path of the storage item, the app’s identity and a secret system key, and maybe it could include a date stamp as well. However, it might be difficult to revoke access in this case. I guess it would be best used in conjunction with suggestion 3.
• Suggestion 5 – This is a really simple suggestion that would cost noting to implement but would make using broadFileSystemAccess where needed much less of a concern for me as a developer. In the app’s listing on the Store, if you use a capability like broadFileSystemAccess, then it says something like ‘This app can access all your files’. This makes it sound like by installing the app, it can access all your files. However, it requires an additional prompt. So why not say ‘This app can access all your files if you agree to an additional dialog’ or something like that? That would be must less off-putting.

Finally, my thoughts on periodic warnings for users about apps using capabilities - personally I find this kind of notification annoying, so I would be against it. Better to make it easy for a user to investigate this, without giving them an intrusive notification. Maybe it could go so far as something like the using your location icon in the task bar, although even that is potentially annoying if there are too many.

Thanks for your patience in reading that!

[eklipse2k8]

Why not create in AppData folder itself?

Sorry @soumyamahunt, but this is a limitation. What if the user wants to copy the file to a USB drive or put it in their OneDrive storage? I'm forced to build a file manager UI so they can "export" the document, when the file system provides all of this functionality already.

AppData is for caches and app metadata, but not user documents that they want to access, move around or do as they please with. AppData should be considered off limits to users mucking around with anyway.

[soumyamahunt]

Sorry @soumyamahunt, but this is a limitation. What if the user wants to copy the file to a USB drive or put it in their OneDrive storage? I'm forced to build a file manager UI so they can "export" the document, when the file system provides all of this functionality already.

AppData is for caches and app metadata, but not user documents that they want to access, move around or do as they please with. AppData should be considered off limits to users mucking around with anyway.

You don't have to build a file manager UI, user can go to the AppData folder of your application and copy it. If you are insistent in creating folder/file anywhere other than that you can ask user to create a folder with file picker and then add the folder/file in FutureAccessList. Automatically creating folder anywhere is huge no for me.

[MarkIvanDev]

These are the limitations I see in UWP file-access APIs besides the ones that are pointed out above

1. The broadFileSystemAccess capability limitations

• StorageLibraryAPIs. You still cannot get the user's libraries when calling StorageLibrary.GetLibraryAsync even though technically, Libraries are part of the file system that is available to the user. Since the docs are saying that the app should not declare the capabilities for accessing the libraries (documentsLibrary, musicLibrary, picturesLibrary, videosLibrary) if it already uses the broadFileSystemAccess capability, declaring broad file system access should also grant access to the user's libraries through the StorageLibrary APIs

• Removable Storage Access. Although accessing removable storage requires another capability (removableStorage), broadFileSystemAccess should help lessen the restrictions. As of now, the app still needs to declare the file type associations just to access those specific file types in the user's removable storage devices.

• UNC folders. As with the previous point, accessing UNC folders needs a combination of capabilities (privateNetworkClientServer, internetClientServer, enterpriseAuthentication) to access them. Combining broadFileSystemAccess should also remove the file type restriction.

2. Low-level ...FromApp API although helps with performance is still severely limited

• Copy, Delete, Replace, Move APIs does not expose a way to monitor progress. CopyFileEx, although available to UWP, still only accepts a file path to work. If only it has a variant that accepts a file handle, a workaround can be made. Or better yet, expose a FromApp api that can monitor progress.

---

To answer the Open Questions

1. Should UWPs be given access to the IShell* family of APIs?

Since these APIs are not universally available, I think UWP should not be given access to these. It would add an added complexity of checking whether the device that the app runs on supports these or not. IMHO, the Windows.Storage is already a major improvement over the Storage APIs available in previous implementations. It just needs to be updated to have more APIs that will better align with the design of UWP

2. Should Reunion warn the user about apps using this capability?

UWP apps already has a robust way of requesting capabilities (i.e. Microphone, Camera) from the user that does not require a restart. I think this design should be implemented when requesting access for ANY capability. As it stands, changing access to the File System requires an app restart that users can mistake for a crash and does not provide a good look for the apps that use these kind of capabilities.

[eklipse2k8]

You don't have to build a file manager UI, user can go to the AppData folder of your application and copy it. If you are insistent in creating folder/file anywhere other than that you can ask user to create a folder with file picker and then add the folder/file in FutureAccessList. Automatically creating folder anywhere is huge no for me.

The problem is, AppData is hidden and I don't believe it's even easy to traverse because the final document would be behind your package folder that could be anything. If anything, it's there, but it's very unfriendly to users. That's why we're writing our software right?

The scenario I see is this,

• App creates a temporary document in AppData until the user chooses to save it somewhere on their system. You shouldn't need any kind of broadFileSystem access to do that. If they have write permission in the location of their choice, and they tell the app to put the document there, then that's it!
• Now, SQLite likes to create a journaling file, and potentially a backup file incase of a bad save. The SQLite that's bundled with Windows winsqlite3.dll will want to write those files next to the actual database file.
• This is the same behavior of Microsoft Office apps. They all get to save backups right next to the actual document file.

This is not about letting an app write junk all over the place, this is about a document based application creating a good user experience.

[duke7553]

Hi. I'm the developer of Files (UWP).

Our primary pain points with the current experience of integrating the native Win32 file-access APIs into a UWP app were the things the Win32 File APIs lack (at least for UWP apps):
1.) A simpler alternative to ReadDirectoryChangesW()
2.) Fast replacement for WinRT GetThumbailAsync()
3.) Viewing hidden items and shortcuts

Also, if we're planning on keeping the file/folder pickers, they should really have a modern UI to look familiar in our WinUI apps.

[jonwis]

@duke7553 - thank you!

Simpler alternative to ReadDirectoryChangesW

Have you tried out StorageFolder.TryGetChangeTracker ?

Hidden items and shortcuts

Does FindFirstFileExFromApp help here?

[duke7553]

@jonwis My app, Files, previously included a purely WinRT file-access implementation. (Meaning, we used StorageFile and StorageFolder) It was highly optimized at that because we retrieved storage items in batches, and prefetched properties for partial storage items, yet users complained this approach was slower than they were familiar with. As of January, we now use the FindFirstFileExFromApp() to load the items themselves, but we do still rely on StorageFile/StorageFolder to return thumbnails and other properties for the items in a directory.

Regarding StorageFolder.TryGetChangeTracker, my understanding of it was that it works only when the user is accessing libraries, so something like ReadDirectoryChangesW was needed here. I implemented it in Files (UWP) with the last update, v0.10.1 .

[duke7553]

Sure, we can access items with the hidden attribute this way, but they are invisible to all of the WinRT file operation methods (i.e. Cut, Copy, Delete)

P.S. I will admit my case is kind of extreme since I built a file explorer. :)

[jonwis]

@duke7553 - There's a whole raft of ...FromApp APIs like CopyFileFromApp, DeleteFileFromApp, and more. I get not wanting to re-implement StorageFile yourself from these primitives, of course. :)

Hey @smaillet-ms - does StorageFolder's change tracker work on any arbitrary storagefolder, or only on libraries?

[michael-hawker]

There's also some weird restrictions on existing capabilities, like access to the Downloads folder only lets you create a sub-directory, see doc issue here. If you're trying to make an alternate browser, this is a weird experience for anyone using any other browser. Maybe there needs to be a AllDownloads capability that's slightly broader?

(Similar to @benstevens48 Scenario 8 above), I think there's a challenge in having any sort of project system at all (and loading these types of file sets from any other source). If a file references the absolute or relative path of another file (or files think '*' wildcard in a csproj); you can't open those references in a UWP today unless you ask the user to select a parent folder that has access to all references. It does seem like a hard challenge to solve though with figuring out an access pattern for detecting and allowing this without needing the full BroadFileAccess capability though... I mean the fallback in UWP is have a folder based project system; however, that doesn't track well with consuming any external formats to your app.

[kmgallahan]

A StorageFolder.MoveAsync method would be highly useful. The current recommendation is... 🤷‍♂️

There is not currently a "MoveAsync" or similar method. One simple implementation of moving a folder might be to get the desired folder, copy it to the desired location, and then delete the original folder.

[soumyamahunt]

@jonwis ...FromApp apis doesn't work reliably either. I tried to implement GetFileAttributesExFromApp to check whether a file is read only when user opens file via picker or file activation, but this doesn't work when app opens the file for the first time (subsequent query works).

[shresthasource]

@duke7553 - There's a whole raft of ...FromApp APIs like CopyFileFromApp, DeleteFileFromApp, and more. I get not wanting to re-implement StorageFile yourself from these primitives, of course. :)

Hey @smaillet-ms - does StorageFolder's change tracker work on any arbitrary storagefolder, or only on libraries?

From what I remember, it is supposed to work on any folder. However, my last attempt to implement it failed. The issue I faced was that it did not trigger the event right after the changes. I remember reading something along the lines of ...it will wait till something else (don't remember what exactly) before the event is triggered.

[shresthasource]

P.S. I will admit my case is kind of extreme since I built a file explorer. :)

Considering WIndows Store has a dedicated subsection titled "File Managers", I would argue, this is not an uncommon scenario.

[ptorr-msft]

There's also some weird restrictions on existing capabilities, like access to the Downloads folder only lets you create a sub-directory, see doc issue here.

This is a missing feature (downloads folder capability) -- please add another issue for it. Thanks!

[mcosmin222]

As a developer of media applications, I find the existing file system access capabilities for UWP are horrendous, and are probably the biggest impediment on the platform not actually taking off at all. Any kind of serious application will require decent levels of file system access, and decent performance for those APIs. The broad file system access capability does solve the problem to some extent, but this does not address the issue of performance.

Not only are the search / access / property APIs stupidly slow, but they are complimented by equally slow read & write stream APIs. It is unacceptable to wait for several seconds to open and parse 25 files, just because of some unknown limitation imposed at API design time. What is the point in having those APIs as async if the user has to wait ridiculous amounts of time just to perform basic operations? Better have the UI freeze for 1 second than wait 10 seconds staring at a loading bar.

An additional problem is the assumption that every use will put their music / video / pictures / documents in their respective libraries. Truth to be told, I have never seen a single user who actually follows the library pattern completely. Not a single one. Broad file system access should in no way be a restricted capability: it should be top and front. Simply restrict access to obvious danger places like C>Windows and other system folders, and you're golden. Any app that passes through the store should be considered safe, i don't see the point in adding additional security impediments.

These APIs need to go back to the drawing board and be designed for speed and scalability. If Storage.* APIs get deleted tomorrow, I will go out and dance on the recycle bin. They are the biggest reason no productivity app is currently following the UWP model.

[jtorjo]

As a developer of media applications, I find the existing file system access capabilities for UWP are horrendous, and are probably the biggest impediment on the platform not actually taking off at all.

This is definitely the biggest impediment. The whole "sandboxed" concept is a half baked idea of someone that probaly listened to some buzzwords at some point.

But as a developer that reimplemented pretty much all of his app in UWP, there's sooo many issues that will bring you to the "curse master" level.

An additional problem is the assumption that every use will put their music / video / pictures / documents in their respective libraries.

I've been saying the same thing for months. Having said all that, while I do hope this will be fixed, I'm almost 100% sure this will be dragged for years before MS will do anything about it (if they will do something ever). The whole "sandbox" is soooo broken, that no one at MS seeems to want to work on it. Just look at compile times - we've been promised they would be improved for close to a year. Promised again they would start working on compile times 2 months ago, but nothing to show for. But nothing is happening - just promises and delays.

[soumyamahunt]

Also, adding System.IO.File and other System.IO APIs support to UWP should be considered. While back System.IO.MemoryMappedFiles (may be System.IO.Pipes too) support was added by using ...FromApp methods explicitly for UWP, similar approach can be taken for other APIs. This will help dotnet developers bringing their existing projects to UWP without making any significant effort in learning new APIs.

[ZodmanPerth]

Hi folks. I'm the lead developer for the Drawboard PDF UWP app. Our application contains a file explorer that allows the user to pick a root folder using the filepicker and navigate up and down folders within that root. We surface a filtered view of the filesystem; only folders and (mainly) PDF files, for selection for loading.

I'm currently rewriting this section of our app and have discovered the pain of UWP filesystem APIs. I've tried various methods to improve the current experience, and have explored a number of approaches to the problem. At the end of the day, the only stable method is to continue to use GetItemsAsync which is the slowest file access I've ever seen (so I agree with others about performance, I average ~4 seconds for a folder of 1000 items, ~45 seconds for 10000 items).

The UWP APIs are terribly complicated and unfriendly. After spending a lot of time with them and believe I understand how they are intended to be used, I have still failed to get the features promising better performance to work. I don't believe they actually work at all. Further, I don't believe the exposing the file indexer is worth the maintenance cost for Microsoft; it's probably better the Reunion API monitor and decide whether or not it should use the indexer or not as an internal implementation detail.

In the end I've had to create my own folder wrapper class that performs two functions:

• Enumerate and cache the (filtered) contents of a folder the UWP app has access to,
• Monitor the Changed event for changes to the folder. This kicks off a background process to re-gather all (filtered) items in the folder using GetItemsAsync, then compare them to the cached list of items, to determine what the changes are. I surface the results through my own event using a lightweight item class. I use locks to ensure I only queue one of these processes at a time, no matter how many times the Changed fires.

This allows me to wrap up all the logic I need to process folders at the snail's pace the API provides, while keeping the UI somewhat responsive.

ASIDE: I'm working on a standalone prototype, which I am planning to share publicly when I've finished my investigation. Please let me know it would be helpful for this discussion and I'll post it here when it's done.

I've tried using FindFirstFileExFromApp which is blindingly fast in comparison, and more than acceptable for my needs. However it fails with Win32Exceptions on some folders (C:\ for example), meaning it isn't robust enough to rely on. I have abandoned it as an option.

What I expect from from a v1 Reunion API for filesystem access is:

• The ability to get the id+name+type of files/folders within a specified folder the UWP app has rights to, given filter options. I don't care if it's an enumerator, collection, or stream of simple objects, I just want it to be very fast. It should be asynchronous.
• A callback/event to notify me when the contents of a folder has changed, given filter options. The args should include the id+name+type of files/folders that have been added/renamed/removed, and grouped as such. It could also provide some id to pass to an async function to get results and allow the callback/event to complete.
• A function (or functions) to call with the id of a file/folder to obtain more details. This could be file properties, image data etc.

I feel that would cover the needs of many UWP apps; perhaps many non-UWP apps as well.

Having spent significant time analysing the UWP tools for working with the containerised filesystem model (StorageItem, FutureAccessList, AppData folders etc) I have to say I think the model has a lot of merit. The core issue seems to be that engineers have trouble transitioning from a culture where they can access anything in the filesystem to the containerised model. It's not an intuitive leap, and despite numerous articles and documentation I think Microsoft have failed to onboard engineers into right mindset to work with the containerised model.

I don't have an opinion on BroadFileAccess as at present; our app simply hasn't had the need for it.

I agree with what others have said about the upper limit of the FutureAccessList being too small. It probably made sense when UWP was in a much smaller container and was only capable of producing simple apps. If we want to allow UWP the capability to create serious tools then performant filesystem access is key, whether it's using containerised permission or broad access.

With WinUI 3 on the verge of being delivered it may be that engineers will gravitate towards Win32/Winforms/WPF apps using WinUI 3, and UWP would remain only able to successfully compete with a subset of apps.

I feel resolving filesystem access with a Reunion API will mostly flatten the playfield of what can be achieved on UWP and give it the competitive support it needs and deserves.

[mcosmin222]

Having spent significant time analysing the UWP tools for working with the containerised filesystem model (StorageItem, FutureAccessList, AppData folders etc) I have to say I think the model has a lot of merit. The core issue seems to be that engineers have trouble transitioning from a culture where they can access anything in the filesystem to the containerised model. It's not an intuitive leap, and despite numerous articles and documentation I think Microsoft have failed to onboard engineers into right mindset to work with the containerised model.

The container itself is fine. The fact that it takes 100+ms to open a file on a SSD is not (while arguably using native code, when a managed application can achieve 100 times better performance). Other OSes managed to get containers and file access at a decent speed, yet somehow UWP failed to do that.

The StorageFile idea has a lot of merit, especially since it can also abstract away a lot of problems (like files on network drives or streamed files), but this has come at the expense of performance. Microsoft needs to address the performance problem, otherwise nobody will bother with the container.

With WinUI 3 on the verge of being delivered it may be that engineers will gravitate towards Win32/Winforms/WPF apps using WinUI 3, and UWP would remain only able to successfully compete with a subset of apps.

I personally plan to migrate my UWP app to a desktop app + winUI just to get decent file performance and implement some features which are simply not feasible with the current performance of the Storage API. I can continue using the Window.Storage API for most of the application, but performance critical sections need better APIs

[ZodmanPerth]

I've completed my prototype and have added it to a public repository at uwpfileexplorer. It serves as a case study into what UWP engineers have to do to achieve performant file access in the context of a file explorer.

The performance is good enough for an in-app file explorer, but not good enough for filesystem-crawling file processing utilities. The code includes a PowerShell script to create large folders that can be used to highlight performance issues with the API. In particular, note how long it takes to refresh a large folder when changes are made to the folder outside of the app.

It also highlights the inadequacies of the UWP filesystem APIs, and how much work needs to be done to wrap them into something useful.

[shresthasource]

To my experience, UWP apps also do not get access to various items such as URL shortcuts or App shortcuts, and hidden files/folders via StorageSpace API. Access to such items would help.

[DrusTheAxe]

@shresthasource > UWP apps also do not get access to various items such as URL shortcuts or App shortcuts

What sort of access are you interested in?

What

• create shortcuts
• update shortcuts
• delete shortcuts
• other?

Who

• shortcuts in user's profile
• shortcuts for all users
• other?

Where

• shortcuts on Start Menu
• shortcuts on desktop
• shortcuts on tray
• other?

There's gaps between shortcuts and tiles. Some of these are old features that may not make much sense these days (Run as Windows95 etc compatibility options?) and some are functional e.g. Secondary Tiles have no RunAsAdministrator property. Do you have specific gaps you'd find helpful?

Is it the functional behavior you're interested in or the physical shortcut itself i.e. a .LNK file?

There's also some gaps between 'primary' and secondary tiles. The latter are created/manipulated via runtime APIs, the former are automagically created when a package containing <Application>s is registered for a user and deleted when the package is removed, an <Application> only gets 1 primary tile e.g. you can't create a tile to launch an app and another tile with uap10:Parameters="--safemode" etc to create a similar tile with different arguments.

Are there particular gaps you're interested in?

On the primary/secondary tile front, would it be helpful if you could specify tiles in appxmanifest.xml e.g.

<Application Id="App" Executeable="catvideos.exe" ...>
    ...
   <Extensions>
        <Extension Category="windows.tile" uap10:Parameters="--type=kittens" uap10:CurrentDirectoryPath="$(knownfolders.videolibrary)\Cats">
            <Tile Logo="kitty.png">

to produce 2 tiles. The 'primary' today for the <Application> runs "...pkgdir...catvideos.exe" with currentdirectory=pkgdir, and a 2nd tile that runs "...pkgdir...\catvideos.exce --type=kittens" with currentdirectory=c:\users\username\videos\Cats

[shresthasource]

This should clarify most of the answers:

Use Case:
A user browses to the Desktop Folder using a UWP file manager and do not see the Application shortcuts that launches Applications such as Edge Browser. A URL shortcut is created from Context Menu -> New -> Shortcut and puts a URL or other paths.

Issue:
A user does not see such shortcuts in a folder and might end up deleing the folder thinking it's empty. But some URL shortcuts or other such items might have useful/important contents. A user blames developer for a file manager that did not show items that were there.

What sort of access are you interested in?

What

• create shortcuts
• update shortcuts
• delete shortcuts
• other?

All are relevant/useful. I tend to think the StorageSpace API does offer something to create a link/shortcut to a different folder (just like copy/cut/paste, there was something along the line of link, haven't tried it though). At very least such links/shortcuts should be visible. Best case scenario, they should have similar methods as other storageitems. Also, I am not sure if storage item type of neither folder nor file is actually implemented or usable for anything as of now.

Who

• shortcuts in user's profile
• shortcuts for all users
• other?

Again, I am looking mostly at from a location where shortcuts are rather than build in shortcuts to something. However, another issue I had is accessing various folders via StorageSpace. As of now, we can access Music, Videos, Documents, and Pictures. However, places such as Desktop and Downloads are not accessible from the API. So, best a developer can do is to try to do some magic with user name and generate a default path (e.g. C:\Users\username). But, such workaround wouldn't work for all the cases.

Where

• shortcuts on Start Menu
• shortcuts on desktop
• shortcuts on tray
• other?

Shortcut items wherever they are located (in StorageSpace). Desktop definitely. StartMenu maybe. Tray, probably not important unless tray shortcuts are also stored somewhere. Others -> Wherever user decides to create shortcuts to files/folders/applications.

There's gaps between shortcuts and tiles. Some of these are old features that may not make much sense these days (Run as Windows95 etc compatibility options?) and some are functional e.g. Secondary Tiles have no RunAsAdministrator property. Do you have specific gaps you'd find helpful?

Probably not much relevant answer in my context here.

Is it the functional behavior you're interested in or the physical shortcut itself i.e. a .LNK file?

Not entirely sure about the question, but I tend to think I am talking about physical shortcuts.

[benstevens48]

@DrusTheAxe I think at a minimum the physical shortcut itself (.lnk file) should be listed when listing the items in its parent folder. I suppose this could be optional but I don't see a reason not to include it. Maybe there could be an option to also resolve shortcuts when listing files in a folder, but this wouldn't work if just file names were returned and I wouldn't want it to impact too much on performance, so I don't think this is essential in a low-level API. There also needs to be a way to get the target file path of the shortcut. Should permission to access the target file automatically be granted? I'm not sure, but it would be difficult to do anything useful if not.

[Qws]

People don't want apps to dump files on random system locations behind the scene.

Only Devs think it's nice to dump their files wherever they want, it's a selfish use case and cause horrible Windows user experience (maybe not immediately, but it will in the long run).

Many people find Windows a confusing system, compared to iOS and even Android. Downloading and installing legacy Win32 app is even a more scarier and confusing process, but on iOS and even Android it's a fun process.

The reason why UWP is considered a fail? Let's just look how you get UWP apps installed on the PC in the first place.

The Windows Store itself a horrible app, look at the Windows Store review, people calling out how terrible the app is. The Windows Store is slow, freezing, horribly designed has boring UI for years. The whole Store needs to be revamped. It feels like it's a sluggish Surface advertisement board + some games.

Please don't blame UWP when the place where you get UWP apps is barely functional.

Another problem is, building an UWP app is still confusing, you don't need to know XAML and C#. Let people know how to make UWP apps with other tech stack, and further broaden the ways to build UWP apps.

UWP has limitation, sure, some of them are problematic, sure. But the file access is not one of them, it's one of the better limitations. Once you break this limitation, it will be abused... then you get back to the old Win32 legacy chaotic software and worst case literally spyware/malware. You just end up walking circles.

Keep UWP tightly locked and safe for the users to use... broaden the ways to build UWP apps, and important one, fix the store.

[jtorjo]

UWP has limitation, sure, some of them are problematic, sure. But the file access is not one of them, it's one of the better limitations. Once you break this limitation, it will be abused... then you get back to the old Win32 legacy chaotic software and worst case literally spyware/malware. You just end up walking circles.

Bro, did you just try to find a place to vent? You clearly have no idea what it's been talked about here. You clearly haven't used UWP storage file/folder API, so please do some tests before you open your mouth.

[Qws]

Bro, did you just try to find a place to vent? You clearly have no idea what it's been talked about here. You clearly haven't used UWP storage file/folder API, so please do some tests before you open your mouth.

What do I need to know about Windows.Storage? You should be more detailed instead of encouraging people to be quiet with just baseless assumptions.

I'm responding on the demand (by few devs) where they want to store Sqlite database (and other app related files) in user's PERSONAL documents or other places where user will never know of, doesn't know what to do with it and most importantly doesn't get properly deleted.

Seriously, no normal user wants to have an Sqlite database in their personal folders. You're literally just spreading garbage on their harddisk if you do that. It's unwanted. Like I said, just selfish dev usecases, which leads to worse user experience for Windows.

If I delete an app, everything related to the app and has no purpose for the user should be deleted, sadly, lots of time that just doesn't get happened with legacy Win32 software and its chaotic nature.

That's why all the app files should be stored in the ApplicationData.Current.LocalFolder, and if the user dislikes the app, they just uninstall it, and nothing of the app should be left on the PC, unless the user explicitly allows it. But that's how simple it should work.

And yes, I have used Windows.Storage apis like FileIO, StorageFolder, CreateFileAsync, etc... don't even know why that's even relevant to the discussion, but oke.

[duke7553]

@Qws That's a very legitimate concern. Luckily, low-trust UWP apps require a capability to access these locations at all. Users can already disable access to the Documents, Downloads, etc in Windows Settings under privacy.

Remember, we all want these improved APIs to provide a quality experience for users, so let's remain civil when stating our perspectives.

[jtorjo]

What do I need to know about Windows.Storage? You should be more detailed instead of encouraging people to be quiet with just baseless assumptions.

Well, a lot - if you're to discuss stuff here - please take a look at the title of this issue, for starters.

I'm certainly not discouraging people from venting about UWP - I've done it quite a bit myself. But this is about File-access APIs, and there are quite a lot of people that want improvements on their speed, myself included.

[shresthasource]

People don't want apps to dump files on random system locations behind the scene.
Totally agreed. I do not want apps to create files in random places, and I myself do not like seeing apps creating files everywhere.

In my response specifically, I am considering a file manager scenario. If your app is a file manager, you cannot say do not create a file here and there or do not show this and that file (well, okay, system directories should be read only type, that's fine).

[shresthasource]

If I delete an app, everything related to the app and has no purpose for the user should be deleted, sadly, lots of time that just doesn't get happened with legacy Win32 software and its chaotic nature.

Would you want to have all your MS word documents deleted because you uninstalled MS Office Package? all PDFs because you uninstalled PDF reader? Images if you uninstalled a specific picture viewer?

So, while your concern in legit, there are scenarios where app should be allowed to read/write/store files is different locations - specifically with user's permission/knowledge of some sort.

[shresthasource]

And yes, I have used Windows.Storage apis like FileIO, StorageFolder, CreateFileAsync, etc... don't even know why that's even relevant to the discussion, but oke.

The whole discussion is titled "Proposal: Improved file-access APIs for UWP and AppContainer." And StorageSpace API is, at very least, one of the main API related to file access. So, not sure why you would think the discussion about StorageFolder.CreateFileAsync, etc. is irrelevant.

[mcosmin222]

@Qws,

The main problem here is the read-only APIs. Most applications need to be able to read the file system in order to do useful things. File manager, photo managers and editors, video and media players, any productively app for that matter needs file system access to work properly. Just image visual studio using Windows.Storage api to do file access. It would essentially kill windows development, that's how bad UWP file access performance is.

You can't compare iOS and Android to windows. Windows produces things that android and ios consume. If your computer use case is facebook, you shouldn't be here.

[MarkIngramUK]

You can't compare iOS and Android to windows. Windows produces things that android and ios consume. If your computer use case is facebook, you shouldn't be here.

I'm not sure why iOS is being thrown in here, but they have a secure sandbox, and sensible APIs (any file the user chooses can be opened by any valid API, e.g. fopen). And it's fast.

[michael-hawker]

I think there's been two points that have been getting mixed here.

One is the fact that the current UWP APIs for file access or slow compared to Win32 equivalents. This just makes them unpleasant to use as a developer and effects end-user perception as well.

Second, is there are legitimate cases where you don't want to be in a sandbox and be able to read a general file that's referenced in data you have from a file the user has already permitted you access to (like a link to an image in some other folder that's referenced in the current file).

I know I find the sandbox a great environment due to its protection on user data; however, it's performance needs to match expectations, and there do needs to be a gradient of permissions available for different types of scenarios.

[ZodmanPerth]

I agree.

In my case at least, I do not mind the way UWP file access is restricted (though helping developers adopt the mindset could use better support from Microsoft).

My issues with the UWP APIs are:

• Performance is too slow if all you want are the name and type of items in a folder, and some means of identifying them for more work (an id/token)
• The current API provides a number of additional ways beyond basic functionality in an attempt to provide better performance. These are complicated to configure and clutter the API. They also don't appear to work.

I'll repeat my expectations (from my original post) to reiterate how ProjectReunion can help address this aspect of the problem:

What I expect from from a v1 Reunion API for filesystem access is:

• The ability to get the id+name+type of files/folders within a specified folder the UWP app has rights to, given filter options. I don't care if it's an enumerator, collection, or stream of simple objects, I just want it to be very fast. It should be asynchronous.
• A callback/event to notify me when the contents of a folder has changed, given filter options. The args should include the id+name+type of files/folders that have been added/renamed/removed, and grouped as such. It could also provide some id to pass to an async function to get results and allow the callback/event to complete.
• A function (or functions) to call with the id of a file/folder to obtain more details. This could be file properties, image data etc.

We need simple, fast access for simple use cases. We know this is possible using ...FromApp functions, but there are holes needing to be addressed. ProjectReunion has the opportunity to manage all this and expose a fast-access facade/API that will solve for these use cases.

[ZodmanPerth]

I've posted a 2 minute video to YouTube showing the speed at which the UWP API performs for large folders.
https://youtu.be/40Nmy3Oon78

There are 3 stages in the video:

• Shows how fast the API can enumerate 1000 items
• Demonstrates the impact of the poor performance when keeping content live by monitoring for changes (on a medium size folder)
• Shows how fast the API can enumerate 10000 items

The app returns results in chunks of 50 to keep the UI usable. If you mark the time between selecting the folder in the app (or modifying the folder in Windows Explorer) and the time when the count finishes, almost the entire time between marks is the speed of the StorageItemQueryResult.GetItemsAsync().

Note that the poor performance demonstrated is entirely from the UWP API itself. Adding items to the UI is very fast and takes an insignificant amount of time.

Observe how the time for a full scan goes up by a linear factor as the number of items increases.

Also observe that to keep the UI in sync with the actual folder, you have to complete a full re-scan of the folder because the API doesn't report changes. Coupled with the performance of the scan, this compounds issues with performance for the app.

I hope this shows the core issue with performance of the UWP API, why it isn't good enough, and why I hope ProjectReunion can resolve the issue gracefully for us.

[mcosmin222]

@ZodmanPerth
That link doesn't work. Says video is private.

[ZodmanPerth]

I double-checked; it's definitely marked as public. Maybe its a timing thing?

[mcosmin222]

Your video shows essentially the best case scenario. If you want to get 10000 individual files by path, things get far worse, especially when certain files are missing.

[ZodmanPerth]

Yes that was the point. This is as fast as things can get, and it's still very slow.

[ptorr-msft]

Do you believe Windows needs to add net-new APIs for the features you're describing, or do existing APIs (Win32 or .NET or C++/STL etc.) cover your use-cases, assuming you could use them from a UWP process?

[eklipse2k8]

@ptorr-msft If i could create a sqlite3 database where ever the user wants, for example something that works with Win32 CreateFile2 API (or c++ fstream, or c fopen) without the need to use StorageFile APIs is all I would need.

Right now the limitation with CreateFile2 only able to create files inside the appdata folder that then gets deleted on app uninstall makes it pretty frustrating to use.

[mcosmin222]

Hello @ptorr-msft

The way I see it, UWP should be able to use any win32/.net/C++ API in the folders & files it already has access to, and this should also work with the broadFileSystemAccess permission.

in a v2 you could have different permissions for read-only and read-write access broad filesystem access. The read-only should be enough for most scenarios.

[benstevens48]

@ptorr-msft - I don't think we need any new low-level APIs for the time being. Just being able to use the existing APIs from a UWP process wouldn't really be sufficient however - we need Project Reunion to write some wrappers for us and help integrate them with the other UWP APIs, otherwise the effort required is too much for the developer. (The .Net APIs are generally quite user-friendly, but we would still need to integrate with other UWP APIs that use IStorageItem like clipboard etc, and also .Net is missing the equivalent of RetrievePropertiesAsync I think).

Also, in terms of the permissions, as I described in my earlier long post, I think we need some adjustments or more of a permissions gradient with additional capabilities than the current situation.

[ptorr-msft]

Thanks @mcosmin222 and @eklipse2k8. That helps with our design process. As a work-around in the meantime, did you see this post for SQLite in particular?

[ptorr-msft]

@benstevens48 thanks also for your input (posts overlapped).

we would still need to integrate with other UWP APIs that use IStorageItem like clipboard etc

If you could just use the Win32 / .NET clipboard APIs, would you still want to use the WinRT APIs? Or what if the WinRT APIs supported file handles? Any preference?

Having a better permissions story is a bigger issue that will take quite some time to rationalize across AppContainer and Full Trust scenarios.

[eklipse2k8]

Thanks @mcosmin222 and @eklipse2k8. That helps with our design process. As a work-around in the meantime, did you see this post for SQLite in particular?

Ohhhh, that's such a hidden feature, but thanks! This strategy will work perfectly for me.

[ZodmanPerth]

Hi @ptorr-msft. I agree with @mcosmin222 and @eklipse2k8, that Win32 APIs would be fine to use from UWP within the folders the app has access to, and that UWP would be served best by exposing these in a friendlier manner. This should solve for fast access to read and display folder contents, and monitor for changes.

The existing UWP API for filesystem access should stay. The models required for consuming a token based system can (and do) look very different to modelse consuming the Win32 APIs. When done well this model is quite neat and shouldn't be written off.

Please note that there are issues with the ...FromApp functions in UWP still. I talk about it in my original post in this thread (which is now behind a fold and hard to find); I make mention to one such example that meant I had to abandon it because they aren't robust enough:

I've tried using FindFirstFileExFromApp which is blindingly fast in comparison, and more than acceptable for my needs. However it fails with Win32Exceptions on some folders (C:\ for example), meaning it isn't robust enough to rely on. I have abandoned it as an option.

So fast access to general properties, changed content from folder Changed events, and access tokens from Win32 (like) APIs, plus the existing UWP API, would work for my use case.

[ptorr-msft]

The ...FromApp APIs are not a complete solution, but FWIW the issue with C:\ is a bug that has been addressed internally but I don't know when it will be released, sorry.

[mcosmin222]

@ptorr-msft

winRT APIs should stay, they do have some advantages, namely being able to abstract all sorts of file types, like files on network drives or files shared by other applications. The way I see it, win32 API and winRT API are complementary, I would still continue to use winRT API for most of my app, and only use .net (since it has a C# code base) in places where performance is crucial.

The ideal way these should cooperate is that if I have access to a path from one side, it should also work with the APIs on the other side. i.e if a user picks a bunch of files with a file picker, then I should be able to open those files with either Windows.Storage.StorageFile or System.IO.File. It should be a seamless transition between the 2, so the best API can be used depending on the circumstances.

Ideally, if you could speed up the winRT APIs to match those of win32, then the whole conversation wouldn't be relevant anymore ^^. But I suppose that's not really possible.

[benstevens48]

@ptorr-msft - I'll do my best to answer the questions, but my knowledge of how low-level file IO works (especially HANDLEs) is very limited, so you might have to enlighten me.

First of all, I'm using C# and I wouldn't want to have to do anything like dllimport, but using WinRT projections is of course fine (as long as the performance isn't too bad). Also, I'm limited to .net standard 2.0 compiled with .Net Native for the foreseeable future.

In terms of using existing .Net APIs for things like clipboard and drag and drop etc, as far as I'm aware these APIs are specific to either WPF or WinForms, and it would be weird/confusing to use them from a UWP app, unless they were refactored into a platform extension package or something. Also, possibly there are some APIs in UWP using IStorageItem that are not present in .Net/Win32 (although I'm not sure off the top of my head), so if you got rid of those, you'd have to write replacements.

I like the idea suggested in the original post, i.e.

APIs taking or returning StorageFile should return/accept these objects instead

where 'these objects' means some sort of file identifier (e.g. path), that we have obtained previously (e.g. using Win32 or .Net file APIs if we had access to them). You mentioned this being a file handle, but I am unsure exactly what that means. Since some of these APIs are async and accept lists of files, wouldn't that mean having to open file handles for a long time, which may prevent simultaneous access to the file? Also, how would the file handles be closed in .Net? One thing it would be good to avoid is having a very expensive operation to convert between the lightweight file identifier we have already retrieved, and the argument accepted by these APIs. An expensive operation may require rearchitecting of apps because it would move the slow part of the process from one place to another (although hopefully there would be some filtering of the files that would happen in between so it would still be an improved situation).

Here are a couple of special cases that could be difficult to handle that are very important to my app.

Neighboring files
My app is a photo viewer, hence a lot of the time it is opened by a double-click on a file in File Explorer and uses the neighboring files query to get the other image files in the folder. It is essential that any fast API for listing files also covers this case, because it is the most common way of opening files for me. One important feature of this is that the files are returned in the current sort order of the folder in File Explorer (that's also essential for me), so it needs to be possible to reproduce that. I really wish this didn't use the indexer. In the past, lots of people have complained that 'the next and previous buttons are missing' in my app, which ultimately boils down to the fact that the neighboring files query sometimes is empty even though there are lost of images in the folder, which can occur when the indexer isn't working properly (I experienced this on my old PC and fixed it by an index rebuild). (However, I haven't had many of these reports recently, so it might have been fixed.)

File properties e.g. date taken, image dimensions
It is essential to have a method like RetrievePropertiesAsync for accessing file properties like date taken, image dimensions etc. I don't think this exists in .Net currently (correct me if I'm wrong). Ideally I'd like a fast implantation which doesn't require the overhead of creating a StorageFile (assuming I can use the new lightweight file APIs suggested by this issue), and one which doesn't use an extra 60kB of memory per file even if I only retrieve a few properties. Also, I don't know what degree of control the underlying Win32 APIs have, but it might be nice to be able to say 'I want to ensure these properties are up to date', or on the other hand 'I only want to retrieve these properties if it's going to be fast and doesn't involve decoding the file', etc. I think there is some attempt at this in some places in the UWP APIs but not really in a meaningful and reliable way.

Another question - what would be the starting point for accessing files? Would the FileOpenPicker still return a StorageFile, (similarly would FolderPicker still return a StorageFolder), or would it return something different? Similarly, what would the FutureAccessList APIs return (it is pretty slow currently).

To make my position clear, I don't mind if I have to change a lot of code to use the new APIs, as long as they are fast and easy to use from .Net, and we don't lose any existing functionality (possibly there might be one or two exceptions to that). The performance benefits will be worth it in my opinion. I also wouldn't mind keeping the same APIs with a massive speed improvement. The one thing I will say is that it's nice to have a unique persistent identifier for a file, e.g. a normalized file path. Ultimately this makes things a lot easier to work with rather than having to deal with the more abstract concept of a file that UWP encourages, even if the UWP approach is a bit more general. In order to perform any sort of diffing of lists of files etc, it's necessary to have some sort of unique identifier, which StorageFile doesn't currently guarantee. The only time I've used files that don't have a path are when using the neighbouring files query for a file on a phone, where the files are transferred using MTP. This is a bit of a nightmare to work with to be honest, and ideally there should be a better solution, but I suppose this doesn't need to go into V1 of the API.

Finally, I realize the permissions issue is complicated, but I hope you can start thinking about it at least!

[ptorr-msft]

The WinRT APIs are not going away. There is already a way to get from StorageFile to HANDLE and a round-about ish way to do the reverse (see this SO post). Theoretically we could provide a better way to do the reverse, although I believe there's a lot of private state held in a StorageFile that isn't possible to re-create from a simple HANDLE so it might not be 100% complete..

[ZodmanPerth]

I'm guessing it's all that additional state that makes StorageItem so slow to retrieve by the UWP API. If so, we probably need a set of Light objects (StorageItemLight, StorageFolderLight, StorageFileLight) containing just the essentials (Name, ItemType) and the means of retrieving more information on demand (AsStorageItem() method).

[eklipse2k8]

@ZodmanPerth I'm assuming the slowness is the RPC handshake since StorageItems are proxied through the storage broker. The AppContainer doesn't have allow direct access to file handles, so when you use StorageFile/Folder API, the broker goes off and gets the file handle for you.

[soumyamahunt]

@ptorr-msft Most of us want existing APIs to work with UWP instead of developing a new set of APIs. Along with that some performance improvement can be done for StorageFile APIs. One thing can be worked on is removing unnecessary annoyances from StorageFile APIs like:

• StorageFile.RenameAsync only works for renaming file to a supported file type that the app supports.

• Adding hidden files and shortcut file support in StorageFile.GetFileFromPathAsync.

Additionally work can be done to interop between these different set of APIs, i.e. getting StorageFile instance from a file handle so app can use StorageFile related methods. This would help developers to implement a new set of API without changing their existing code. Right now we can get file handle by calling StorageFile.CreateSafeFileHandle, adding the reverse would be helpful, i.e. StorageFile.GetFileFromSafeFileHandle(SafeFileHandle handle) or SafeFileHandle.CreateStorageFile(..additional parameters).

[soumyamahunt]

Theoretically we could provide a better way to do the reverse, although I believe there's a lot of private state held in a StorageFile that isn't possible to re-create from a simple HANDLE so it might not be 100% complete..

If it is possible to get a StorageFile from a path string, I think it should be easier to get StorageFile from HANDLE.

[ptorr-msft]

If it is possible to get a StorageFile from a path string, I think it should be easier to get StorageFile from HANDLE.

A filename and a HANDLE are not equivalent. For example the HANDLE could be duped from another process, and the current process doesn't have the rights to access the file directly (eg to get its extended properties).

[duke7553]

Shortcuts & Hidden Item Support for Virtualized File Vectors

There currently exists a way in WinRT to access a IStorageItem vector which is virtualized at a low level. Provided you're using a supported control like ListView/GridView, this approach provides slightly better performance than getting every storage item at once.

We like this approach and aim to eventually explore using it in Files UWP, however there is a need for the following to make it relevant for modern file management scenarios:

• Better documentation on how/when it should be used
• DataGrid support
• Support for hidden items and shortcut files

I just wanted @ptorr-msft and the team to know that this approach seems like a valuable area to improve upon.

[sylveon]

Not sure if that was mentioned here. but I would love being able to create StorageFile (or StorageFolder) from an existing Win32 handle. Currently it works from StorageFile -> Win32 handle, but not the other way around.

[mcosmin222]

Another thing that would be cool for UWP apps to do is to somehow register themselves as property providers for associated file types. For example, assume that the user has some .opus files, which the system does not read music metadata tags by default, then the app should be get a small background task call to provide the properties for that file type. This should be extended to all file types associations for the app.

[mfe-]

Another thing that would be cool for UWP apps to do is to somehow register themselves as property providers for associated file types.

Sounds good! I'm looking for a way for generating thumbnails for the file types which are associated for my sandboxed uwp app. I would suggest to create a separate feature request for that @mcosmin222 ? Maybe you like this issue as well:
Proposal: FileTypeAssociation/ context menus: context menus support for FileTypeAssociation where the app is not default for the file typ

Related links:
https://docs.microsoft.com/en-us/windows/win32/shell/thumbnail-providers

[h82258652]

For existing apps, we can use ImportRedirectionTable. But it does not suitable for the 3rd libraries.

[ptorr-msft]

@sylveon :

Not sure if that was mentioned here. but I would love being able to create StorageFile (or StorageFolder) from an existing Win32 handle. Currently it works from StorageFile -> Win32 handle, but not the other way around.

See my comment above. Theoretically this could be possible some of the time, but not always. You can get HANDLEs to things you don't normally have access to, and there is internal state in the StorageFile that isn't available from the HANDLE. What specific scenarios are you thinking of? Maybe there's a better way.

[sylveon]

I have an HANDLE to my log file, and wanted to use the UWP launcher APIs to open the log file upon user request. Those launcher APIs take a StorageFile. I can use the file path with GetFileFromPathAsync but it's more code (for me) than directly calling ShellExecuteEx (and C++ coroutines have shown to be annoying, in regards to IDE support and error diagnostics, so I try to avoid them).

However, in the end I decided to keep using ShellExecuteEx because I am using the lpVerb member to force my file to open as if was a .txt file, and AFAIK the UWP launcher APIs have no equivalent.

[mcosmin222]

Another thing missing in UWP is a DVD/CD/BLURAY drives access API.

[ZodmanPerth]

Something else loosely related is the need for better configuration options around file pickers (such as FileSavePicker). Something that is crucially missing is the ability to specifically configure the folder to open to. Currently you can only suggest well-known folders like libraries or the desktop, not a folder the app previously opened and has access to. The ability to provide a path of access token to the folder would be a huge improvement.

Also, some ability to define how file overwrites could be considered. If we want the user to provide a unique filename in a folder of their choosing, the only way to do this with a picker is to allow the picker to prompt the user to overwrite if they choose an existing file, return, detect it's an existing file, then open the picker again.

[Felix-Dev]

Something else loosely related is the need for better configuration options around file pickers (such as FileSavePicker). Something that is crucially missing is the ability to specifically configure the folder to open to. Currently you can only suggest well-known folders like libraries or the desktop, not a folder the app previously opened and has access to. The ability to provide a path of access token to the folder would be a huge improvement.

@ZodmanPerth This is being addressed by Project Reunion's File/Folder Picker. See the spec here.

[jtorjo]

Guys, can we have an actual API document or something, where we can give feedback upon? This discussion started in May, and I'm still not sure what/how/when will anything happen.
If there's already such an API doc, can I please get a link to it? Thanks!

[ptorr-msft]

The current discussion internally is to not require new APIs, but to just fix existing things. If we have something to share later, we will. Sorry and thanks for your patience.

[mcosmin222]

If the performance of winRT api was improved to be at least equally as fast as win32, that would be great, thanks.

[benstevens48]

@ptorr-msft - the one API that I think needs adding is an event on StorageLibraryChangeTracker that fires when the files/folders being watched have changed. At the moment, we have StorageFolderQueryResult.ContentsChanged which doesn't tell us which items have changed, and StorageLibraryChangeTracker which doesn't have a changed event. (I assume previous assertions that StorageLibraryChangeTracker can be used on any StorageFolder are correct - I haven't tested it myself).

I'd like to respond to changes as quickly as possible, so polling is not really viable. Also, I might like to watch many folders (e.g. 20) so I'd like the implementation to be as lightweight as possible (the background task provided by StorageLibraryChangeTrackerTrigger is not appropriate). I'm not sure if one of StorageLibraryChangeTracker and StorageFolderQueryResult.ContentsChanged is more heavyweight than the other.

Ideally registering such an event should not create a new thread per StorageLibraryChangeTracker. If I was writing a low-level method, I'd use something like RegisterWaitForSingleObject to wait on the wait object, so that an optimally small number of threads is used. (Note this API doesn't seem to be available to UWP, but it would be possible to polyfill or do something equivalent).

[sylveon]

Ideally registering such an event should not create a new thread per StorageLibraryChangeTracker. If I was writing a low-level method, I'd use something like RegisterWaitForSingleObject to wait on the wait object, so that an optimally small number of threads is used. (Note this API doesn't seem to be available to UWP, but it would be possible to polyfill or do something equivalent).

You can use ReadDirectoryChangesW (combined with IStorageFolderHandleAccess) to fire a callback on the current thread when a file/folder is changed. That API is also available to UWP. There's no thread creation involved here and it should be the cheapest method.

[benstevens48]

@sylveon - OK, thanks. Ideally there'd be an easier to use WinRT projection for use in C# though.

[benstevens48]

@ptorr-msft - If you are still thinking about fixing existing things, I just thought of another API that might need to be added/modified because it won't be possible to fully fix it. That is SorageFile.GetBasicPropertiesAsync (or SorageFolder.GetBasicPropertiesAsync). As I said before, this API is very slow compared to Win32 (typically takes around 5ms per file compared to 0.05ms). However, it can sometimes take over 100ms per file. Based on some heuristics, I think this is because it returns the ItemDate property, which could e.g. be the date taken for a Photo. Whilst this is useful, in my opinion it was a mistake to include this because it can be hugely expensive to return. For example, HEIC files (at least as of Windows 10 1909) are not properly indexed by the system for some reason. So (based on timing), it seems that calling GetBasicPropertiesAsync on a HEIC file will cause the date taken property to be read from the file metadata, which is very expensive. Moreover, if it's the first HEIC file you are calling this on, then the system will need to load 1 or 2 DLLs containing the codecs in order to decode the file! Thus we end up taking over 100ms just to get the file modified date!

So I think there needs to be a variant of GetBasicPropertiesAsync that just returns the date modified and file size but not the item date. (And it should also take approx 0.05ms per file, not 5ms, and certainly not 100ms!). Alternatively, you could fix StorageItemContentProperties.RetrievePropertiesAsync so that if called with System.DateModified or System.Size or similar then it returns an up to date value rather than a value it has cached from the previous call. This would not involve a new API. In fact in my opinion you should do both!

[jtorjo]

Based on some heuristics, I think this is because it returns the ItemDate property, which could e.g. be the date taken for a Photo

It's very very likely you are correct. The idea here is, there should be a simple StorageFile/StorageFolder.GetBasicProperties - which is sync, and it should completely ignore ItemDate. The fact that a simple function such as GetBasicProperties... is async, simply makes me wanna cry. This is an ubiquitous function - it's beyond outrageous that it's not sync.

As a side note, when porting my code to UWP, I simply ended up commenting every part of the code where I would need a file's length (because I'd need to have an async call) - I'm beyond nervous for having to do that, but there's not anything viable I can do.

[shresthasource]

Currently, I am able to compress files using ZipCompressionCommand and UWP. But extracting seems impossible with UWP API only. Extracting requires System.IO.Compression.FileSystem (ZipArchive.ExtractToDirectory()), which does not appear to be supported in pure UWP.

[ptorr-msft]

It's very very likely you are correct

I will second that "very very likely." File Explorer will chug to a halt if you try to sort a large directory by "date" since it tries to crack all those EXIFs to get "Date Taken" information, rather than just looking at the file attributes.

[duke7553]

@jtorjo Not that it solves the problem stated here, but Windows.Storage does currently have a way to get synchronous access to properties normally from a StorageFile. Although, it appears to only be designed for items fetched in a query.

[jtorjo]

@duke7553 I forgot about this one ;) I think I came across it a while ago. However, if I remember correctly, I think the API is kinda' useless, because you bind it to an IStorageQueryResult, which is insanely slow anyway.

So it's as if - sure, I do have I would say "sure, I'm giving you sync access, after you call GetFilesAsync and you can look at the result synchronously. Which pretty much takes us back to square one :D @duke7553 Have you used it - are there any performance gains compared to GetBasicPropertiesAsync?

[jtorjo]

Currently, I am able to compress files using ZipCompressionCommand and UWP. But extracting seems impossible with UWP API only. Extracting requires System.IO.Compression.FileSystem (ZipArchive.ExtractToDirectory()), which does not appear to be supported in pure UWP.

@shresthasource The simple workaround here is to copy the zip file to any of your "app's bubble" folders (for instance, ApplicationData.Current.LocalFolder.Path or any of its subfolders. Here, you can use System.IO and simply "do your thaaaang" - and it'll work awesomely. This is pretty much the thing that keeps me sane :D - a lot of things I can't do otherwise - I simply copy them there then sky's the limit.

[sylveon]

There's ApplicationData.Current.TemporaryFolder for that purpose.

[jtorjo]

There's ApplicationData.Current.TemporaryFolder for that purpose.

Yeah, that too - all I'm saying is that in the "app's bubble" you can use System.IO, which is great.

[shresthasource]

@jtorjo Not that it solves the problem stated here, but Windows.Storage does currently have a way to get synchronous access to properties normally from a StorageFile. Although, it appears to only be designed for items fetched in a query.

I have spent quite some time trying to get that to work. The Windows.Storage.BulkAccess FileInformationFactory GetVirtualizedItemsVector() works really fast, probably comparable to win32. But, it's basically useless. Why? Only thing you can do with those Virtualized Vectors are to bind them to a Grid or List. ItemsVector does not give you much info (not even item name, but does give sizes and dates). FilesVector and FolderVector gives more info, but there is no documentation on manipulating them before binding them to ListViews. For example, I cannot extract some information from those two and create a new list of file/folder and bind them to a view. I cannot directly figure out how many items are in the virtualized vectors.

The GetFilesAsyn() types might be a bit faster, but it also has hurdles that I could not pass to make anything useful out of that.

[shresthasource]

Currently, I am able to compress files using ZipCompressionCommand and UWP. But extracting seems impossible with UWP API only. Extracting requires System.IO.Compression.FileSystem (ZipArchive.ExtractToDirectory()), which does not appear to be supported in pure UWP.

@shresthasource The simple workaround here is to copy the zip file to any of your "app's bubble" folders (for instance, ApplicationData.Current.LocalFolder.Path or any of its subfolders. Here, you can use System.IO and simply "do your thaaaang" - and it'll work awesomely. This is pretty much the thing that keeps me sane :D - a lot of things I can't do otherwise - I simply copy them there then sky's the limit.

Thanks a lot! I was shocked I didn't think about this simple workaround. 👍 Funny enough, compressing the file does not need such workaround.

[shresthasource]

I don't know if this has been discussed before in other places. It appears that UWP API does not allow us to create a folder that has the same name as one of the files. Same for renaming. I thought it was probably my code in the past, but I am checking it again, and it appears to be a bug in the API. If it's a bug, it is a serious bug from my perspective.

[jtorjo]

I have spent quite some time trying to get that to work. The Windows.Storage.BulkAccess FileInformationFactory GetVirtualizedItemsVector() works really fast, probably comparable to win32. But, it's basically useless. Why? Only thing you can do with those Virtualized Vectors are to bind them to a Grid or List. ItemsVector does not give you much info (not even item name, but does give sizes and dates). FilesVector and FolderVector gives more info, but there is no documentation on manipulating them before binding them to ListViews.

@shresthasource Yeah, this is useless for my use case. I need to actually get this and get info about each file on a non-UI thread - don't need to bind them to anything.

[mcosmin222]

BulkAccess namespace looks like the kind of API that was put in place cause it served some specific purpose for some Microsoft application.

It is kind of sad that we cannot create FileInformation objects from outside the results of a query. Like for example, from the results of a file picker operation.

[duke7553]

@mcosmin222 Several of the methods you'll find in that namespace have very specific behavior, and were designed specifically for displaying storage items easily to a supported control like ListView and GridView. Sorry if I misunderstood your scenario.

Regardless, this is a space for improvement because it sounds like you and @jtorjo need to access storage items with very low-overhead and outside the context of a user interface. One of the goals for this effort is to expose an object oriented view of the existing Win32 *FromApp() methods which I interop with in my current project for high performance. Perhaps the new object-oriented version of those primitives should have a Thumbnail property which allows speedy access to item thumbnails. I'd like to see this scenario addressed, while making improvements to existing WinRT APIs to allow for viewing hidden items. Not sure if that is on the roadmap at this point.

Lastly, I opened a discussion here earlier in the summer about packaged UWP apps being unable to access network locations with FindFirstFileExFromApp() only in release mode. This is still an issue.

[jtorjo]

eed to access storage items with very low-overhead and outside the context of a user interface

@duke7553 Yes, that is correct. I've given up on waiting for an API to happen. I'm just using *FromApp() for now, and frankly, I've rewritten part of my app, to really seldom need this. And yeah, it hasn't been easy :D

[shresthasource]

Recently I noted that IndexerOption.OnlyUseIndexerAndOptimzeForIndexedProperties gives partial result sometimes even when the folder is fully indexed. In my specific case, the items were in OneDrive, but all of them were downloaded/available. It continued to show partial results for quite some time. Not sure if the Windows indexer just failed to note the change update the indexer or if it has something to do with the author of the docx/pdf documents. The result without using indexer included those files.

[shresthasource]

@ptorr-msft - the one API that I think needs adding is an event on StorageLibraryChangeTracker that fires when the files/folders being watched have changed. At the moment, we have StorageFolderQueryResult.ContentsChanged which doesn't tell us which items have changed, and StorageLibraryChangeTracker which doesn't have a changed event. (I assume previous assertions that StorageLibraryChangeTracker can be used on any StorageFolder are correct - I haven't tested it myself).

StorageLibraryChangeTracker does work on any directory and it does tell us what changed (moved, deleted, etc.). You can also get the item that changed (unless it is deleted). The speed and reliability is a different thing.

@ptorr-msft I would like to see StorageLibraryChangeTracker with an option of monitoring only the top-level items (shallow). As of now, if I am to monitor, say, C: drive, it will fire almost infinite number of events and the app would tend to freeze. StorageFolderQueryResult.ContentsChanged can monitor shallow change, but it does not provide any means of knowing what changed.

[shresthasource]

Another thing missing in UWP is a DVD/CD/BLURAY drives access API.

You can get DriveLists from UWP APIs. Although, the API might almost appear to be broken cause things that it's supposed to return returns nulls or exceptions, and it only works in certain way.

[Jaiganeshkumaran]

Another thing missing in UWP is a DVD/CD/BLURAY drives access API.

Windows no longer provides the required support for playing DVD movies due to licensing costs and newer PCs not having a DVD drive however you should be able to access CD/DVD drives for UWP already with broadFileSystemAccess. I have seen OEM UWP apps and Microsoft's DVD player app do that which are both UWP with no full trust

[goodmorningsky]

StorageFile.MoveAsync is super slow and not consistent. 4.4 seconds for 745k file, 3.6 sec for 2.49M, 78milisecon for 1.38MB and so on..
It's copied from C:\ sub folder to C:\ sub folder in local. It's simple test. It's so bad.

[mcosmin222]

I think performance also depends on the disk controller. However, this isn't very obvious with trasitional APIs because they are significantly faster anyways.