Powershell: Web Cmdltes Lack FTP Support
The move to HttpClient PowerShell Core brought with it a loss of functionality. The FTP and FILE URI schemas were supported in Windows PowerShell, but HttpClient only supports HTTP and HTTPS schemas. While parity for the FILE schema is debatable, FTP is still widely used and more especially in the Linux world.
For future versions of PowerShell we should support FTP. Though, it may not be possible to do so with the Web Cmdlets as they are now. I really like HttpClient and don't think switching APIs is the right course, Perhaps this may require a new cmdlet.
Steps to reproduce
Invoke-WebRequest ftp://repos-jnb.psychz.net/HEADER.html
Expected behavior
StatusCode : 0
StatusDescription :
Content : {60, 104, 116, 109...}
RawContent :
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8">
<title>CentOS Mirror</title>
<link rel="stylesheet" type="text/css" charset="utf-8" media="screen" href...
Headers : {}
RawContentLength : 1234
Actual behavior
Invoke-WebRequest : Only 'http' and 'https' schemes are allowed.
Parameter name: requestUri
At line:1 char:1
+ Invoke-WebRequest ftp://repos-jnb.psychz.net/HEADER.html
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Invoke-WebRequest], ArgumentException
+ FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
Environment data
Name Value
---- -----
PSVersion 6.0.0-beta.9
PSEdition Core
GitCommitId v6.0.0-beta.9
OS Microsoft Windows 10.0.15063
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
markekraus
All 28 comments
Should we ask CoreFX to enhance HttpClient?
iSazonov
on 17 Nov 2017
@iSazonov I don't think we should ask, no. HttpClient's intended purpose is HTTP and HTTPS only. It is intentionally limited in schema support to provide specialization for HTTP related requests, much of which has no relevance to FTP or other protocols.
markekraus
on 17 Nov 2017
@markekraus Thanks for clarify - I see CoreFX support FTP and File UriSchemeHttp in another code path.
iSazonov
on 17 Nov 2017
FileWebRequest and FtpWebRequest? I believe WebRequest is what was being used in Windows PowerShell. It may be possible to fall back to those when the URI is FTP or FILE, but, many of the options and features in place are built around HttpClient and wont work for WebRequest. I'm not a fan of that.
This somewhat falls in line with my suggestion for separating the web cmdlets (and related cmdlets) from the rest of the project and leaving a very simple file downloader cmdlet baked in. My gut tells me that there is a missing cmdlet here. I realize it is somewhat redundant with IWR and IRM, but a cmdlet that just downloads a file with very basic options and no processing overhead seems desirable. All of the value in IRM and IWR is tied to their post-processing capabilities. They support -OutFile but so much overhead goes to processing the request before it reaches that point. The download cmdlet could be expanded to support a wide variety of protocols (i.e FTP & TFTP) that make no sense for the HTTP processing of IWR and IRM,
Also, FTP might be a prime candidate for a PSProvider and to be used with Get-Content and friends. I think managing an FTP server with those would be awesome as there is way more to FTP than get and put.
markekraus
on 17 Nov 2017
I think TFTP is specific and we should track the idea in separate Issue.
iSazonov
on 17 Nov 2017
TFTP might be specific (and also probably has a low demand), The point is that a separate cmdlet for just remote downloads opens up the possibility of plugging in support for additional transport protocols that would make no sense being in the web cmdlets. Maybe Get-RemoteFile, Invoke-Download, or something. I'm not exactly sure adding FTP support back to the web cmdlets is the right direction. So providing additional justification for a separate cmdlet makes sense to me.
markekraus
on 17 Nov 2017
Personally, I would prefer separate cmdlets for FTP
SteveL-MSFT
on 17 Nov 2017
TFTP might be specific (and also probably has a low demand),
Network administrators can live only with TFTP :-) - this is a huge number of consumers. I think they will be happy to discover TFTP cmdlets in PowerShell.
iSazonov
on 17 Nov 2017
Oh I know they will. I meant "low" in relative terms to demand for HTTP based features and FTP support. I have definitely longed for a native TFTP client implementation in PS myself. Just being able to verify a PXE config with PS would be a dream, and that's not even touching the networking equipment. 馃槃
markekraus
on 17 Nov 2017
From the approved verbs list, Receive, Send, and Sync make the most sense with a -Protocol parameter to specify FTP and then later SFTP, TFTP, etc.
Not sure if the noun should be Item or File though. I see pros and cons to either one.
dragonwolf83
on 18 Nov 2017
I would definitely got for a cmdlet like Invoke-FtpRequest with a -send or -receive parameters.
:)
MaximoTrinidad
on 18 Nov 2017
Would it be better to have a FTP PSDrive (perhaps using SHiPs)? Then you would do something like:
Connect-FTP -Host foo -Credential $cred
cd ftp:\foo
copy *.md ~\Downloads
copy ~\Documents\*.md .\documents
In copy *.md ~\Downloads where FTP drive and where Filesystem drive?
iSazonov
on 18 Nov 2017
@SteveL-MSFT for a full FTP solution, sure. But, I'm willing to bet 90% or more of all FTP user stories are "I have an FTP URI to a file I need to download".
With Windows PowerShell 5.1 it is as simple as iwr ftp://server.com/path/file.txt -outfile file.txt On linux and mac it's similarly easy to do that with wget/curl. If we make it painful, the user will shell out. I'd rather we have a native solution.
I see room for both, a network file PS drive (for NFS, FTP, SFTP, FTPS, TFTP, etc) and a simple download cmdlet (for HTTP and the others).
I'd envision the PS drive being defined with New-PSDrive like all other providers. I may want to connect to more than one FTP server and possibly even copy between them or copy/move between FTP and NFS. This would accommodate the user story "I have a network file resource I need to manage." It would be similar to how SMB is managed by the FileSystem provider.
I'd envision the download cmdlet to be very basic taking a URI, optional credentials and optional path. Invoke-Download ftp://server.com/path/file.txt would be the minimum required and it would save the file in the current folder similar to how wget operates, unless the file path is supplied
Invoke-Download -Uri <Uri> [-Credential <PSCrdential>] [-OutFile <String>]
It would work off the URI schema and we could plug network/transport protocols as needed. It would be intentionally feature poor to become the de facto file download cmdlet in PowerShell for all protocols.
@dragonwolf83
Receive doesn't really make sense. That implies it is server initiated, in most cases it would not be. in FTP, downloads are client initiated. Get and Set make sense, but if we are to generalize a cmdlet that may also work with protocols that send/receive, Invoke makes sense.
markekraus
on 18 Nov 2017
@iSazonov in the sample I wrote, the cd ftp:\foo would change your current working directory to ftp:\foo\ so *.md would be relative to ftp:\foo\*.md and ~\Downloads would always resolve to the Downloads folder in your home directory. Should work exactly the same as any other PSDrive.
@markekraus I agree that having cmdlets for simple cases makes sense. Personally, I haven't touched ftp in a LONG time. The last time I used FTP heavily I was using NcFTP on Unix at University of Washington :P
SteveL-MSFT
on 19 Nov 2017
FTP user has a home directory too.
iSazonov
on 19 Nov 2017
ftp:\<server>\<path> is what I was thinking, so ftp:\foo\ would be root.
SteveL-MSFT
on 19 Nov 2017
Considering cross-platform, I think it's better to prepare another cmdlet based on libcurl's thin wrapper.
Although it may not be appropriate for this topic.
takekazuomi
on 20 Nov 2017
@takekazuomi For cross-platform we would just need to make calls to CoreFX API's. They take care of the cross-platform low-level specifics for us. On Linux and macOS they are wrapping libcurl at the low level. I'm not sure of the exact mechanism on windows, but it doesn't really matter.
markekraus
on 20 Nov 2017
Interesting, if we'll many file providers - Filesystem:, FTP:, SSH:, WebDAV: we should beforehand optimize our code #5002
iSazonov
on 20 Nov 2017
@iSazonov Were those optimizations needed in the FileSystem provider? If so, they wouldn't affect this as this would be a new provider or possibly a provider for each protocol. They wouldn't necessarily need to be baked into the project either. As @SteveL-MSFT mentioned, using SHiPS is a possibility.
If the improvements are needed in the providers meta code, then it's needed more than ever with SHiPS making PSProviders something worth doing now. (side note, this makes me really happy as Provider are a severely underappreciated feature of PowerShell).
markekraus
on 20 Nov 2017
Yes, this may require changes to internal API. So it is better first optimize the API.
iSazonov
on 20 Nov 2017
It can be done in parallel, no? Breaking changes to PS Provider couldn't be added until 7.0 at this point. I think I could probably manage a provider, I don't think I could manage optimizations of the provider meta code.
markekraus
on 20 Nov 2017
It is internal API - we can (I hope) avoid breaking changes and it is a huge work.
iSazonov
on 20 Nov 2017
I'm confused. The ability to add PS providers is a public API, not an internal one, Correct? The FTP et. al Provider(s) would be new providers that use the public API. The creation of the new provider could be done in parallel with the improvement of the Provider meta code improvements so long as those improvements do not break the public API. if the improvements are just in the FileSystem provider, then it could definitely be done in parallel as it shouldn't have any effect on a new provider.
markekraus
on 20 Nov 2017
It is three level API: cmdlets, intermediate (common provider API) and low level provider implementation. We should start optimizations (I guess. It can actually be harder) in intermediate level to exclude reparsings, add cachings and so on - this will require a great change in (Filesystem) provider implementation.
Of course we can implement new provider now and then start optimizations but we'll do double work.
We're very slow in Filesystem provider on PowerShell level and on CoreFX level too. PowerShell Core is worse than Windows PowerShell. It follows that this is critical to the quality of the product and we should work in this direction the sooner the better.
iSazonov
on 20 Nov 2017
Here's a cross-platform FTP PS Provider, is this something that can get merged into the PowerShell Core codebase? Is that how this works?
alexjordan6
on 17 Jun 2018
@alexjordan6 that looks like a good start, but looking at the README it appears incomplete (copying a file requires calling .Net apis). It would make sense to me to have community members help get that provider complete and published to PSGallery. I don't think there's a need to include more into the PSCore6 pkg. We actually want to pull some of the in-package modules out as separate modules published to the gallery.
SteveL-MSFT
on 17 Jun 2018
Related issues
lzybkr
路
3Comments
mklement0
路
3Comments
Michal-Ziemba
路
3Comments
pcgeek86
路
3Comments
JohnLBevan
路
3Comments
Most helpful comment
Personally, I would prefer separate cmdlets for FTP