Powershell: SOAP support in all platforms

[Sarafian]

Summary of the new feature/enhancement

Currently, the only way to consume a SOAP API from PowerShell 6 or higher, is to render manually the XML envelopes and do a POST with Invoke-RestMethod. Before PowerShell 6, there was New-WebServiceProxy which managed to provide some of the WS* standards and worked quite well with most simple SOAP interfaces. When this was not enough, one could fall back to coding in C# leveraging the System.ServiceModel assembly which unfortunately is not ported to .NET Core and therefore it is not an alternative in PowerShell 6 or higher.

SOAP is not as dead as some people proclaim. Yes, it is not the current choice, but there are so many established API in SOAP, that makes it still alive. The one I've most recently encountered is the Amadeus which is used in the travel industry extensively and it doesn't seem to be going away soon. The travel industry, banking and insurance sector are known for keeping what works and for this reason the existing SOAP endpoints will not go away anytime soon. This makes PowerShell effectively a non-suitable scripting language and it shouldn't be.

SOAP support was already there and needs to become available again for all platforms.

Use cases I can think of are:

• Use SOAP APIs from PowerShell on all platforms.
  
  
  
  • Cheaper NX based VMs
  
  
  • NX based containers support which are in general more lightweight than then windows server core counterparts.
  
  
• Leverage a much-improved PowerShell 6 experience and performance when working with SOAP. SOAP tends to be generally heavy and I've already noticed how much faster the newer versions are compared to the previous ones
• Integrate with SOAP using Azure Functions which currently supports only PowerShell 6. This would allow:
  
  
  
  • A quick and dirty abstraction over REST
  
  
  • Azure Logic apps would benefit as well
  
  
  • Overall integration with other cognitive services and chatbots when SOAP is involved.
  
  
• Windows Server Nano containers support.

Proposed technical implementation details

I think that this is not a PowerShell issue alone and support from the .NET team is required to port some of the missing APIs and assemblies and support the otherwise known WCF on other platforms.

Notice that on the roadmap announcement for .NET 5, many comments are about the missing support for WCF and there is quite a recognizable frustration.

Known repositories currently blocked from PowerShell 6

I've added some repositories that I've been in involved with over the last years when working with SOAP interfaces.

• 1ASOAP provides a toolkit for working with Amadeus (aka 1A) endpoints. The repository has three internal modules that are probably going to be split into 3 different repos. The SOAPProxy is the one that depends on New-WebServiceProxy to drive SOAP clients and this one blocks it from PowerShell 6. The remark is available here as well.
• WcfPS was built around C# and the System.ServiceModel and System.IdentityModel assemblies. It generates in memory proxies but it can support even federated authentication (WSTrust 1.3) which is not available in PowerShell. The dependency on the System.ServiceModel effectively blocks it from PowerShell 6.
• ISHRemote is an automation toolkit for a CMS over DITA by SDL known as Tridion Docs. It is built around C# and the System.ServiceModel and System.IdentityModel assemblies to leverage support for SOAP over federate authentications using WSTrust 1.3. The dependency on these assemblies blocks it PowerShell 6 and non-Windows based operating systems.

[SteveL-MSFT]

I don't know when or if .NET will support this, so an alternate would be if there was an Open Source library that is .NET Core compatible that could be used.

[BrucePay]

WRT a delivery vehicle, given that this is not a core scenario (anymore), I think making it availables as a module on PSGallery makes more sense. The trickier bit is getting the underlying .NET classes that PowerShell requires ported to Core. It might be helpful to open an issue in the .NET Core repository requesting this functionality.

[iSazonov]

It might be helpful to open an issue in the .NET Core repository requesting this functionality.

:-) And answer will

this is not a core scenario (anymore),

Although they could make a standalone package with the API.

[SteveL-MSFT]

Agree with @BrucePay, making it a standalone module makes sense regardless if we ship it with PowerShell package. Long term plan has been to have less cmdlets/modules ship with PowerShell and have them independently maintained, updated, and published on PSGallery.

[ChrisMagnuson]

In light of the news about PowerShell 7 trying to unify the fork and get us back on one version of PowerShell I came looking to see if New-WebServiceProxy was going to be addressed and found this issue.

I use New-WebService proxy to communicate with things like Adobe InDesign Server, Adobe Scene 7 Dynamic Media classic, ConnectShip Progistics (apis for shipping packages via Fedex, USPS, UPS, etc.), and in the past used this for managing Cisco Unified Communications Manager (and all the other on premise Cisco Collaboration tools).

Being able to quickly work with SOAP APIs is an important piece of the puzzule to enable System Administrators to automate a wide array systems they come in contact with using PowerShell.

[iSazonov]

@ChrisMagnuson Thanks for your feedback! Based on MSFT team suggestion could you please clarify - do you ready to grab this in separate community project?

[SteveL-MSFT]

Now that .NET Team has announced a community project for WCF support, this may be viable. Haven't looked to see how complete it is. If there is interest by the community to talk this forward, I can create a public repo for you to use.

[ChrisMagnuson]

In the article they mention “NET Core WCF Client“.

Given that in powershell we are acting as a soap client could that be used?

Googling for that name didn’t come back with something that matched that exactly, maybe this is what they are referring to: https://github.com/dotnet/wcf

[SteveL-MSFT]

@ChrisMagnuson I believe the right repo is https://github.com/CoreWCF/CoreWCF

[Sarafian]

@ChrisMagnuson and @SteveL-MSFT, I've done some more digging into this topic from the perspective of a couple of repositories I've in mind and mentioned on my original post.

The CoreWCF repo allows a soap client to function based on two requirements and their respected issues.

• The interface and the data contracts are defined in code. This means a binary module. But, PowerShell as a scripting language, needs to ability to consume a SOAP endpoint dynamically like New-WebServiceProxy does.
• Not all bindings are available, so complicated soap flows will not be supported. This limits the possibilities but I consider this a low impact issue. Most architect knew about the short commings of SOAP due to its complexity and have steered away from the most advanced bindings.

In the past, I've had already implemented an equivalent of the New-WebServiceProxy that could produce a more raw proxy but with much more support on the advanced bindings. I did this because I needed dynamic proxies that supported ws-trust authentication. This is WCFPS binary module and in short follows the following flow.

1. Create a wsdl importer
2. Extract the endpoint's interface and data contracts
3. Create a service endpoint based on a matching binding
4. Combine the interface and the service endpoint, to form a channel. The channel is the raw proxy. If you could wrap some code around it, then you could manage headers and add properties to mimic New-WebServiceProxy. Here is the example code.

$wsImporter=New-WcfWsdlImporter -Endpoint $svcEndpoint -HttpGet
$proxyType=$wsImporter | New-WcfProxyType
$endpoint=$wsImporter | New-WcfServiceEndpoint -Endpoint $svcEndpoint
$channel=New-WcfChannel -Endpoint $endpoint -ProxyType $proxyType

This is full WCF and hence it supports the most complicated bindings as they are described in the wsdl endpoint. Problem is that the necessary .net types are not implemented in the CoreWCF and I'm not sure if they are going to.

So, I considered a different approach. I've considered packaging dotnet-svcutil tool or its code into a module that behind the scenes would generate the code, compile it into memory and wrap it around an object. The problem is that I can't find the code for dotnet-svcutil and when I looked deeper I run into an answer stating "that this is not open source yet. ". I've also not been able to find the code for the New-WebServiceProxy to try and .NET standar-ize it and offer it a binary module.

I wonder is someone has some more insight about the source of dotnet-svcutil and New-WebServiceProxy

[iSazonov]

@Sarafian See https://github.com/PowerShell/PowerShell/blob/bd6fdae73520931f0d27a29d6290e18761772141/src/Microsoft.PowerShell.Commands.Management/commands/management/WebServiceProxy.cs#L38

[Sarafian]

@Sarafian See

https://github.com/PowerShell/PowerShell/blob/bd6fdae73520931f0d27a29d6290e18761772141/src/Microsoft.PowerShell.Commands.Management/commands/management/WebServiceProxy.cs#L38

Thanks @iSazonov . The workhorse in the New-WebServiceProxy is the DiscoveryClientProtocol offered by the System.Web.Services.dll as already mentioned. The DiscoveryClientProtocol has some dependencies to other types from the same assembly and from a sampling I did, none of them are available in the upcoming netcore-3.0 (currently preview 6).

System.Web.Services.dll is the driver for ASMX web services and I'm not sure that any solution should be based on this codebase. This is the reason that New-WebServiceProxy is limited and supports very basic soap configurations.

Has anyone ever tried to convert its code to .NET standard?

I'm more and more convinced that this is .NET team issue and maybe the PowerShell team can put an extra word :smile: . I hope the new WCF in .net core 3, shares similarities in terms of features with New-WebServiceProxy. If they offer some the constructs to configure a Web Service, then hopefully they offer the drivers to generate in memory proxies like e.g. DiscoveryClientProtocol. In WCF this was WsdlImporter and still unavailable for .net core 3 (preview 6)

[SteveL-MSFT]

@Sarafian Unfortunately, I don't see .NET team treating this as a priority at all which is why they invested in the CoreWCF project (even if incomplete). They are investing in REST and gRPC. We're doing some work in PS7 to make it more seamless using Windows PowerShell modules, so perhaps we can make it easier to call Windows PowerShell's New-WebServiceProxy from PS7.

[Sarafian]

@Sarafian Unfortunately, I don't see .NET team treating this as a priority at all which is why they invested in the CoreWCF project (even if incomplete). They are investing in REST and gRPC.

@SteveL-MSFT thanks for reaching out and clarifying. It is expected what you said.

We're doing some work in PS7 to make it more seamless using Windows PowerShell modules, so perhaps we can make it easier to call Windows PowerShell's New-WebServiceProxy from PS7.

That would solve homogenity within the powershell experience, because the current way of executing windows specific modules with implicit remoting is really not good. Anyhow, this will not solve azure function with powershell, unless the functions teams decides to "strengthen" the powershell support.

[SteveL-MSFT]

Azure Functions can run on a Windows sandbox which contains Windows PowerShell. Even today, you could from PSCore6 call out to powershell.exe to run whatever Windows PowerShell pipeline you need.

[Sarafian]

Azure Functions can run on a Windows sandbox which contains Windows PowerShell. Even today, you could from PSCore6 call out to powershell.exe to run whatever Windows PowerShell pipeline you need.

Hmmm didn't know that. This is in general good news. Potential problem is that with every powershell.exe .... from the Core session, the soap proxy would need to intiatiated. I'm not sure how it would be with the core session, if the process is kept alive. So with much tentativeness, my comment.

Thanks @SteveL-MSFT .

[mikeTWC1984]

I was looking for a way to mirror New-WebServiceProxy on linux powershell (to consume SharePoint soap services), and dotnet-svcutil seem to be a good option. It's not generating client automagically like windows powershell, but actually doing this manually might be even better approach in many cases (you just generate client code once and then reuse it). Below are some notes to get started (if not familiar with this tool)

To generate cs file with client definition, install dotnet sdk 2 (it's not compatibale with v3 yet, but you can have both), then install the tool itself (dotnet tool install --global dotnet-svcutil) and create some dummy app (dotnet new console)
Then download all wsdl files you need and run: dotnet-svcutil *.wsdl -sync
If no error, build the project (dotnet build) and import project dll (Add-Type -path ./bin/Debug/netcoreapp3.0/dummy.dll)
You can also load the type from cs file directly:

Add-Type -TypeDefinition (get-content -raw ./ServiceReference/Reference.cs) -ReferencedAssemblies  netstandard, 
 System.ServiceModel,
 System.ServiceModel.Primitives,
 System.Private.ServiceModel,
 System.Runtime.Serialization.Xml,
 System.Xml, System.Diagnostics.Tools,
 System.Xml.ReaderWriter,
 System.Diagnostics.Debug,
 System.Runtime.Serialization.Primitives 
# might need some extra libs to refer

Once type is loaded you should instantiate Client object. Type name likely will be "ServiceName" + "SoapClient", or something containing Client word. In case of SharePoint lists service:

$defaultConfig = [ServiceReference.ListsSoapClient+EndpointConfiguration]::ListsSoap
$client = [ServiceReference.ListsSoapClient]::new($defaultConfig)

If there is no auth or you are on windows network with NTLM you are likely all set. Otherwise try to configure auth using $client.ClientCredentials property, or you can also use other constructor for the client:

$result = [System.ServiceModel.BasicHttpBinding]::new()
$result.Security.Transport.ClientCredentialType = [System.ServiceModel.HttpClientCredentialType]::Ntlm; # choose the one you need
$endPoint = [System.ServiceModel.EndpointAddress]::new("https://sharepoint.site/_vti_bin/Lists.asmx")
$client = [ServiceReference.ListsSoapClient]::new($result, $endPoint)

You can also use this constructor to consume same service from different endpoint.

[Sarafian]

@mikeTWC1984 have considered wrapping everything in and generating proxies dynamically like it is done in the WcfPS module? It uses .net classes that normally the wsdl importers use to generate the service references but instead of saving in file it compiles in memory.

[mikeTWC1984]

@Sarafian is WcfPS compatible with powershell core, at least with Windows version?

[Sarafian]

@Sarafian is WcfPS compatible with powershell core, at least with Windows version?
I am not sure. You will probably need to change some namespaces. Keep in mind that the advanced stuff related to token are definitely not available.

[vazome]

Please add this