Runtime: Simple .NET app throws `(407) Proxy Authentiation Required`

[larssb]

Related to https://github.com/PowerShell/PowerShell/issues/9495#issuecomment-627207665

Why

I was requested to create a .NET app as I'm having this issue.

The context of that request has to come from the fact that PowerShell uses .NET Core internally.

The issue summarized

In PowerShell v7.1 preview 2. On a company network, where direct access to the Internet is not allowed and a PAC file, distributed via DHCP, is used to to set the proxy that the computer should use. This gives issues in PowerShell. Namely this error:

```
VERBOSE: Repository details, Name = 'PSGallery', Location = 'https://www.powershellgallery.com   /api/v2'; IsTrusted = 'False'; IsRegistered = 'True'.
VERBOSE: Using the provider 'PowerShellGet' for searching packages.
VERBOSE: Using the specified source names : 'PSGallery'.
VERBOSE: Getting the provider object for the PackageManagement Provider 'NuGet'.
VERBOSE: The specified Location is 'https://www.powershellgallery.com/api/v2' and PackageManagementProvider is 'NuGet'.
WARNING: Unable to resolve package source 'https://www.powershellgallery.com/api/v2'.
VERBOSE: Total package yield:'0' for the specified package 'invokebuild'.
Install-Package: C:\program files\powershell\7-preview\Modules\PowerShellGet \PSModule.psm1:9709
Line |
9709 |  … talledPackages = PackageManagement\Install-Package @PSBoundParameters
| 
| No match was found for the specified search criteria and module name 'invokebuild'. Try
| Get-PSRepository to see all available registered module repositories.
```

When executing the Install-Module -name invokebuild -Verbose -Scope CurrentUser -Repository PSGallery -Force PowerShell cmdlet.

As described in the issue on the PowerShell repo that I link to in the beginning. I've tried setting the below in a PowerShell session (quickly listed).

• [System.AppContext]::SetSwitch("System.Net.Http.UseSocketsHttpHandler", $false)
• [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12]
• 
  [system.net.webrequest]::defaultwebproxy = new-object system.net.webproxy('http://10.90.100.24:8080') [system.net.webrequest]::defaultwebproxy.credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials [system.net.webrequest]::defaultwebproxy.BypassProxyOnLocal = $true

Simple .NET app

```c#
using System;
namespace test
{
class Program
{
static void Main(string[] args)
{
var a = System.Net.WebRequest.Create("https://ya.ru");
var b = a.GetResponse();
//Console.WriteLine("b");
}
}
}

Executing the above app (the project was created with `dotnet new console`, result in the following err:

Unhandled exception. System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required.
at System.Net.HttpWebRequest.GetResponse()
at test.Program.Main(String[] args) in C:WORKrepostestProgram.cs:line 10
```

From here

If any additional info is needed I'll happily provide it. To the extent of my power and allowances. Also the PAC script in use might return more than one proxy addresses. But according to the issue (in the PowerShell repo) which I link to in the beginning, this should now be solved. So why could this error be occurring?

Thank you and looking forward to hear from you.

[msftbot[bot]]

Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

[davidsh]

The problem you are seeing is not related to the original issue which was fixed.

Since you are getting a response from the proxy:

(407) Proxy Authentication Required

the proxy discovery logic is working fine. The problem you are having is because you are not passing in credentials to the proxy. The proxy on your company's network requires credentials.

The manner in which you pass credentials to a proxy can vary. In particular, it depends on whether you need to pass in explicit username/password or just 'defaultCredentials'.

By the way, these lines of code will not help you with this problem:

[System.AppContext]::SetSwitch("System.Net.Http.UseSocketsHttpHandler", $false)
With PowerShell| Core and .NET Core, you will need to use HttpClient and not HttpWebRequest most likely. And you can use a variety of methods to set the proxy credentials for the default system proxy.

For .NET Core 3.1, we added a new static property on HttpClient called DefaultProxy which can be used to set the credentials for a default system proxy. Not sure which version of PowerShell Core will use .NET Core 3.1.

Here is some sample code in C#. You'll need to translate it appropriately for PowerShell use.

```c#
// Set credentials for the default system proxy.
HttpClient.DefaultProxy.Credentials = new NetworkCredential("myuser", "mypassword");

// Do the HTTP request.
var client = new HttpClient();
HttpResponseMessage response = await client.GetAsync("http://example.com");
```

[davidsh]

You might also try the following code in .NET to see if it works on your network.

```c#
using System;
namespace test
{
class Program
{
static void Main(string[] args)
{
// Set credentials for the default system proxy.
WebRequest.DefaultWebProxy.Credentials = new NetworkCredential("myuser", "mypassword");

        var a = System.Net.WebRequest.Create("https://ya.ru");
        var b = a.GetResponse();
        //Console.WriteLine("b");
    }
}

}
```

[iSazonov]

@davidsh Thanks! PowerShell works well with explicit credentials. So a question is why doesn't HttpClient use default network credentials from current user session?

[davidsh]

So a question is why doesn't HttpClient use default network credentials from current user session?

HttpClient, HttpWebRequest, etc. doesn't automatically use any credentials, even default network credentials. You must explicitly opt-in to using credentials.

In the case of "default network credentials", i.e. the credentials of the logged in user, you also need to specify those explicitly with CredentialCache.DefaultCredentials.

```c#
using System;
namespace test
{
class Program
{
static void Main(string[] args)
{
// Set credentials for the default system proxy.
WebRequest.DefaultWebProxy.Credentials = CredentialCache.DefaultCredentials;

        var a = System.Net.WebRequest.Create("https://ya.ru");
        var b = a.GetResponse();
        //Console.WriteLine("b");
    }
}

}
```

[iSazonov]

you also need to specify those explicitly with CredentialCache.DefaultCredentials

We use UseDefaultCredentials https://github.com/PowerShell/PowerShell/blob/master/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs#L982

I'd expect that already does WebRequest.DefaultWebProxy.Credentials = CredentialCache.DefaultCredentials if no explicit proxy credentials present.

[davidsh]

We use UseDefaultCredentials https://github.com/PowerShell/PowerShell/blob/master/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs#L982

Actually, the 'UseDefaultCredentials' only applies to using default credentials for communications with a server directly, i.e. a server that returns 401 Unauthorized.

It doesn't apply to passing in default credentials to a proxy that requires credentials and returns back 407. I know this sounds un-intuitive, but it is the behavior of HttpClientHandler as originally designed.

Looking at the implementation of this PowerShell commandlet:
https://github.com/PowerShell/PowerShell/blob/1bf5cc93175840fc22ee708c69a87fe704f77702/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/Common/WebRequestPSCmdlet.Common.cs#L982-L992

I would say that perhaps the bug is in the implementation of the PowerShell commandlet.

In fact, a better implementation would be perhaps the following (see // *** Also set any default system proxy to use default credentials ***)

```c#
internal virtual HttpClient GetHttpClient(bool handleRedirect)
{
// By default the HttpClientHandler will automatically decompress GZip and Deflate content
HttpClientHandler handler = new HttpClientHandler();
handler.CookieContainer = WebSession.Cookies;

        // set the credentials used by this request
        if (WebSession.UseDefaultCredentials)
        {
            // the UseDefaultCredentials flag overrides other supplied credentials
            handler.UseDefaultCredentials = true;

            // *** Also set any default system proxy to use default credentials ***
            handler.DefaultProxyCredentials = CredentialCache.DefaultCredentials;
        }
        else if (WebSession.Credentials != null)
        {
            handler.Credentials = WebSession.Credentials;
        }

```

[iSazonov]

@davidsh Thanks for clarify! We could use the fix in PowerShell.

I know this sounds un-intuitive, but it is the behavior of HttpClientHandler as originally designed.

I wonder if it is "by-design". The documentation says nothing about it:
https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.usedefaultcredentials?view=netcore-3.1
Reading the docs I expect that UseDefaultCredentials applies implicitly CredentialCache.DefaultCredentials.

[davidsh]

Reading the docs I expect that UseDefaultCredentials applies implicitly CredentialCache.DefaultCredentials.

Yes, it does imply CredentialCache.DefaultCredentials. However, it only applies for server credentials and not proxy credentials.

So, without the code changes I suggested to the PowerShell module, the proxy will not receive any credentials.

I wonder if it is "by-design". The documentation says nothing about it:

This is good feedback. We will work on improving our documentation in this area to be more explicit.

@karelz

[iSazonov]

This is good feedback. We will work on improving our documentation in this area to be more explicit.

Is there a reason CredentialCache.DefaultCredentials cannot be used by default for proxy?

Your code looks very wonder - like a duplication:
```c#
// the UseDefaultCredentials flag overrides other supplied credentials
handler.UseDefaultCredentials = true;

            // *** Also set any default system proxy to use default credentials ***
            handler.DefaultProxyCredentials = CredentialCache.DefaultCredentials;

```

(Right link to docs https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.defaultproxycredentials?view=netcore-3.1)

[davidsh]

Is there a reason CredentialCache.DefaultCredentials cannot be used by default for proxy?

It would be a breaking change in behavior to do that. And having any option to pass credentials by default has security concerns.

[davidsh]

Also:

```c#
handler.UseDefaultCredentials = true;

is really just a shortcut to doing this:

```c#
handler.Credentials = CredentialsCache.DefaultCredentials;

[karelz]

It seems that the remaining work is to document that UseDefaultCredentials does NOT affect proxy credentials and that devs should set DefaultProxyCredentials if they desire so.
We may add comment that UseDefaultCredentials is just shortcut for handler.Credentials = CredentialsCache.DefaultCredentials.

Is that correct summary?

[larssb]

I'll try to use the HttpClient tomorrow, when I'm back at work. To see if that fixes the issue. Sounds very plausible from the great walk-through in this issue. Great stuff. Thank you very much.

And as I understand it @iSazonov .. there is something that gives you info for a potential fix, to be back-ported into the PowerShell repo? Or am I too hasty here?

[karelz]

@aik-jahoda can you please include you in your docs work? I'll assign it to you -- see recommended changes summary: https://github.com/dotnet/runtime/issues/36274#issuecomment-627532033

[iSazonov]

Is that correct summary?

I agree. I'd add that .Net does not send a credential by default to a server or a proxy for security reason.

[iSazonov]

@larssb I am looking how fix PowerShell behavior in best way.

[larssb]

I got it to work by using the following code in my PowerShell profile script:

# For PowerShell to use the Proxy system settings. The bewlow is PS 6 core + compatible
# $true to the System.Net.WebProxy constructor is to indicate that the proxy should be bypassed for local addresses
[System.Net.Http.HttpClient]::DefaultProxy = New-Object System.Net.WebProxy('http://PROXY_IP:PROXY_PORT', $true)
[System.Net.Http.HttpClient]::DefaultProxy.Credentials = [System.Net.CredentialCache]::DefaultCredentials

Thank you guys. I can finally install PowerShell modules in PS7 Core. Greatness achieved 💯

@iSazonov cool if a solution/fix will be ported into a coming PS7 release.

[iSazonov]

I thing only second line enough to have.

[iSazonov]

@larssb Please re-open to track docs update.

[davidsh]

@larssb

I think only second line enough to have.

[System.Net.Http.HttpClient]::DefaultProxy = New-Object System.Net.WebProxy('http://PROXY_IP:PROXY_PORT', $true)

You don't need this line if automatic proxy discovery is working correctly, i.e. WPAD protocol, PAC files, etc. However, if it is not working, then setting an explicit proxy is ok too.

[larssb]

Thank you @davidsh. That is really good to know. I guess I should dive into my System.Net.HTTPClient classes ... some day when I get more time 👍

[tolgabalci]

[System.Net.Http.HttpClient]::DefaultProxy.Credentials = [System.Net.CredentialCache]::DefaultCredentials

What's the difference between the above and this:

[System.Net.WebRequest]::GetSystemWebProxy().Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials

This is a really difficult problem to troubleshoot as failed network connections do not always bubble up properly to the user, for example when Installing Modules. I'm not sure I understand the security concern around using Default Network Credentials when used in conjunction with the WINHTTP configured system proxy server. Maybe it could be allowed in such a conditional (yet highly common) predicament.

[iSazonov]

I'm not sure I understand the security concern around using Default Network Credentials when used in conjunction with the WINHTTP configured system proxy server.

We shouldn't _implicitly_ send a credential to external network.

[wfurt]

Part of this depends on authentication type. While connection to proxy is typically cleartext, basic auth would leak credentials. Others may be better but I agree with @iSazonov that this should not be implicit - even if it looks convenient. Since this is specific error code, it should be easy to handle it.

[tolgabalci]

@wfurt, the error might be specific, but many times does not bubble up to the user. For example one of the most common encounters of this error will be when a user is setting up PowerShell on a new system. One of the first steps users tend to do when setting up PowerShell is install modules. Here is the error you get when the Credentials have not been set to DefaultCredentials:

A similar message occurs no matter which module is attempted to be installed.

How about Update-Help? Another common setup step:

As you can see it is impossible to tell from the above errors that just the proxy credentials need to be set to DefaultCredentials. Instead the user is left in a frustrating situation, wondering what could be wrong with their PowerShell setup since their browser is likely connecting just fine through the proxy.

[wfurt]

unfortunately I'm not familiar with PowerShell code. Perhaps it should surface better error.