Runtime: HttpWebRequest in .NET Core 2.0 throwing 301 Moved Permanently

[lviana1]

The following single-line console program works just fine in .NET Framework 4.6.1:

```c#
Console.WriteLine(XDocument.Load("http://feeds.thisamericanlife.org/talpodcast"));

However, in .NET Core 2.0 it throws the following exception:

Unhandled Exception: System.Net.WebException: The remote server returned an error: (301) Moved Permanently.
at System.Net.HttpWebRequest.GetResponse()
at System.Xml.XmlDownloadManager.GetNonFileStream(Uri uri, ICredentials credentials, IWebProxy proxy, RequestCachePolicy cachePolicy)
at System.Xml.XmlDownloadManager.GetStream(Uri uri, ICredentials credentials, IWebProxy proxy, RequestCachePolicy cachePolicy)
at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn)
at System.Xml.XmlTextReaderImpl.FinishInitUriString()
at System.Xml.XmlTextReaderImpl..ctor(String uriStr, XmlReaderSettings settings, XmlParserContext context, XmlResolver uriResolver)
at System.Xml.XmlReaderSettings.CreateReader(String inputUri, XmlParserContext inputContext)
at System.Xml.XmlReader.Create(String inputUri, XmlReaderSettings settings)
at System.Xml.Linq.XDocument.Load(String uri, LoadOptions options)
at System.Xml.Linq.XDocument.Load(String uri)
at ConsoleApp1.Program.Main(String[] args)
```

This has always worked in the full .NET Framework, presumably because it automatically follows the redirection. However, it does not do that in .NET Core (it seems that AllowAutoRedirect is simply ignored), so I'm assuming this is a bug. Please let us know if there's a workaround.

[EDIT] Fixed C# syntax highlight & exception code formatting by @karelz

[karelz]

Are you able to reproduce the same problem with just using Networking APIs? (without XML APIs) -- that will help us identify if it is problem in XML or Networking area.

BTW: The URL above does not seem to be forwarded anywhere in browser. Is that expected?

[lviana1]

Yes, I can reproduce the same problem using just Networking APIs. You can use the simple console app below:

```c#
using System.Net;

namespace ConsoleApp1
{
class Program
{
static void Main(string[] args)
{
var request = WebRequest.Create("http://feeds.thisamericanlife.org/talpodcast");
var response = request.GetResponse();
}
}
}

If you target that console app against .NET Framework 4.6.1 it works fine. However, it you target .NET Core 2.0 using the very same code, on the very same IDE (Visual Studio 2017 15.3), running on the very same OS (Windows 10), it will throw the exception below:

Unhandled Exception: System.Net.WebException: The remote server returned an error: (301) Moved Permanently.
at System.Net.HttpWebRequest.GetResponse()
at ConsoleApp1.Program.Main(String[] args)
```

[EDIT] Fix C# syntax highlighting & exception code formatting by @karelz

[sandersaares]

The site has an interesting redirect pattern:

1. Redirect to HTTPS variant of the same URL.
2. Redirect to a different URL over HTTP.

.NET Core appears to be throwing the exception on the second redirect.

[lviana1]

FWIW, I've seen similar exceptions being thrown with other sites in .NET Core. For instance: http://www.npr.org/rss/podcast.php?id=35.

[karelz]

https->http redirection is not supported - see dotnet/runtime#21446

[lviana1]

Are you saying that this is by design in .NET Core, but NOT in the full .NET Framework? As I mentioned above, the exact same code has always worked just fine when targeted against the full .NET Framework, so if this is by design in .NET Core and NOT in the full .NET Framework that seems inconsistent...

[davidsh]

Are you saying that this is by design in .NET Core, but NOT in the full .NET Framework?

Yes, this is by-design in .NET Core. This is a security-related change to prevent HTTPS -> HTTP redirection which is inherently insecure. For apps that need to follow those redirections, you can turn off automatic redirection in the handler and follow the Location: response header manually.

[tynor88]

If you do:

HttpWebRequest webRequest = HttpWebRequest.CreateHttp("https://someurl.com");
webRequest.AllowAutoRedirect = false;

And read the response from an Uri that gets redirected to _http_ the following code throws an exception:

var resp = (HttpWebResponse)webRequest.GetResponse()

It looks like the AllowAutoRedirect = false doesn't work properly.

Is that also by design @davidsh ?

[davidsh]

If you set AllowAutoRedirect, then you will end up not following the redirect. That means ending up with the 301 response.

HttpWebRequest (unlike HttpClient) throws exceptions for non-successful (non-200) status codes. So, getting an exception (most likely a WebException) is expected.

So, if you need to handle that redirect (which is HTTPS -> HTTP by the way), you need to trap it in try/catch block and inspect the WebException etc. That is standard use of HttpWebRequest.

That is why we recommend devs use HttpClient which has an easier use pattern.

[tynor88]

Makes sense. Thanks for the swift explanation!

[krispenner]

Can you check the original URL and if it was insecure http, then follow any future redirects that are considered a security downgrade such as https > http that pop up in the redirect pipeline, because the user was never requesting https to begin with. The WebClient was made for this purpose to just work as easily as possible to download a resource and the number of sites that redirect http up to https and then back down to http seem common, Yahoo included, so I think this should be addressed better.

[RevealedFrom]

I have AllowAutoRedirect set to false, and the exact same code and web site that works fine in normal .NET throws an exception in Core when the response is 302.

In my case, the redirect was from HTTPS to HTTPS, same web server.

[karelz]

@RevealedFrom if you set AllowAutoRedirect=false, then it is expected behavior to get the exception for 302 based on previous comment https://github.com/dotnet/corefx/issues/23422#issuecomment-327589873

[RevealedFrom]

But it wasn't the case in old .NET. I simply copied and pasted the code from Visual Studio to VS Code, and ran against the same url. In previous versions of .NET, HttpWebRequest does not throw an exception when the response is 301/302.

My url called was https://10.x.x.x/cis.jsp and the response was 302 Location: https://10.x.x.x/loginPage.jsp.

[jgilbert2017]

No, it's not the "standard use" of HttpWebRequest.

The flag AllowAutoRedirect has always allowed for the caller to manually process the redirects which is an important piece of functionality. This breaks exisintg applications and is NOT a good API design decision. There should be a way to manually process the redirects without resorting to try/catch.

@karelz you need to trap it in try/catch block and inspect the WebException etc. That is standard use of HttpWebRequest.