Runtime: Progress while downloading a large file with HttpClient

[eriksendc]

Hi All,

Googling around I found that Windows.Web.Http.HttpClient provides a way to get progress of a download. Is there an way to get some kind of progress information when downloading a large file using ASP.NET Core's HttpClient?

Thanks,
-Brian

[davidsh]

There are no progress callback features of System.Net.Http.HttpClient. There are no plans to add this. The feature set of CoreFx HttpClient matches that of .NET Framework (Desktop).

You can add this feature yourself by using the appropriate APIs to "download" content. For example, if you use the HttpClient.SendAsync(HttpRequestMessage, HttpCompletionOption.ResponseHeadersRead) API call then you will get an HttpResponseMessage with a .Content that hasn't yet been downloaded. So, you could then grab the Stream and read from it yourself and report back progress.

[eriksendc]

Thanks, @davidsh ! I actually found this after further googling:

http://stackoverflow.com/questions/21169573/how-to-implement-progress-reporting-for-portable-httpclient

I addapted it and created this:

using (HttpResponseMessage response = client.GetAsync(downloadUrl, HttpCompletionOption.ResponseHeadersRead).Result)
{
    response.EnsureSuccessStatusCode();

    using (Stream contentStream = await response.Content.ReadAsStreamAsync(), fileStream = new FileStream(dropZipPathAndFileName, FileMode.Create, FileAccess.Write, FileShare.None, 8192, true))
    {
        var totalRead = 0L;
        var totalReads = 0L;
        var buffer = new byte[8192];
        var isMoreToRead = true;

        do
        {
            var read = await contentStream.ReadAsync(buffer, 0, buffer.Length);
            if (read == 0)
            {
                isMoreToRead = false;
            }
            else
            {
                await fileStream.WriteAsync(buffer, 0, read);

                totalRead += read;
                totalReads += 1;

                if (totalReads % 2000 == 0)
                {
                    Console.WriteLine(string.Format("total bytes downloaded so far: {0:n0}", totalRead));
                }
            }
        }
        while (isMoreToRead);
    }
}

I picked the buffer size of 8192 after measuring the max bytes read in the call to contentStream.ReadAsync().

I'm trying to download a fairly big file from Visual Studio Team Services ("TFS in the cloud"). One thing I found when developing / debugging, is that VSTS will close a connection if a minimum throughput isn't being reached. (see http://blogs.msdn.com/b/taylaf/archive/2010/02/10/team-foundation-server-unable-to-read-data-from-the-transport-connection-an-existing-connection-was-forcibly-closed-by-the-remote-host.aspx). Originally I was writing out to the console on every read, but that slowed things down too much and I'd eventually get a "connection was forcibly closed" exception.

-Brian Eriksen

[vincentshow]

You can use ProgressMessageHandler when creating httpclient as it has HttpSendProgress and HttpReceiveProgress methods to tell you current progress. Something like this:

var processMsgHander = new ProgressMessageHandler(new HttpClientHandler());
processMsgHander.HttpSendProgress += (sender, e) =>
                {
                     //add your codes base on e.BytesTransferred and e.ProgressPercentage
                };

processMsgHander.HttpReceiveProgress += (sender, e) =>
                {
                      //add your codes base on e.BytesTransferred and e.ProgressPercentage
                };
var client = new HttpClient(processMsgHander);

[Misiu]

@davidsh anything changed with reporting progress? ProgressMessageHandler in only useful if You are doing ASP.NET.
I don't want to open another issue to ask for the same thing.
I have a workaround with streams, but having this inside CoreFx would be useful.

[davidsh]

This is a closed issue. There are no changes to reporting progress.

My previous feedback is still accurate:

There are no progress callback features of System.Net.Http.HttpClient. There are no plans to add this. The feature set of CoreFx HttpClient matches that of .NET Framework (Desktop).
You can add this feature yourself by using the appropriate APIs to "download" content. For example, if you use the HttpClient.SendAsync(HttpRequestMessage, HttpCompletionOption.ResponseHeadersRead) API call then you will get an HttpResponseMessage with a .Content that hasn't yet been downloaded. So, you could then grab the Stream and read from it yourself and report back progress.

[marcussacana]

Well... there are no alternative to track the download progress?
This useless httpclient is what works with the blazor...

[bugproof]

IMO there should be the ability to get progress and transfer speed out of the box with HttpClient for upload/download in a fashion similar to FluentFTP. Where you just pass IProgress<HttpProgress> and you can display it somewhere.

I don't understand why there are no plans to add this to make everything nicer. Please consider re-opening this.

Like in FluentFTP:

/// <summary>
/// Class to report Http Transfer Progress (Up and Donwload)
/// </summary>
public class HttpProgress
{
    /// <summary>
    /// A value between 0-100 indicating percentage complete
    /// </summary>
    public double Progress { get; set; }

    /// <summary>
    /// A value representing the current Transfer Speed in Bytes per seconds
    /// </summary>
    public double TransferSpeed { get; set; }

    /// <summary>
    /// A value representing the calculated 'Estimated time of arrival'
    /// </summary>
    public TimeSpan ETA { get; set; }

    /// <summary>Contructor for the class</summary>
    public HttpProgress(double progress, double transferspeed, TimeSpan remainingtime)
    {
        Progress = progress;
        TransferSpeed = transferspeed;
        ETA = remainingtime;
    }

    /// <summary>
    /// Convert Transfer Speed (bytes per second) in human readable format
    /// </summary>
    public string TransferSpeedToString()
    {
        var num = TransferSpeed > 0.0 ? TransferSpeed / 1024.0 : 0.0;
        return num < 1024.0 ? $"{Math.Round(num, 2)} KB/s" : $"{Math.Round(num / 1024.0, 2)} MB/s";
    }
}

@davidsh

The feature set of CoreFx HttpClient matches that of .NET Framework (Desktop).

Sounds like "We don't want to make HttpClient better because .NET Framework didn't support this"

[GF-Huang]

@davidsh

The feature set of CoreFx HttpClient matches that of .NET Framework (Desktop).

Sounds like "We don't want to make HttpClient better because .NET Framework didn't support this"

This seems to imply that people are going to use the old WebClient. LMAO. 😆

Although a variety of workarounds of response.Content.ReadAsStreamAsync() can be found on google, but when you want to download a large text and report progress, you need to implement the encoding/charset detection yourself to achieve the same behavior as GetStringAsync (HttpContent.cs#L188).

Great ! Microsoft used a clever way to guide me to learn their source code !