Powershell: Invoke-RestMethod is slow and uses a lot of memory

[KarlGW]

Steps to reproduce

Quick explanation, when describing it as slow and uses a lot of memory, I'm comparing Invoke-RestMethod from Windows PowerShell 5.1 with Invoke-RestMethod in PowerShell 6.0.1.

I'm running the below snippt inside a function in my module:

while ($continue) {

    $completed = $false
    $retries = 0

    # To retry until limit is reached.
    while(-not $completed) {

        try {
            Write-Verbose "Calling $uri"
            $call = Invoke-RestMethod -Method Get -Uri $uri -Headers $headers -Verbose:$false
            $data += $call.data
            $completed = $true
        } catch {
            if ($retries -ge $RetryCount) {
                Write-Verbose "API call failed after $retries retries."
                $completed = $true
            } else {
                Write-Verbose "API call failed. Retrying..."
                Start-Sleep -Milliseconds $RetryDelay
                $retries++
            }
        }
    }

    if ([string]::IsNullOrEmpty($call.nextLink)) {
        $continue = $false
    } else {
        $uri = $call.nextLink
    }   
}

Expected behavior

Expected behaviour is that the same code snippet should take about the same time and memory usage when running.

Actual behavior

When running on Windows PowerShell the snippet takes about 4 minutes, and take about 1,5 GB RAM (some objects that are stored in memory to be transformed by another function).

With PowerShell 6.0.1 (both tried within a Docker Container, and with the pwsh executable locally). The snippet takes between 12-20 minutes, and RAM goes up to 4-5 GB.

Environment data

Name                           Value
----                           -----
PSVersion                      6.0.1
PSEdition                      Core
GitCommitId                    v6.0.1
OS                             Microsoft Windows 10.0.16299
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

If any more information is needed, I can hopefully provide this.

[markekraus]

can you try your code without the extra stuff to eliminate other potential .NET Framework vs .NET Core differences?

something simpler like this:

Measure-Command  {
    $data = do {
        $call = Invoke-RestMethod -Method Get -Uri $uri -Headers $headers -Verbose:$false
        $uri = $call.nextLink
        $call.data
    } while ($uri) 
}

[KarlGW]

I ran your snippet (with the correct URIs of course):

Days              : 0
Hours             : 0
Minutes           : 12
Seconds           : 9
Milliseconds      : 674
Ticks             : 7296743233
TotalDays         : 0,00844530466782407
TotalHours        : 0,202687312027778
TotalMinutes      : 12,1612387216667
TotalSeconds      : 729,6743233
TotalMilliseconds : 729674,3233

In 6.0.1.

And for the case of comparison (5.1):

Days              : 0
Hours             : 0
Minutes           : 3
Seconds           : 16
Milliseconds      : 401
Ticks             : 1964017691
TotalDays         : 0,00227316862384259
TotalHours        : 0,0545560469722222
TotalMinutes      : 3,27336281833333
TotalSeconds      : 196,4017691
TotalMilliseconds : 196401,7691

Also, 6.0.1 does not release the memory until I kill the process, or do a [System.GC]::Collect().

[markekraus]

Is the URI localhost by any chance (something like http://localhost:8762/api/someendpoint)? If so, there is a known issue with .NET Core and resolving localhost. Try switching it to 127.0.0.1 or ::1. We ran into this in the tests inthe project and just switching to the IP brought the test runtimes down significantly.

On the memory usage, you can't really compare .NET Framework and .NET Core. The garbage collection works differently. Unless the memory is an issue (resource exhaustion), I wouldn't be concerned about the difference in ram usage between 5.1 and 6.0.1.

[KarlGW]

The URI is an external source, so not localhost.

The memory becomes somewhat of an issue if I'm running it inside a container in Docker, since it will kill the process. I can always increase the memory size of the container to mitigate that though.

[markekraus]

.NET manages the memory for you so it should start garbage collection when it exhausts.

What kind of data is being returned by the API? JSON, XML, plaint-text, binary?

[KarlGW]

It returns it's data in JSON. In the form of

{
    id: "id",
    data: {
        <more JSON with a lot of properties>
    },
    nextLink: <uri for next chunk of data>
}

[markekraus]

Is it possible to share some details about the API? Perhaps some documentation and/or mocked/scrubbed JSON result? Everything I have tested shows 6.0.1 out-performing 5.1 in both time and memory consumption. So, I'm wondering if the slow down is in the JSON deserialization. That would be heavily dependent on the object shape but one change between 5.1 and 6.0.1 was the JSON engine.

[iSazonov]

We need live repo. Is the URI public?

[KarlGW]

Yes, it's not our own API, but one provided by Microsoft.

Here is a link to the documentation: https://docs.microsoft.com/en-us/rest/api/billing/enterprise/billing-enterprise-api-usage-detail

The JSON return is a bit down in the article. But I'll provide a mockup response (provided by the article) example below:

{
    "id": "string",
    "data": [
        {                       
        "accountId": 0,
        "productId": 0,
        "resourceLocationId": 0,
        "consumedServiceId": 0,
        "departmentId": 0,
        "accountOwnerEmail": "string",
        "accountName": "string",
        "serviceAdministratorId": "string",
        "subscriptionId": 0,
        "subscriptionGuid": "string",
        "subscriptionName": "string",
        "date": "2017-04-27T23:01:43.799Z",
        "product": "string",
        "meterId": "string",
        "meterCategory": "string",
        "meterSubCategory": "string",
        "meterRegion": "string",
        "meterName": "string",
        "consumedQuantity": 0,
        "resourceRate": 0,
        "Cost": 0,
        "resourceLocation": "string",
        "consumedService": "string",
        "instanceId": "string",
        "serviceInfo1": "string",
        "serviceInfo2": "string",
        "additionalInfo": "string",
        "tags": "string",
        "storeServiceIdentifier": "string",
        "departmentName": "string",
        "costCenter": "string",
        "unitOfMeasure": "string",
        "resourceGroup": "string"
        }
    ],
    "nextLink": "string"
}

[KarlGW]

@iSazonov unfortunately the API is not public, you need an enrollmentnumber and an API key.

[KarlGW]

I did a test with Invoke-WebRequest and yes, you are correct, it's the part that converts the data to JSON that is significantly slower.

[markekraus]

@KarlGW In your test, did it include ConvertFrom-Json?

[KarlGW]

@markekraus Yes, in the tests after the mentioned one (first I only pulled the results), I put in a ConvertFrom-Json and time increased by a significant bit.

[markekraus]

OK. Do you have an estimate of the number of objects per page that API is returning? since it looks like the slowdown is in the JSON processing, We may be able to reproduce problem with the mockup.

[KarlGW]

The API returns 1000 objects per page.

[KarlGW]

Any updates on this? I tried messing around a bit with it and comparing, and indeed seems to be with the JSON-processing.

[markekraus]

No Update from my end. I'm aware that there are a bunch of performance issue in the JSON cmdlets. I have some ideas of how to fix some of them (such as allowing for object models and direct NewtonSoft.Json calls) but nothing concrete.

[iSazonov]

I could help if there is simple repos.

[markekraus]

@iSazonov the example in https://github.com/PowerShell/PowerShell/issues/6199#issuecomment-367006327 if you duplicate that 1000 times you can repro the problem.

What I would really like is for #6177 to be Implemented. Much of the slow down is likely in our code which is reinventing the wheel of what NewtonSoft.Json is doing internally (and better). Improving that looked to be a huge mess with a high risk of regression. Adding the feature to supply object models to the JSON cmdlets (also exposed so it can be consumed by internal APIs from Invoke-RestMethod) would allow us to fix 2 problems at once quickly and cheaply.

We'd still need to address performance in the normal deserailiztion code... but I think most instances where performance is a real issue could be solved with object models.

[iSazonov]

I suppose this wrapping comes from the fact that the .Net Framework was used earlier. Can anybody clarify - why we use Newton.Json today? Has CoreFX JSON support?

[SteveL-MSFT]

@isazonov seems there's a debate on having JSON support as part of corefx. Looks like System.Json is back, but only for compatibility reasons. We used Newtonsoft.Json early on because dotnetcore 1.0 didn't have any json support.

[iSazonov]

My concern is that if we use Newtonsoft.Json deeper, it won't be easy for us to go to CoreFX later.

[kclinden]

So is the problem when the REST Method returns JSON?

Mac Invoke Rest Method with PowerShell Core 6.1.1

Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 15
Milliseconds      : 594
Ticks             : 755948676
TotalDays         : 0.000874940597222222
TotalHours        : 0.0209985743333333
TotalMinutes      : 1.25991446
TotalSeconds      : 75.5948676
TotalMilliseconds : 75594.8676

Server 2016 Invoke Rest Method with PowerShell 5.1.14393.2189

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 86
Ticks             : 867426
TotalDays         : 1.00396527777778E-06
TotalHours        : 2.40951666666667E-05
TotalMinutes      : 0.00144571
TotalSeconds      : 0.0867426
TotalMilliseconds : 86.7426

[RealDrGordonFreeman]

@SteveL-MSFT

Still experiencing issues here with Invoke-WebRequest on WIN10-1809-x64.

My workaround was to use a new download function based on [System.Net.HttpWebRequest].

I think the PowerShell team needs to carefully review Invoke-WebRequest as there are many complaints about it's performance in multiple forums and sites, including here.

[iSazonov]

there are many complaints about it's performance in multiple forums and sites

@RealDrGordonFreeman could you please add references to these scenarios and use cases?

[RealDrGordonFreeman]

there are many complaints about it's performance in multiple forums and sites

@RealDrGordonFreeman could you please add references to these scenarios and use cases?

Sure.

https://stackoverflow.com/questions/28682642/powershell-why-is-using-invoke-webrequest-much-slower-than-a-browser-download

https://stackoverflow.com/questions/48268279/powershell-invoke-webrequest-is-consistently-very-slow-how-can-i-fix-this

https://stackoverflow.com/questions/14202054/why-is-this-powershell-code-invoke-webrequest-getelementsbytagname-so-incred

https://github.com/PowerShell/PowerShell/issues/5284

https://github.com/PowerShell/PowerShell/issues/2238

And many more. This cmdlet has problems. I think it needs to be reviewed.

(And also my own recent experiences with it. I had to write my own download function as a workaround.)

[iSazonov]

@RealDrGordonFreeman Thanks for links. Issue with ProgressBar was fixed. Please try _latest_ PowerShell Core version and report problems you find with it.

[j0hnwhyte]

I have to add my own issue with this, unfortunately uploading a sequence of large files using this function results in the process using gigabytes of RAM and then showing clueless errors. This is the configuration I use:

Invoke-WebRequest `
                -Uri "$url/$name" `
                -Credential $cred `
                -Method Put `
                -ContentType 'application/octet-stream' `
                -InFile $File

The issue is reproducible systematically, and it's just a simple webdav server on the endpoint.
I know we're talking about Invoke-RestMethod, but it seems that they share the same code underneath.

[vexx32]

@j0hnwhyte does that reproduce in the latest v7 preview releases?

[iSazonov]

@j0hnwhyte Please open new issue with repo steps.

[j0hnwhyte]

I can confirm that the v7 preview does not exhibit the issue, therefore the thing seems to be solved. I'll also try v6 and see if that works as well. Thanks

[david-tap]

https://github.com/PowerShell/PowerShell/issues/7698 was closed as a duplicate for this issue, but converting a JSON file of 287MB is still taking 8GB of RAM on the latest preview.

[9whirls]

Reproduced this issue on 7.0.1 on Mac.

measure-command {Invoke-webrequest -Uri 'https://disease.sh/v2/states' }                                                                       
                                                                                                                                                                                                      Days              : 0                                                                                                                                                                                 Hours             : 0                                                                                                                                                                                 Minutes           : 0                                                                                                                                                                                 
Seconds           : 1
Milliseconds      : 30
Ticks             : 10306922
TotalDays         : 1.19293078703704E-05
TotalHours        : 0.000286303388888889
TotalMinutes      : 0.0171782033333333
TotalSeconds      : 1.0306922
TotalMilliseconds : 1030.6922


measure-command {Invoke-restmethod -Uri 'https://disease.sh/v2/states' }

Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 16
Milliseconds      : 175
Ticks             : 761751440
TotalDays         : 0.000881656759259259
TotalHours        : 0.0211597622222222
TotalMinutes      : 1.26958573333333
TotalSeconds      : 76.175144
TotalMilliseconds : 76175.144

get-host

Name             : ConsoleHost
Version          : 7.0.1
InstanceId       : 47287e23-00c2-4ba5-a503-cbaa155f27cc
UI               : System.Management.Automation.Internal.Host.InternalHostUserInterface
CurrentCulture   : en-US
CurrentUICulture : en-US
PrivateData      : Microsoft.PowerShell.ConsoleHost+ConsoleColorProxy
DebuggerEnabled  : True
IsRunspacePushed : False
Runspace         : System.Management.Automation.Runspaces.LocalRunspace

[iSazonov]

@9whirls Please check on latest PowerShell 7.1 Preview. If you will see the problem please open new issue. Also please check on other platforms if you can.