Newtonsoft.json: Out of memory exception

Created on 18 Mar 2019 · 11Comments · Source: JamesNK/Newtonsoft.Json

Hi,

thanks for this great Library!
i use this Library to open HAR (Debug-Files) from Chrome Webbrowser.
if i exceed 200 Mb i get an Out of Memory Exception.
Can you help to fix this issue?

Source

Phreak87

Most helpful comment

It should be noted that, in the following code, you are not actually reading directly from a stream:

Dim RDR As New System.IO.StreamReader(OFD.FileName)
Dim Json = New Newtonsoft.Json.JsonTextReader(RDR)
JRoot = JsonConvert.DeserializeObject(RDR.ReadToEnd)

What you are actually doing here is opening a StreamReader, allocating a JsonTextReader, then ignoring the JsonTextReader and reading the entire stream into a string, then parsing the string. This violates the suggestion from Performance Tips: Optimize Memory Use to deserialize large files directly from a stream.

To deserialize directly from a stream in VB.NET, you would create a method like:

Public Module JsonExtensions
    Public Function LoadFromFile(Of T)(path As String, Optional ByVal settings as JsonSerializerSettings = Nothing) As T
        Using reader As New System.IO.StreamReader(path)
            Using jsonReader as New Newtonsoft.Json.JsonTextReader(reader)
                Return JsonSerializer.CreateDefault(settings).Deserialize(Of T)(jsonReader)
            End Using
        End Using
    End Function
End Module

And then use it like:

Dim JRoot as JToken = JsonExtensions.LoadFromFile(Of JToken)(OFD.FileName)

The fact that you're not actually streaming may account for some of your problem.

Demo fiddle: https://dotnetfiddle.net/ClyNbD

dbc2 on 22 Mar 2019

👍3

All 11 comments

Buy more RAM.
Use a 64-bit operating system.

elgonzo on 20 Mar 2019

👎9

@elgonzo please do not give such advise to people.

@Phreak87 I'm having similar issues with very large json sets, in my case it is due to memory control of the dotnet framework itself and the only cure I was able to find is to split the files one way or another.

yrest on 20 Mar 2019

👍1

@yrest, why not? Since the circumstances and any details of the problem scenario involving the OOM exception are unbeknownst to us, it would be rather premature to discard such an advice...

elgonzo on 20 Mar 2019

😕2

I get your point, they should have given more information 😜
Because of having similar exceptions and them occuring on all sorts of hardware, including massive servers, the amount of ram and os in my experience makes no difference.

yrest on 20 Mar 2019

Its a 64 Bit Windows 10 with 24 GB Ram. This should be enouth.
Dotnet Framework is 4.0 Splitting is not possible for me.

I load a 200MB HAR(json) file via the following code in VB.Net:

    Dim RDR As New System.IO.StreamReader(OFD.FileName)
    Dim Json = New Newtonsoft.Json.JsonTextReader(RDR)
    JRoot = JsonConvert.DeserializeObject(RDR.ReadToEnd)

the Node-Stucture (only Stucture) is shown in a Treeview. so i think
this can not cause that much RAM. Complete Ram usage after open this
file exceeds 1GB and a OOM Exception occurs.

If this is to less information feel free to ask!

Phreak87 on 20 Mar 2019

Am facing similar issues with c# on large sets and if somebody can give useful advice would be greatly appreciated!

yrest on 20 Mar 2019

What version of Newtonsoft.Json are you using? There is a memory leak in 12.0.1.

If you're not already, you could try using 11.0.2.

JamesNK on 21 Mar 2019

👍2

Version is 12.0.1.22727. later i will try to change to 11.0.2.
Maybe it would be better the library saves the structure of the
file in memory and not the complete contents.
If access to the content is required the structure can save the exact
position in the stream and return the content from file instead of mem.

in my case (and most HAR cases) the content contains Base64 strings
with more then 5MB.

... Update 1 ...
Version 11.0.2 has the same problems with the OOM as 12.0.1 Version.
Memory for a 230MB File is at the point of the OOM 1450MB.

Phreak87 on 21 Mar 2019

It should be noted that, in the following code, you are not actually reading directly from a stream:

Dim RDR As New System.IO.StreamReader(OFD.FileName)
Dim Json = New Newtonsoft.Json.JsonTextReader(RDR)
JRoot = JsonConvert.DeserializeObject(RDR.ReadToEnd)

To deserialize directly from a stream in VB.NET, you would create a method like:

Public Module JsonExtensions
    Public Function LoadFromFile(Of T)(path As String, Optional ByVal settings as JsonSerializerSettings = Nothing) As T
        Using reader As New System.IO.StreamReader(path)
            Using jsonReader as New Newtonsoft.Json.JsonTextReader(reader)
                Return JsonSerializer.CreateDefault(settings).Deserialize(Of T)(jsonReader)
            End Using
        End Using
    End Function
End Module

And then use it like:

Dim JRoot as JToken = JsonExtensions.LoadFromFile(Of JToken)(OFD.FileName)

The fact that you're not actually streaming may account for some of your problem.

Demo fiddle: https://dotnetfiddle.net/ClyNbD

dbc2 on 22 Mar 2019

👍3

What version of Newtonsoft.Json are you using? There is a memory leak in 12.0.1.

If you're not already, you could try using 11.0.2.

That's this issue, right?

That memory leak only occurs when using StringEnumConverter, doesn't it? OP is deserializing to an untyped object (i.e. JToken) using JsonConvert.DeserializeObject(JsonString) so I don't think it would apply.

dbc2 on 22 Mar 2019

Or change parser ;-) My C# translation of Gason has 10x better speed comparing to this one also use much less memory - 1/4GB JSON in 3-4s, 950MB, this one ~30s/5-7GB RAM... Or you can deserialize on the fly it is also quite slow, but consume around 14MB memory.
https://github.com/eltomjan/Gason
Example here:
https://stackoverflow.com/questions/55372425/way-to-read-or-edit-big-json-from-to-stream