Nlog: JsonLayout not writes in an array to log file

[grifin85]

Feature request for NLog version: 4.2.3

My config

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  autoReload="true">
  <variable name="logDirectory" value="${basedir}/App_Data/Logs" />
  <extensions>
    <add assembly="NLog.Xml"/>
  </extensions>
  <targets>
    <target name="jsonFile" xsi:type="File" fileName="${logDirectory}/${shortdate}_log.json">
      <layout xsi:type="JsonLayout">
        <attribute name="time" layout="${date:format=HH\:mm\:ss}" />
        <attribute name="level" layout="${level}" />
        <attribute name="message" layout="${exception:format=Message}" />
        <attribute name="stackTrace" layout="${exception:format=StackTrace}" />
      </layout>
    </target>
  </targets>
  <rules>
    <logger name="*" minlevel="Info" writeTo="jsonFile" />
  </rules>
</nlog>

And get in file records

{ "time": "17:34:13", "level": "Error", "message": "Test Exeption", "stackTrace": "" }
{ "time": "17:34:15", "level": "Error", "message": "Test Exeption", "stackTrace": "" }

But should

[
{ "time": "17:34:13", "level": "Error", "message": "Test Exeption", "stackTrace": "" },
{ "time": "17:34:15", "level": "Error", "message": "Test Exeption", "stackTrace": "" }
]

[304NotModified]

If we write (without batches) 100 records, do we have 1 or 100 arrays?

[grifin85]

Maybe I something do not understand. But how I can deserialize data if he have incorrect? Maybe you should add setting in config for JsonLayout?

[304NotModified]

Well the problem is when nothing is buffered, all the records will be arrays (as we don't know when logging is done).

With combining with the compound layout you could do that.

You could parse the current result by parsing every row one by one.

Maybe we have to thing to insert to files instead of append (but could be performance tricky), the same for xml

[grifin85]

At the moment JsonLayout is useless. And if we have 100 rows and makes 100 deserializate it was bad for performance.

[304NotModified]

you could add the comma with the CompoundLayout and the [ and ] with the header/footer. See File target docs

[TheXenocide]

I agree the header and footer with a wrapper appending commas would probably work well enough. Another option, while reading the file, is to use something like JSON.NET's JsonTextReader with SupportsMultipleContent set to true. Here is an example excerpt from a prototyping project I've been using in LINQPad (.Dump() is an extension method that displays the output)

public static class JsonExtensions
{
    public static IEnumerable<JObject> ReadJsonStream(this TextReader reader)
    {
        using (JsonTextReader jsonReader = new JsonTextReader(reader))
        {
            jsonReader.SupportMultipleContent = true;

            while (jsonReader.Read())
            {
                yield return JObject.Load(jsonReader);
            }
        }
    }
}
void Main()
{
    // JSON Stream Test
    using (var reader = new StringReader(@"{""key"":""value"", ""another"":""prop\r\nerty""}" + "\r\n" + @"{""Test"":""Val""}{""Same"":""Line""}"))
    {
        reader.ReadJsonStream().Dump();
    }
}

[joeherwig]

This https://github.com/NLog/NLog/wiki/CompoundLayout - workoaround still doens't work that easy.

It produces in best case something like
[ {}, {}, {}, ] or
[ ,{} ,{} ,{} ]
which is still some json-ish looking text but no valid JSON.

So we'd need to determin either the first log-entry into the file and skip a leading comma from the compound layout ,{} or NLog would have to remove the last comma from {}, before saving the file.
For me it seems, that the only way for proper json logging is by

either

1. fetching the files content into an array of objects,
2. appending the array with the new object
3. overwrite the previously fetched file with this extended valid json array of objects.

or log as separate datasets into a db instead of a file - at least if you anyway have a db for the application available. Probably the best solution in that case. :-)

Well, as long as we're taking about appending text-files instead of thinking in objects and arrays this json-array-extending problem won't be solved.

[304NotModified]

@joeherwig which version of NLog did you use? Please try in Nlog 4.5 RC2

[tpilat]

I'm using Nlog 4.5.0-rc04 and always the same result. Json file is invalid [ {}, {}, {}, ]
Is there any other solution?

[304NotModified]

I'm using Nlog 4.5.0-rc04 and always the same result. Json file is invalid [ {}, {}, {}, ]
Is there any other solution?

Unfortunately not yet. It's now a list of JSON fragments, and not a JSON file.

It's hard to create a JSON while streaming the log events to it. (because of the closing ]). I need some ideas to fix this :)

[tpilat]

One of the simplest solution is configure NLog.Layouts.CompoundLayout to concat NLog.Layouts.JsonLayout and NLog.Layouts.SimpleLayout(","). Then set Header to "[" and the Footer to "{}]" :)

Then the log file will be closed correctly and the result is valid json.
[
{ "level": "error", "message": "msg1"},
{ "level": "info", "message": "msg2"},
{}]

[WahidBitar]

One of the simplest solution is configure NLog.Layouts.CompoundLayout to concat NLog.Layouts.JsonLayout and NLog.Layouts.SimpleLayout(","). Then set Header to "[" and the Footer to "{}]" :)

Then the log file will be closed correctly and the result is valid json.
[
{ "level": "error", "message": "msg1"},
{ "level": "info", "message": "msg2"},
{}]

Thanks @tpilat
We did the same and it worked good. But we want to have the footer at any time we open the file. I mean NLog will write the footer just when it will close the writing to the file. how could we have a valid file whenever we read it?

[TheXenocide]

@WahidBitar what you're requesting requires non-append behavior which, even if it were implemented in NLog (I don't believe it is) causes a significant performance overhead. I'm not sure what you're using to view the files, but I have previously used a JSON library capable of reading multiple separate JSON objects from a stream, which I believe is now also supported by JSON.NET. Does the solution here work for you? https://stackoverflow.com/questions/18805735/deserialize-multiple-objects-from-stream-by-json-net

[TheXenocide]

Note that this suggestion also does not require the above CompoundLayout wrapper (using it would require special handling of the header and commas if you try to use the streaming approach and the CompoundLayout at the same time)

[304NotModified]

what you're requesting requires non-append behavior which, even if it were implemented in NLog (I don't believe it is)

Indeed, that's not implemented. Injection in the file is expensive (and tricky for multitrheading). It's not implemented.

We use it like this -> write partial json (NLog) -> translate to real JSON ([] before and ] after) (not NLog) -> process/view

[asartem]

Hello!
Are there any updates about this bug? Because now the Json Layout is still useless. Myabe NLog writer should check and form JSON correctly?
The fix above with header and CompoundLayout is a non-working workaround.
For example there could be 2 logged messages separated in time. They will look like this:

The first try to log:
[ {"time": 00:00:00, "message":"log 1"},
{"time": 00:01:00, "message":"log 2"},
{} ]

Second try to write to same file later:

[ {"time": 00:00:00, "message":"log 1"},
{"time": 00:01:00, "message":"log 2"},
{} ] <---- footer. And there is no header for new logs:
{"time": 00:10:00, "message":"log 3"},
{"time": 00:11:00, "message":"log 4"},

and the json is broken for log 3 and log 4 and whole file. Header will be not applied as well as footer for second log.
The expected behaviour: {}] should be removed. log 3 and log 4 should be added to the end. ANd the {}] should be added after.
Please, help.

[snakefoot]

@asartem Just like NLog will not be able to write a proper xml-file-output (with correct root-node), then NLog will also not be able to write a proper json-file-output.

Instead you have fix the log-viewer tool (to handle missing begin and end brackets), or have a tool to fix the log-file (insert missing begin and end brackets) before loading it into the log-viewer-tool.

Ofcourse if you have an elegant fix for how to improve this, then PullRequests are always welcome.

[TheXenocide]

Many JSON reading libraries can read objects from streams where you can just loop and read one object at a time, as opposed to reading the whole file as an array. This is especially helpful on any larger log file since the memory usage of loading a 2GB string from file and then parsing it is rather expensive. As long as your JSON isn't pretty printed and only takes up a single line you can use the following with Newtonsoft.Json.

// creating sample data
var ms = new MemoryStream();
using (var writer = new StreamWriter(ms, Encoding.UTF8, 4096, true))
{
    writer.WriteLine(@"{ ""id"":1, ""prop1"":""value 1"" }");
    writer.WriteLine(@"{ ""id"":2, ""prop1"":""value 2"" }");
}

ms.Seek(0, SeekOrigin.Begin);
// now imagine this memory stream is actually a file stream, just opened

using (var reader = new StreamReader(ms, Encoding.UTF8))
{
    string line;
    while ((line = reader.ReadLine()) != null) // read all lines until end of file
    {
        JsonConvert.DeserializeObject(line).Dump();
    }
}

Dump is from the tool called LINQPad, but you could also yield return or something from that part once you have your deserialized object. This is a very basic example.

If the JSON data takes up more than one line (usually only from pretty printing) you can find a JSON library that supports reading multiple objects from the same stream (I found one of these at one point, but don't use it in my current log parsing solution so don't remember what it was).

[snakefoot]

I guess one could create a PullRequest that adds a new option to re-write the Footer after each file write-operation (and not just write Footer on file-close). Special logic has to be implemented so it correctly overwrites the existing Footer when appending to an existing file.

If the code is not too gnarly then I think a pull-request will be accepted.