Nlog: Configure from JSON instead of XML

[304NotModified]

Feature,

Configure NLog from a JSON format.

todo

How to handle the multiple XML elements to a dict/list?

e.g.

    <parameter name="@msg" layout="${message}" />
    <parameter name="@level" layout="${level}" />

to

   "parameters": {
        "@msg": "${message}",
        "@level": "${level}",
      },

"XSD"

How to generate the schema def? How does this work with objects?

Large Example

Something like this:

{
  "internalLog": {
    "level": "Trace",
    "file": "d:\\work\\log.txt"
  },
  "autoReload": "true",
  "variables": {
    "gmailUsername": "${trim-whitespace:${file-contents:${basedir}/gmailusername.txt}}",
    "gmailPassword": "${trim-whitespace:${file-contents:${basedir}/gmailpassword.txt}}"
  },
  "extensions": {
    "NLog.Extended": {},
    "Custom": {
      "filePath": "..."
    }
  },
  "targets": {
    "file": {
      "type": "File",
      "fileName": "${basedir}/logs/${shortdate}.log",
      "layout": "${longdate} ${aspnet-request:servervariable=URL} ${uppercase:${level}} ${message}"
    },
    "db": {
      "type": "Database",
      "commandText": "INSERT INTO [LogEntries](TimeStamp, Message, Level, Logger) VALUES(getutcdate(), @msg, @level, @logger)",
      //todo how to rename? (was no list/dict here)
      "parameters": {
        "@msg": "${message}",
        "@level": "${level}",
        "@logger": "${logger}"
      },
      "dbProvider": "System.Data.SqlClient",
      "connectionString": "server=.\\SQLEXPRESS;database=MyLogs;integrated security=sspi",
      "install-commands": [
        {
          "text": "CREATE DATABASE MyLogs",
          "connectionString": "server=.\\SQLEXPRESS;database=master;integrated security=sspi",
          "ignoreFailures": "true"
        },
        {
          "text": "CREATE TABLE LogEntries( id int primary key not null identity(1,1), TimeStamp datetime2, Message nvarchar(max), level nvarchar(10), logger nvarchar(128))"
        }
      ],
      "uninstall-commands": [
        {
          "text": "DROP DATABASE MyLogs",
          "connectionString": "server=.\\SQLEXPRESS;database=master;integrated security=sspi",
          "ignoreFailures": "true"
        }
      ]
    },
    "eventLog": {
      "type": "EventLog",
      "source": "NLog Demo",
      "layout": "${message}${newline}Call site: ${callsite:className=true:methodName=true}${newline}Logger: ${logger}"
    },
    "pc1": {
      "type": "PerfCounter",
      "categoryName": "My Log",
      "counterName": "My Counter"
    },
    "mail": {
      "type": "Mail",
      "smtpServer": "smtp.gmail.com",
      "smtpPort": "587",
      "enableSsl": "true",
      "smtpAuthentication": "Basic",
      "smtpUserName": "${gmailUsername}",
      "smtpPassword": "${gmailPassword}",
      "from": "${gmailUsername}",
      "to": "${gmailUsername}",
      "subject": "NLogDemo logs",
      "addNewLines": "true"
    },
    "asyncMail": {
      "type": "AsyncWrapper",
      "wraps": [ "mail" ]

    },
    //no anounous wrappers
    "postFilteringWrapper": {
      "type": "PostFilteringWrapper",
      "wraps": [ "splitGroup" ],
      "defaultFilter": "level >= LogLevel.Info",
      //todo
      "when": {
        "exists": "level >= LogLevel.Error",
        "filter": "level >= LogLevel.Trace"
      }
    },
    "splitGroup": {
      "type": "SplitGroup",
      "wraps": [
        "file",
        "db",
        "eventLog",
        "pc1",
        "asyncMail"
      ]
    },
    "allOutputs": {
      "type": "AspNetBufferingWrapper",
      "wraps": [ "PostFilteringWrapper" ]
    }

  },
  "rules": [
    {
      "filter": {
        "loggerName": "*" //optional,
        "minLevel": "Trace"
      },
      "writeTo": "allOutputs"
    }
  ]
}

changes:

• more flat, all targets must have names
• internallog to object

95% conversion from https://github.com/NLog/NLog/wiki/NLog-config-Example

More generic feature: https://github.com/NLog/NLog/issues/951

[jods4]

This is getting more interesting with .NET Core projects, as people (like me!) probably prefer to merge all their configs into a single file appsettings.json rather having several config files in various formats *.xml floating around.

That said the use case would not be as much JSON as it would be ConfigSection, which is similar but slightly different. 😄
It is also much more interesting as you can define most settings in your config file, and then just override the exact file location in an environment specific key override!

[304NotModified]

Thanks for the feedback!

his is getting more interesting with .NET Core projects, as people (like me!) probably prefer to merge all their configs into a single file appsettings.json rather having several config files in various formats *.xml floating around.

How it that working with auto-reload?

It is also much more interesting as you can define most settings in your config file, and then just override the exact file location in an environment specific key override!

You can do that now with the ${appsettings} ;) (or using <include)

[jods4]

@304NotModified

How it that working with auto-reload?

Honestly I have no idea, as my first app with ASP.NET core doesn't support config auto-reload. Full app restart was a requirement.

You can do that now

This is interesting! Can you point to the specific doc please?
Slight drawback is that this means thinking every extension point in advance + coming up with appsettings key for them, while true .NET Core Configuration support would mean _any_ aspect could be changed for free: log level, file retention, file location, adding a console trace, etc.

BTW, have you looked at how Serilog supports both API config + old appsettings xml + new Configuration ?

[304NotModified]

https://github.com/nlog/nlog/wiki/Configuration-file#include-files

you can include and overwrite <variable>

or using https://github.com/nlog/nlog/wiki/AppSetting-Layout-Renderer with a default.

I hope this is what you we're looking for. Not sure ;)

[jods4]

Thanks for the info!
It's close but not _exactly_ what we need.

Basically, yes we want to relocate the log file on the production server, so ${logFolder}\log.txt is perfect but...

But using ASP.NET Core we already have a two layers config system with all the common settings deployed by our CI and env. specific config available from the machine. We don't really want to have a special case of included nlog config files just for this variable. There is value in having _everything_ handled the same way.

We currently treat logFolder like all our other env. settings, but we had to write some very specific code to plug it into the NLog config where we want to.

But don't worry with that. It works and all of this will be solved when NLog supports configuring from other sources than xml, especially the Configuration object in .net core.

[phenixdotnet]

Hi.
Autoreleoad in .NET core app work if it's configured at startup:

var builder = new ConfigurationBuilder()
                .SetBasePath(env.ContentRootPath)
                .AddJsonFile("appsettings.json", optional: true, **reloadOnChange**: true)
                .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
                .AddEnvironmentVariables();
            Configuration = builder.Build();

So if NLog config is put in appsetting.json any changes done in this file will trigger the reload of the config object.
As part of this configuration (which is from the default .NET core web app template) you should notice changing the appsettings.ENV.json file doesn't trigger autoreload

I didn't look at how autoreload is done in NLog but I think we will have some work to do to get it working in .NET Core

[304NotModified]

Any idea of the apppool gets a reset ? (Like changing web.config now)

[jods4]

@304NotModified No, the main idea behind this is to be able to change config values _without_ a restart.

My understanding of this (but I'm not using it so...):

• In RTM 1.0, reloadOnChange doest work but its only effect is to update your untyped IConfigurationRoot.
• In RC2, trackConfigChanges and IOptionsMonitor<> could be used to refresh strongly-typed options but they were removed of 1.0 and are expected to make a come back in a future release.

This is the best doc about this topic that I've found:
http://andrewlock.net/reloading-strongly-typed-options-when-appsettings-change-in-asp-net-core-rc2/

[304NotModified]

Thanks!

I'm Also curious after feedback on the json example. Is it good? Is it better then the XML? What could be better?

[jods4]

@304NotModified I dug a concrete example of config reload in action. The built-in logger abstraction does use that. You can have a look at Microsoft.Extensions.Logging.Console for a trivial example.

Basically it comes down to:

• Always read the config from IConfiguration because this is reloaded and typed objects are not supported in RTM.
• Ask IConfiguration for a reload token: ChangeToken = configuration.GetReloadToken();
• Register a method that you want to be called when the config changes: _settings.ChangeToken.RegisterChangeCallback(OnConfigurationReload, null);
• Then do whatever you have to do in OnConfigurationReload, see: https://github.com/aspnet/Logging/blob/a63a9d05c1938e50f376e316f47c2a44f6c9851f/src/Microsoft.Extensions.Logging.Console/ConsoleLoggerProvider.cs#L46

Note that the token is one-shot, you need a new one after the config changed.

[304NotModified]

Thanks that's very helpful!

[luoxy123]

It's Easy To define a wrapper in the configuration file with xml formatting,but not with Json formatting, Because the warpper is flexible. Is there a better way to define a warpper with Json formatting?

[304NotModified]

What do you mean? In the proposed json there is a wrap property.

[luoxy123]

I mean that how to define a wrap property in the proposed json , such as asynchronous wrap

[304NotModified]

@Jamesxql commented 3 days ago:

I think the Database target can define like this:

{
    "targets":{
        "db": {
            "type": "Database",
            "commandText": "INSERT INTO [LogEntries](TimeStamp, Message, Level, Logger) VALUES(getutcdate(), @msg, @level, @logger)",
            "parameters": [
                "@msg": {
                    "layout": "${message}"
                },
                "@level": {
                    "layout": "${level}",
                    "size": 0
                },
                "@logger": {
                    "layout": "${logger}"
                }
            ],
            "dbProvider": "System.Data.SqlClient",
            "connectionString": "....."
        }
    }
}

The default wrapper definition is:

{
    "targets":{
        "defaultWrapper": {
            "type": "BufferingWrapper",
            "bufferSize": 100
        },
        "f1": {
            type: "File",
            fileName: "f1.txt"
        },
        "f2": {
            type: "File",
            fileName: "f2.txt"
        }
    },
    "targets": {
        "defaultWrapper": {
            type: "AsyncWrapper",
            wrap: {
                "type": "RetryingWrapper"
            }
        },
        "n1": {
            "type": "Network",
            "address": "tcp://localhost:4001"
        },
        "n2": {
            "type": "Network",
            "address": "tcp://localhost:4002"
        },
        "n3": {
            "type": "Network",
            "address": "tcp://localhost:4003"
        }
    }

The default target parameters definition is:

{
    "targets": {
        "defaultTargetParameters": [
            {
                "type": "File",
                "parameters": {
                    "p1": "k1",
                    "p2": "k2"
                }
            },
            {
                "type": "Database",
                "parameters": {
                    "p1": "k1",
                    "p2": "k2"
                }
            }
        ],
        "f1": {
            "type": "File",
            "fileName": "f1.txt"
        },
        "db": {
            "type": "Database",
            ...
        }
    }
}

The problem is: in JSON format, how do we handle the hierarchical relationship of the target wrapper?
What do you think? @304NotModified

@Jamesxql I think wrapping is a bad design choice in JSON, so therefor I have proposed the "wrap" property

[YaoNiGu]

is it can work now?
do i need to install something?

thank you!

[gotoxu]

I'm working on it. But I am more busy recently, so the progress will be slower

[wangsu]

any update for this issue?~

[304NotModified]

No changes from my side. Working hard on getting NLog 4.5 RTM

If you like to help, please do

[snakefoot]

For easy access to a Json-parser that can read entire sections, then this could be a separate nuget-package with a dependency on:

<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="1.0.0" />

Then create a class called NLog.ExtensionLoggingConfiguration with a constructor that takes IConfigurationSection as input. Then one can use IConfigurationSection.GetSection to access sub-sections.

[snakefoot]

I guess rules should have id (or name) like the targets, which will fit well with #2394

[304NotModified]

I guess rules should have id (or name) like the targets, which will fit well with #2394

as an optional property, of the json list as json dictionary?

[snakefoot]

In your initial example then the id of first logging rule will be "filter" ("writeTo" should probably be moved into the scope of "filter")

[snakefoot]

Have modified XmlLoggingConfiguration so it can read configuration from an interface:

```c#
public interface ILoggingConfigurationSection
{
string Name { get; }
IEnumerable> Values { get; }
IEnumerable Children { get; }
}

Then created the following wrapper around `IConfigurationSection` (Microsoft.Extensions.Configuration.Abstractions):

```c#
    public class ExtensionLoggingConfiguration : LoggingConfigurationReader 
    {
        public class LoggingConfigurationSection : ILoggingConfigurationSection
        {
            readonly IConfigurationSection _configurationSection;

            public LoggingConfigurationSection(IConfigurationSection configurationSection)
            {
                _configurationSection = configurationSection;
            }

            public string Name => _configurationSection.Key;

            public IEnumerable<KeyValuePair<string, string>> Values
            {
                get
                {
                    var children = _configurationSection.GetChildren();
                    foreach (var child in children)
                    {
                        if (!child.GetChildren().Any())
                            yield return new KeyValuePair<string, string>(child.Key, child.Value);
                    }
                }
            }

            public IEnumerable<ILoggingConfigurationSection> Children
            {
                get
                {
                    var children = _configurationSection.GetChildren();
                    foreach (var child in children)
                    {
                        if (child.GetChildren().Any())
                        {
                            yield return new LoggingConfigurationSection(child);
                        }
                    }
                }
            }
        }

        public ExtensionLoggingConfiguration(IConfigurationSection nlogConfig, string basePath = null)
            :base(NLog.LogManager.LogFactory)
        {
            LoadConfig(new LoggingConfigurationSection(nlogConfig), basePath);
        }
    }

So it can be loaded like this:

```c#
ConfigurationBuilder builder = new ConfigurationBuilder();
builder.AddJsonFile("appsettings.json");
var configRoot = builder.Build();

        var config = new ExtensionLoggingConfiguration(configRoot.GetSection("nlog"));
        LogManager.Configuration = config;

This is an example of the json-file:

```json
{
    "NLog": {
        "internalLogLevel": "Trace",
        "internalLogFile": "C:/Temp/internalLog.txt",
        "throwConfigExceptions": "True",
        "variables": {
            "gmailUsername": "${trim-whitespace:${file-contents:${basedir}/gmailusername.txt}}",
            "gmailPassword": "${trim-whitespace:${file-contents:${basedir}/gmailpassword.txt}}"
        },
        "extensions": [
            {
                "assembly": "NLog.Extended"
            }
        ],
        "targets": {
            "file": {
                "type": "File",
                "fileName": "${basedir}/logs/${shortdate}.log",
                "layout": "${longdate} ${uppercase:${level}} ${message}"
            },
            "db": {
                "type": "Database",
                "commandText": "INSERT INTO [LogEntries](TimeStamp, Message, Level, Logger) VALUES(getutcdate(), @msg, @level, @logger)",
                "parameters": [
                    {
                        "name": "@msg",
                        "layout": "${message}"
                    },
                    {
                        "name": "@level",
                        "layout": "${level}"
                    },
                    {
                        "name": "@logger",
                        "layout": "${logger}"
                    },
                ],
                "dbProvider": "System.Data.SqlClient",
                "connectionString": "server=.\\SQLEXPRESS;database=MyLogs;integrated security=sspi",
            },
            "postFilteringWrapper": {
                "type": "PostFilteringWrapper",
                "target": {
                    "file": {
                        "type": "File",
                        "fileName": "${basedir}/logs/${shortdate}.log",
                        "layout": "${longdate} ${uppercase:${level}} ${message}"
                    }
                },
                "defaultFilter": "level >= LogLevel.Info",
                "when": {
                    "exists": "level >= LogLevel.Error",
                    "filter": "level >= LogLevel.Trace"
                }
            },
            "splitGroup": {
                "type": "SplitGroup",
                "targets": {
                    "file1": {
                        "type": "File",
                        "fileName": "${basedir}/logs/1-${shortdate}.log",
                        "layout": "${longdate} ${uppercase:${level}} ${message}"
                    },
                    "file2": {
                        "type": "File",
                        "fileName": "${basedir}/logs/2-${shortdate}.log",
                        "layout": "${longdate} ${uppercase:${level}} ${message}"
                    },
                },
            },
        },
        "rules": [
            {
                "logger": "*",
                "minLevel": "Trace",
                "writeTo": "SplitGroup"
            },
            {
                "logger": "*",
                "minLevel": "Trace",
                "writeTo": "postFilteringWrapper"
            },
        ]
    }
}

IConfigurationSection is a little special as the sections/children are sorted, so the order of sections in the json-file is ignored. This means things like variables has to be pre-loaded (just like extensions). There is also no support for autoReload or including external json-files.

[304NotModified]

IConfigurationSection is a little special as the sections/children are sorted

I was thinking about this. The JSON specs tell us:

An object is an unordered set of name/value pairs

(from https://json.org)

so we need maybe a list for the variables, if the order is important (in think it is in this case because of recursion)

[snakefoot]

Yes Json spec doesn't require ordering. But the Microsoft Json Config enforces ordering. You are welcome to create your own Json config parser with your own rules.

The ordering issue comes when having multiple variable and targets sections (something that is possible with xml config)

[304NotModified]

Yes Json spec doesn't require ordering.

The JSON spec specifies that objects are unordered and lists are ordered? Then the MS parser is correct?

Or am I missing something?

[snakefoot]

Yes lists seems to be the way forward if wanting to preserve order and allow multiple targets-sections within the nlog-section

[PureKrome]

Just touching base on this issue -> what's the current status of Json NLog files? It's it currently the default norm? I'm not seeing any documentation that suggests this as such?

[snakefoot]

@PureKrome I'm waiting for NLog 4.6 to be released that includes #2891.

Then I will make a PR for https://github.com/NLog/NLog.Extensions.Logging with support for loading JSON from Microsoft.Extension.Configuration

[PureKrome]

Great news @snakefoot ! I'll be eagerly watching this space :)

[snakefoot]

@PureKrome Any input you have for how to extend the AddNLog-methods to provide access to IConfiguration in NLog.Extension.Logging is welcome. See also https://github.com/NLog/NLog.Extensions.Logging/pull/248 (Trying to inject IConfiguration so the new ${configsetting} will work automatically from fluent configuration (Later the NLog-LoggingConfiguration can be loaded from the IConfiguration-interface).

My idea is that the UseNLog() method in NLog.AspNetCore will setup everything in a default way (Attempt to load NLog-config if available from IConfiguration etc.). But one can also setup NLog using the AddNLog-methods in NLog.Extensions.Logging if having special requirements.

[snakefoot]

Created https://github.com/NLog/NLog.Extensions.Logging/pull/263 for people who would like to test NLog-config from appsettings.json. Depends on NLog 4.6-rc1

[snakefoot]

Nuget-package is now available, for pre-release preview:

https://www.nuget.org/packages/NLog.Extensions.Logging/1.5.0-rc1

Still missing NLog variables and some of the more exotic stuff like default-wrapper and default-target-parameters. XML config has lots of strange features. So trying to focus on the important ones.

[snakefoot]

This issue can now be resolved with the release of NLog.Extension.Logging ver. 1.5.0:

https://www.nuget.org/packages/NLog.Extensions.Logging

Wiki: https://github.com/NLog/NLog.Extensions.Logging/wiki/Json-NLog-Config

[snakefoot]

Closing this issue as resolved with NLog.Extension.Logging ver. 1.5.0 (see comment above)