Nlog: OutOfMemoryException causing the window service to terminate abnormally

[amanpreetSO]

NLog version: (4.5.6)

Platform: .Net 4.8

Current NLog config

<nlog>
  <targets>
    <target xsi:type="AsyncWrapper" name="MyWebLogger" overflowAction="Grow">
    <target xsi:type="WebService" name="logger"
            protocol="JsonPost"
            encoding="UTF-8">
         <parameter name="">
            <layout xsi:type="JsonLayout" includeAllProperties="false">
                <attribute name="level" layout="${level:lowercase=true}" />
                <attribute name="message" layout="${message}" />
                <attribute name="index" layout="${event-properties:item=index}" />
                <attribute name="meta" encode="false">
                    <layout xsi:type="JsonLayout" includeAllProperties="true" excludeProperties="index"/>
                </attribute>
            </layout>
        </parameter>
    </target>
</target>
  </targets>
  <rules>
    <logger name="ElasticLogger" level="Info" writeTo="MyWebLogger" />
    <logger name="ElasticLogger" level="Error"  writeTo="MyWebLogger" />
  </rules>
</nlog>

• What is the current result?
  I am using Nlog in window service and target is web service. I am facing OutOfMemoryException from Nlog in peak hour that is causing the service to restart. Its restarting the service at least 10 times. To make sure these exception are from Nlog I temporarily stopped the logging then window service was not restarted even a single time for last 10 days.

• What is the expected result?
  it should log without exception

• Did you checked the Internal log?
  No

• Please post full exception details (message, stacktrace, inner exceptions)

Description: The process was terminated due to an unhandled exception.
Exception Info: System.OutOfMemoryException
   at System.IO.MemoryStream.set_Capacity(Int32)
   at System.IO.MemoryStream.EnsureCapacity(Int32)
   at System.IO.MemoryStream.SetLength(Int64)
   at NLog.Internal.StringBuilderExt.CopyToStream(System.Text.StringBuilder, System.IO.MemoryStream, System.Text.Encoding, Char[])
   at NLog.Targets.WebServiceTarget+HttpPostTextFormatterBase.WriteContent(System.IO.MemoryStream, System.Object[])
   at NLog.Targets.WebServiceTarget+HttpPostFormatterBase.PrepareRequest(System.Net.HttpWebRequest, System.Object[])
   at NLog.Targets.WebServiceTarget.DoInvoke(System.Object[], NLog.Common.AsyncContinuation, System.Net.HttpWebRequest, System.Func`2<System.AsyncCallback,System.IAsyncResult>, System.Func`2<System.IAsyncResult,System.IO.Stream>)
   at NLog.Targets.WebServiceTarget.DoInvoke(System.Object[], System.Net.HttpWebRequest, NLog.Common.AsyncContinuation)
   at NLog.Targets.WebServiceTarget.DoInvoke(System.Object[], NLog.Common.AsyncLogEventInfo)
   at NLog.Targets.MethodCallTargetBase.Write(NLog.Common.AsyncLogEventInfo)
   at NLog.Targets.Target.Write(System.Collections.Generic.IList`1<NLog.Common.AsyncLogEventInfo>)
   at NLog.Targets.Target.WriteAsyncThreadSafe(System.Collections.Generic.IList`1<NLog.Common.AsyncLogEventInfo>)
   at NLog.Targets.Target.WriteAsyncLogEvents(System.Collections.Generic.IList`1<NLog.Common.AsyncLogEventInfo>)
   at NLog.Targets.Wrappers.AsyncTargetWrapper.WriteEventsInQueue(Int32, System.String)
   at NLog.Targets.Wrappers.AsyncTargetWrapper.ProcessPendingEvents(System.Object)
   at System.Threading.TimerQueueTimer.CallCallbackInContext(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireQueuedTimerCompletion(System.Object)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

• Are there any workarounds? yes/no
  I have disabled the log because service restart was causing the more harm then capturing the logs.
• Is there a version in which it did work?
  I didn't check because its happening only on production.

[welcome[bot]]

Hi! Thanks for opening your first issue here! Please make sure to follow the issue template - so we could help you better!

[snakefoot]

Think you have reached the maximum performance of the WebServiceTarget. The WebServiceTarget performs a http-request for each LogEvent, so I guess you have the following options:

• Reduce the number of LogEvents from the application.
• Change to a different NLog-Target that can perform batching against ElasticSearch.

Http Batching can probably be implemented with one of these targets:

• https://www.nuget.org/packages/NLog.Targets.ElasticSearch/
• https://www.nuget.org/packages/NLog.Targets.Http/

Yet another alternative is writing to a json-file, and instead have an external application like FileBeat or LogStash to push the file-contents to ElasticSearch.

[amanpreetSO]

Thanks @snakefoot for quick reply

• Reduce the number of LogEvents from the application.

This is not a good solution because as the feature in the application is growing so we need to add the more logs.

Http Batching can probably be implemented with one of these targets:

My target is WebService although batching is also supported by this target https://github.com/NLog/NLog/wiki/AsyncWrapper-target
but I am not sure what would be the optimal values. Can you please help in that ?

[snakefoot]

Yes AsyncWrapper-target can be used to activate batching for synchronous-targets that supports batching (Ex File-Target / Console-Target). But WebServiceTarget is not a synchronous target and will translate one LogEvent into one HttpRequest.

[amanpreetSO]

you mean to say that WebServiceTarget is Async so batching is not supported for this kind of target ?

[snakefoot]

Async and batching is not mutual exclusive. Can only repeat that WebService-target makes a http-request for each logevent, thus it doesn't support batching.

[amanpreetSO]

so what the solution you are proposing? changing the target to http ?

[snakefoot]

Think I have already listed my proposed solutions in my first reply.

[snakefoot]

Closing this issue, as answered. Please re-open if not.

[amanpreetSO]

@snakefoot I tried to use the HTTP target instead of Webservice but I am not getting the logs. Same URL should work with HTTP target as well ?

[snakefoot]

Sorry cannot help you there. Know little about splunk or Http target. Have you checked with fiddler (or some other tool) if the same Http-request is made? Have you checked the NLog InternalLogger output? Have you tried to flush?

Sent from my Sony Xperia

---- Aman Preet Singh wrote ----

@snakefoothttps://github.com/snakefoot I tried to use the HTTP target instead of Webservice but I am not getting the logs. Same URL should work with HTTP target as well ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/NLog/NLog/issues/4149#issuecomment-733683884, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACXZ7HANLEJKESDAQAPU3ELSRT3WTANCNFSM4S7FPWIQ.