Loki: Windows EventLog support

[tdabasinskas]

Is your feature request related to a problem? Please describe.
Windows logs are stored in Event Log (.evtx files), which currently not possible to scrape it via currently available promtail methods.

Describe the solution you'd like
Since we do have systemd journal support for Linux, it would be nice to have support for Event Log on Windows in a similar matter.

Describe alternatives you've considered
Key part of the solution is actually being able to parse the logs. If I haven't missed anything, it seems that there are currently two Golang modules that can do that: github.com/0xrawsec/golang-evtx and github.com/elastic/beats/winlogbeat/eventlog.

[randomchance]

This would instantly make Loki viable in my environment - fluentd requires ruby, which is no-go, but having a single go executable would be perfect.

Alternatively, being able to accept data from winlogbeat (_or beats in general!_) and run it through the pipline would be amazing!

[PWSys]

+1 I have many Windows systems in my environment.
randomchance's suggestion of leveraging winlogbeat would work as well!

[steenstra]

This would be great!

I'm currently using InfluxDB/Telegraf as Syslog receiver with NXlog (https://nxlog.co/) to convert Windows Event logs to Syslog, using the im_msvistalog module.

I see that Promtail can be used as Syslog target (https://github.com/grafana/loki/blob/master/docs/clients/promtail/scraping.md#syslog-target), so maybe something like that would be a temporary solution until this is implemented?

[cosmo0920]

Another alternative is using Fluentd's Windows EventLog plugin.

Fluentd ecosystem has fluent-plugin-windows-eventlog's in_windows_eventlog2 plugin which can consume .evtx format Windows EventLog.
But this workaround requires Ruby as randomchance's suggested.

[pomazanbohdan]

receiver with NXlog (https://nxlog.co/) to convert Windows Event logs to Syslog

If you convert Windows events to a log, then it can already be sent to loki via promtail

[randomchance]

Just wanted to let interested parties know - winlogbeat (and all the elastic beats) can be configured to output to rolling files instead of logstash, so you can scrape them with promtail!

[cyriltovena]

We're going to add logstash soon, this is nice to know for windows users.

[azawawi]

I am currently working on this one (i.e a golang prototype to get windows event logs directly to promtail). So far the executable size is ~2.5MB. Your feedback is appreciated for the following:

• Output:

  it is writing to win-event.log in the current directory.

• Log Name:

  It supports the following log names:

  • Application
  • System
  • Security (requires admin privileges)
  • Setup (requires admin privileges)
  • "Forwarded Events". Please note that this is actually missing in the fluentd plugin implementation.

• Position:

  Should we take Source + EventID + Timestamp to provide tailing? We can read all the logs and we can modify the log query to show events after last read timestamp.

• Output type / Size:

  On my development machine I have like 8790 log entries. It finishes writing them in less than 3 seconds.

  Windows Event Log renders its event as XML output and fluentd has an option to convert it to JSON. What is the most useful output (XML or JSON) to focus on? I have implemented both so far.

  For ~9K log entries on my windows machine:

  • XML (native): Size is a bit more but it is a bit faster since there is no JSON conversion
  • JSON: is like 392K lines (~7 MB and when prettified ~9 MB)

• Testing:
  Using the following powershell commandlet to test:

  Write-EventLog -LogName Application -Source "Windows Error Reporting" -EntryType Information -EventId 1 -Message "This is a Test Message"
  

Kindly provide example production workload numbers. Your feedback is appreciated :smile:

[azawawi]

And here is the result so far (Promtail / Loki / Windows Event Log prototype on Windows 10):

Please note there is a usability bug in Grafana's explore / query. To get past the error parse error at line 1, col 11: invalid char escape, you need to escape \ with \\.

[randomchance]

That's awesome, thanks so much for working on this!

As for feedback:

• It supports the following log names:

I think this will need to be able to support arbitrary log names, just try all the configured ones and ignore ones that fail. It's pretty standard practice for enterprise applications to create their own logs that need to be monitored, even ignoring all of the other Microsoft ones. Here are a just a few of the ones I would need to monitor:

• Veeam Agent
• Windows PowerShell
• Microsoft-Windows-DSC/Admin
• Microsoft-Windows-DSC/Operational
• Microsoft-Windows-DiskDiagnostic/Operational
• Microsoft-Windows-Hyper-V-VmSwitch-Operational
• Microsoft-Windows-Kerberos/Operational

You can see the list of ones available in powershell by running:

Get-WinEvent -ListLog *

My desktop has 510 logs registered!

Should we take Source + EventID + Timestamp to provide tailing?

As for tailing, I would think that using logname + source+timestamp or even just logname+timestamp would be ideal, the Event ID's are often repeated or reused, especially by non-Microsoft sources.

it is writing to win-event.log

This would need to be rotated and cleaned up - my personal favorite method to rotate logs that are being consumed is to include the date in the file name, such as always writing to logname.{day}{month}{year}.log and have a configured limit on the number of files retained - this way you are not changing the names of files promtail should read, and you can easily say "Keep 30 days of data". Also, if you don't want to handle cleaning the old files up it's easier for me to have a script that just deletes files that match a pattern with a LastWriteTime older than X days.

in the current directory

This would need to be configurable. The executable will generally be in the program files directory which will require admin access to write to, and for some of our servers I would want to move it to another drive for space considerations - the logs easily reach a couple of gigs a month, and we need to retain them for regulatory reasons.

Thanks again, and I'm happy to provide feedback!

[cyriltovena]

From what I understand you're listening to event and writing them to a temporary files. This is nice, although alternatively later we could read directly from promtail with a new windows target.

Really thank you for contributing this is awesome ! and @azawawi join #loki-dev if you need anything.

[azawawi]

@randomchance Thanks for the useful info and feedback. I really appreciate it.

@cyriltovena Yes that sums it up. So from I understand, I need to add a new windows-only build target (i.e. wineventtarget.go, wineventmanager.go and winevent_test.go) inside the targets folder to implement it. Anything else I missed?

[randomchance]

@azawawi If it's possible, I agree with @cyriltovena that it would be better to send directly to promtail, technically you can already use Winlogbeat to write eventlogs directly to files.

Thanks again!

[randomchance]

@azawawi You asked for example production workload numbers, so I got some for you!

I just checked one of our installations were we batch logs for retention.

| Interval | Entries | Size
| -------- | ------- | -----
1 Hour (Busy) | 140966 | 69.1 Mb
1 Hour (Calm) | 9236 | 6.5 Mb

... and I just realized this is an aggregate number so not super applicable, but maybe if you divide it by the 9 servers?

[cyriltovena]

@cyriltovena Yes that sums it up. So from I understand, I need to add a new windows-only build target (i.e. wineventtarget.go, wineventmanager.go and winevent_test.go) inside the targets folder to implement it. Anything else I missed?

No that’s the idea you got it right, we can help you of course along the way.

[randomchance]

Since the eventlog API supports xpath queries, I think that would be a good low hanging fruit for any solution.

@azawawi I did some digging into how DotNet handles persisting the last read location in the EventLog stream. DotNet has an EventBookMark class that you use, but under the hood that is just storing the channel and RecordID.

This means that if you use an XPath query, you can filter it with something like:
Event[System[EventRecordID > 83005]]

Where 83005 is the RecordID of the last record stored.

This query gets records after record 83005 and older than 86400000 milliseconds
Event[System[EventRecordID > 83005 and TimeCreated[timediff(@SystemTime) <= 86400000]]]

Or if you want to get the specific event you left off on:
Event[System[EventRecordID = 83005]]

I think it would be smart to look for the single event first, and if it's no longer there you can assume the the log has dropped it and just start at the beginning, or also store the timestamp and fall back to it.

I don't know if that's helpful, but I was looking into writing something similar in DotNet Core and this was stumping me for a while.

You can test the filters/queries in PowerShell pretty easily:

$query = "Event[System[EventRecordID > 83005  and TimeCreated[timediff(@SystemTime) <= 86400000]]]"
Get-WinEvent -FilterXPath $query -LogName System

[Jacq]

I hope to hear more about this, I am currently using winlogbeat with elasticsearch, but if the use of loki for these event logs is possible I could remove the elasticsearch instance and save cpu resources.
I could not find a similar solution with telegraf, it might be in the same situation as loki: https://github.com/influxdata/telegraf/issues/4525
Cheers,
Jacq

[Jacq]

This would instantly make Loki viable in my environment - fluentd requires ruby, which is no-go, but having a single go executable would be perfect.

Alternatively, being able to accept data from winlogbeat (_or beats in general!_) and run it through the pipline would be amazing!

Why not using fluentbin input winlogevent plugin https://docs.fluentbit.io/manual/pipeline/inputs/windows-event-log ?
,and the grafana loki output plugin https://github.com/grafana/loki/tree/master/cmd/fluent-bit
No ruby is required for fluentbit which relies on C and Go for the loki plugin, both combined are below 30MB.

I have both winlogbeat (which is amazing) and the above config with fluent bit and seems to work, only caveat is that I have yet to include eventlog level information (info,error...) in the loki labels.

[cosmo0920]

fluent-bit winlogevent plugin does not support to retrieve eventlog's description which should be supported with <winevt.h> API but fluent-bit winlogevent plugin does not use the new Windows EventLog API.

[simnv]

I could not find a similar solution with telegraf, it might be in the same situation as loki: influxdata/telegraf#4525

@Jacq Check influxdata/telegraf#8000

[secustor]

@Jacq Can you share how you have built the Loki FluentD-bit plugin?
I get errors in the syscall package when I'm trying to compile with 'windows/amd64' as target platform. Thanks!

[Ulfy]

I'm also in need of a Loki compliant log shipper for Windows Event Logs. I'm currently trying to get Fluentbit working, since it has a winlog plugin. I'm not sure if I can build a Loki plugin using /cmd/fluent-bit. It just creates a file for *nix usage, not a Windows DLL/library. Is there a way to target a different arch for the file output?

[JacoboDominguez]

@Jacq Can you share how you have built the Loki FluentD-bit plugin?
I get errors in the syscall package when I'm trying to compile with 'windows/amd64' as target platform. Thanks!

I have to check, I think I tried to build also but finally grab the binary from the online repo.

[randomchance]

As far as I can tell, all of the fluent options only support the older style logs, not the newer "channels" such as Microsoft-Windows-DiskDiagnostic/Operational which ended up being a deal breaker for me.

Right now I'm using winlogbeat => logstash => loki and while I like winlogbeat, I really dislike running logstash on windows.

[cosmo0920]

As far as I can tell, all of the fluent options only support the older style logs, not the newer "channels" such as Microsoft-Windows-DiskDiagnostic/Operational which ended up being a deal breaker for me.

AFAIK, the newer Windows EventLog should be retrieved with <winevt.h> API.

[secustor]

I have found the online repo @JacoboDominguez is referencing.
It is the repo which has been used before the plugin has been adopted by the Grafana team. https://github.com/cosmo0920/fluent-bit-go-loki

Further I have opened an issue to supply the fluent-bit loki plugin for windows as binary or add a make command for this. #2563

[carwyn]

@azawawi are you able to share the code for the work you are doing? We have many Windows servers I can potentially test this on.

[Ulfy]

Right now I'm using winlogbeat => logstash => loki and while I like winlogbeat, I really dislike running logstash on windows.

@randomchance what kind of mutations are you doing to add labels for loki and drop high cardinality ones? I'm configuring this rn, but the output from winlogbeat is massive...

[randomchance]

@Ulfy sorry for the delay - I'm really only doing a couple of things, and taking advantage of the fact that top level fields with multiple values are dropped.

A quick summary:

1. I only add three labels right now (but that may change) Instance,Level and Weight - Weight {low|audit|normal|high} is just a measure of the importance of the log or source the entry is from and makes surfacing problems faster. I plan to have alerts triggered by high weight error level entries.
2. I create a header of name=value pairs such as event_id=1753 to make filtering easier and add it as the first line of the message.
3. Metadata about the entry that I want to save is serialized to JSON and added as the last line of the message.
4. Other top level fields I rename to something like [loki][temp][field_name] so they are discarded when sent to Loki, but still show up if I send them to a file while debugging.

The guidelines for loki stress how important it is to not add a ton of labels, and point out that the regex filtering is very powerful.
After creating a header to filter on I can say that the search time and low storage cost is impressive, however there are serious pain points.

• While Grafana parses the header and they show as "Parsed Fields" the built in filter does nothing with them.
• While you can get a count of errors with "provider=Microsoft-Windows-FailoverClustering-Clusport-Diagnostic" you can not get a list of providers and their counts.
• Basically, only "true" labels are first class citizens if you are doing anything other then using the "Explore" viewer or searching for pattern you already know.

It's possible that sending logs to promtail first and generating metrics there will help address some of the issues I've met, but I'm probably going to need to add some indication of the log channel as a label, though I'm not sure what that will look like yet.

[calebcoverdale]

@randomchance would you be willing to share your config file for Logstash to Loki? I got winlogbeat to talk to Logstash, now I am a bit lost in actually formatting the data to get into Loki.

[randomchance]

@calebcoverdale
I can't share the my full config, but I have mocked up something similar.
One thing I can't stress enough is to double check your string quotation - the logstash config language is frankly horrible when it comes to rules on quoting identifiers.

Here is a excerpt from a lessons learned KB I put together for my team:

See Accessing Event Data and Fields in the Configuration and Field References Deep Dive

Field names (object properties) MUST NOT be quoted in conditionals, but MUST in other situations - I'm explicitly not going to specify where because you should go re-read the documentation. Again. Every time something does not work as expected.

The string handling in configuration files alone will, eventually, go a long way towards either convincing you to develop test input that fully exercises any conditionals in your configuration, or convincing you to >find another log processing solution, if you can.

Here is full example config, I tried to add explanatory comments:

input{
    beats { 
        port => 5044 
        add_field => {       
            "[loki][header]"  => " "      
            "[weight]"  => "unknown" 
            "[loki][save][agent-version]" => "%{[agent][version]}"
        }
    }
}

filter{
# a lot of the channels have events that just say this :( 
    if  [message] =~ "/^For internal use/" {
        drop{}
    }

  if [agent][type] == "winlogbeat" {

      # create the top level field [level], which becomes a label.
      # create the field [loki] and the nested field [level] in case I want to use it later
    mutate {
      add_field => {
        "[loki][level]" => "%{[log][level]}"
        "[level]" => "%{[log][level]}"
        }
    }

    # If the event is from the "classic" event logs
    # I use the provider name as the "source" for the event and store it in [loki][identifier]
    if  [winlog][channel] =~ "(System|Application|Security|Setup)" {
      mutate {
        add_field => {
          "[loki][identifier]" => "%{[winlog][provider_name]}"
        }
        add_tag => ["classic_log","%{[winlog][provider_name]}"]
      }
    }else {
      mutate {
        add_field => {
        # The newer log channels are generaly single application/system so I use the channel name for those
          "[loki][identifier]" => "%{[winlog][channel]}"
        }
        add_tag => ["channel","%{[winlog][channel]}"]
      }
    }

    #################################################################
    # now we can set the weight 
    #################################################################

    # this uses the translate function to pull a weight value from an external json file
    # the if none of the keys match fallback to "normal"
    translate {
        field => "[loki][identifier]"
        destination => "[loki][weight]"
        regex => true
        dictionary_path => "D:/logstash/config/log-weights.json"
        fallback => "normal"    
        # I add tags everywhere so I can follow the event progression while debugging
        add_tag => [ "sys_translate_provider_weight" ] 
      }

    #################################################################
    # Now set the category - audits get special treatment first
    #################################################################

    # add an Audit line to security audits and the audit header

    if [winlog][provider_name] =~ "Security-Auditing" {
      mutate{
      replace => { 
        "[loki][header]" => " \n\t type=%{[event][type]} category=%{[event][category]} action=%{[event][action]} outcome=%{[event][outcome]} %{[loki][header]}"  
        "[loki][weight]" => "audit"
        "[loki][category]" => "%{[event][category]}-audit"
        }
        # these fields only exists on audit events
        add_field  => {
          "[loki][event_data][logon]" => "%{[winlog][logon]}"
          "[loki][event_data][keywords]" => "%{[winlog][keywords]}"
          }
          add_tag => [ "sec-audit" ]
      }
    }  else { 
      # having the same field and source means a replace is done, otherwise it only seems to work on new fields
      # correction, replace does not seem to work, contrary to the docs (at least on windows)
      translate {
        field => "[loki][identifier]"
        destination => "[loki][category]"
        dictionary_path => "D:/logstash/config/category-mapping.json"
        fallback => "windows"
        add_tag => [ "channel_translate" ]
      }
    }

    # clean up the channel and provider names
    mutate {
      # provider can have spaces, so replace them
            gsub => [
            # replace all spaces with minus
          "[winlog][provider_name]", "\s", "-"
          "[winlog][channel]", "\s", "-"
          ]
    }

# I'm mostly building a header string here

    # Add the channel and provider to the front of the [loki][header], creating it if it's not there.
    mutate {
      replace => {
        "[loki][header]" => "channel=%{[winlog][channel]} provider=%{[winlog][provider_name]} %{[loki][header]}"
        }
    }

    if [host][ip] {
        mutate {   
            rename => {"[host][ip]" => "[loki][event_data][host_ip]"}    
        }
    }

    if [host][mac] {
        mutate {   
            rename => {"[host][mac]" => "[loki][event_data][host_mac]"}    
        }
    }


    # We don't want a User label, so rename it, I'm storing it in a subfield of [loki] called [event_data]
    # I'm going to save [event_data] later
    if [user] {
        mutate {
            rename => {"[user]" => "[loki][event_data][user]"}
        }
    }

    # reboot events get flagged as critical
    if "[winlog][event_id]" {
        mutate {   
            replace => {
              #"message" => "event_id=%{[winlog][event_id]} %{[message]}"
              "[loki][header]" => "event_id=%{[winlog][event_id]} %{[loki][header]}"
              }
            add_field  => {"[loki][event_data][event_id]" => "%{[winlog][event_id]}"}    
        }

         if [winlog][provider_name] =~  "Kernel-(General|Power|Boot)" and  [winlog][event_id] =~ "(12|13|109)" {
          mutate {
            replace => {
            "[loki][category] "=> "boot"
            "[loki][weight]"=> "high"
            }

          }
         }
    }
  }  
  # end if-winlogbeat


# this is where I populate the [event_data] with any info I might want later

  if [host][ip] {
      mutate {   
          rename => {"[host][ip]" => "[loki][event_data][host_ip]"}    
      }
  }

  if [host][mac] {
      mutate {   
          rename => {"[host][mac]" => "[loki][event_data][host_mac]"}    
      }
  }
  if [agent][hostname] {
      mutate {   
          rename => {"[agent][hostname]" => "[loki][event_data][computer_name]"}    
      }
  }

# I use the [instance] (which is the computer name) as a label, 
# and it's nice to have consistent casing.
    if [instance] {
      mutate {   
            capitalize => ["[instance]"]  
      }
  }

# might as well keep the oridginal event data

  if [winlog][event_data] {
      mutate {   
          rename => {"[winlog][event_data]" => "[loki][event_data][event_data]"}    
      }
  }

# Add the category to the header

 mutate {
    replace => {
      #"message" => "level=%{[log][level]} %{[message]}"
     "[loki][header]" => "category=%{[loki][category]} %{[loki][header]}"
      }
  }

# Add the level to the header
# this is last so it's first on the log line.

  if [log][level] {
      mutate {   
        #add_field  => {[loki][event_data][level] => "%{[log][level]}"}              
        replace => {
          #"message" => "level=%{[log][level]} %{[message]}"
          "[loki][header]" => "level=%{[log][level]} %{[loki][header]}"
          }    
      }
  }

  mutate {   
    # set the weight to the top level field so it will be a label
    # this replaces / creates [weight] with the content of [loki][weight]
    replace => {
      "weight" =>"%{[loki][weight]}"
    }
  }

  # this serializes the [loki][event_data] object and overwrites it with the json string
  json_encode {
    source => "[loki][event_data]"
  }

   mutate {   
    # add the built header to the message line and push the message to the next line
    # add the event data json string to the end on a new line
    replace => {
      "message" => "%{[loki][header]} \n%{[message]} \n\tEVENTDATA= %{[loki][event_data]}"
    }
    # move the tags array to a sub field so it won't be a label
    # it's not supposed to become one, but I've had some weird things happen
    rename => { "[tags]" => "[temp][tags]"  } 
  }

}
output {

    # I do this to check that the events look like I expect, then swap to the loki output
  file {
    path => "D:\logstash.log"
  }
  # loki {
  #   url => "http://sweet.loki.goodness/loki/api/v1/push"
  # }
}

[fifofonix]

This thread was useful getting a working fluentd->loki setup going for Windows EventLog using in_windows_eventlog2 as @cosmo0920 suggested. This supports ?new? channels.

This is a ruby-based solution but @randomchance this does allow you to specify channels like Microsoft-Windows-DiskDiagnostic/Operational, or even include all channels with a separate option.

When specifying channels in the fluentd config the key is no quoting or escaping which tricked me out initially:

<source>
  @type windows_eventlog2
  @id windows_eventlog2
  # Do not quote "" or escape \ characters in channel names...
  channels application,system,security,HardwareEvents,Windows PowerShell, Microsoft-Windows-Diagnosis-PCW/Operational 
</source>

[randomchance]

@fifofonix That's awesome! The docs do not support that, so I opened an issue to get them updated.

For anyone curious, the documentation says/implies that the standard four logs are the entire set of possible options.

One or more of {'application', 'system', 'setup', 'security'}.

I'm already pretty invested in my current config, but I'll definitely try out a fluentd configuration if I get a chance. Having more options is definitely better.

Now that promtail supports syslog input, using a log shipper that outputs syslog is also an option.
If anyone tries that, be aware promtail is geared towards getting logs into loki, it's not nearly as flexible and does not allow the crazy level of processing / editing that you can do in logstash, which is probably a good thing.

[danfoxley]

...and Telegraf now has an input plugin for Windows Event Log

https://github.com/influxdata/telegraf/blob/v1.16.0/plugins/inputs/win_eventlog/README.md

[AMoghrabi]

Hey @randomchance, thanks a lot for your logstash example, I appreciate it. I'm currently setting up something similar for my company and I had a question about the [loki][header] portion -- what does that provide you exactly?

My understanding is Loki tags must be key, value pairs, and the values cannot be nested. When you add a new field such as "[loki][level]", I'm not sure where that gets shown or used in Loki. I'm testing it out right now and when viewing the logs in Loki, I can't see it as a label nor in the logs that are coming from logstash.

I'm fairly new to Logstash so I'm probably misinterpreting this entirely. I'd appreciate your guidance. Thanks!

Edit: Nevermind -- I had some time today to go through the entire config and I see it gets amended to message.

[Ulfy]

...and Telegraf now has an input plugin for Windows Event Log

https://github.com/influxdata/telegraf/blob/v1.16.0/plugins/inputs/win_eventlog/README.md

@danfoxley Does Telegraf have a Loki plugin to output w/ labels? I'm still looking for something to ship windows event logs to Loki...