Beats: [Filebeat] Add support for more IIS log file formats

[kaiyan-sheng]

Right now, iis module seems to only support W3C Extended Log File Format with different versions. Based on Microsoft documentation
https://docs.microsoft.com/en-us/previous-versions/iis/6.0-sdk/ms525807(v%3Dvs.90), there are other log formats like IIS Log File Format and NCSA Common Log File Format will be useful too.

IIS Log File Format example:
192.168.114.201, -, 03/20/01, 7:55:20, W3SVC2, SALES1, 172.21.13.45, 4502, 163, 3223, 200, 0, GET, /DeptLogo.gif, -,
172.16.255.255, anonymous, 03/20/01, 23:58:11, MSFTPSVC, SALES1, 172.16.255.255, 60, 275, 0, 0, 0, PASS, /Intro.htm, -,

NCSA Common Log File Format example:
172.21.13.45 - Microsoft\fred [08/Apr/2001:17:39:04 -0800] "GET /scripts/iisadmin/ism.dll?http/serv HTTP/1.0" 200 3401

[philippkahr]

Hi,

I recently stumbled across the IIS module and I am using it in conjunction with Microsoft Exchange. The logs from IIS in combination with Exchange look a bit more different because Exchange puts a lot of fields inside the IIS.

Here are some log messages and a working GROK pattern. I tested the grok pattern using the Kibana Grok analyzer inside the dev tools.

Log format 1

2019-10-20 15:13:55 POST /Microsoft-Server-ActiveSync/default.eas Cmd=Ping&User=philipp.kahr&DeviceId=9c731cc0e9fe8798783cb57e5977edd5e&DeviceType=Outlook&CorrelationID=<empty>;&ClientId=MJKNJEDEUJUUUID&cafeReqId=896c4201-8uue-489c-b193-f1e73aa6287a; - 212.166.112.250 Outlook-iOS-Android/1.0 - 200
2019-10-20 16:56:06 POST /EWS/Exchange.asmx &CorrelationID=<empty>;&ClientId=LYBSLCQWEAJUYAMAAAA&cafeReqId=b45739f5-8de8-4e3a-1q2w-5054f4627012; - 212.166.112.250 MS-WebServices/1.0 - 401

Grok filter 1

Example Grok Pattern, I cannot figure out how to skip `-`character, so I put it inside a field called `drop`.

%{TIMESTAMP_ISO8601:iis.access.time} %{WORD:http.request.method} %{URIPATH:url.path} %{NOTSPACE:url.query} %{NOTSPACE:drop} %{IPORHOST:source.address} %{NOTSPACE:user_agent.original} %{NOTSPACE:drop} %{NUMBER:http.response.status_code:long}

{
  "drop": "-",
  "iis": {
    "access": {
      "time": "2019-10-20 15:13:55"
    }
  },
  "http": {
    "request": {
      "method": "POST"
    },
    "response": {
      "status_code": 200
    }
  },
  "source": {
    "address": "212.166.112.250"
  },
  "user_agent": {
    "original": "Outlook-iOS-Android/1.0"
  },
  "url": {
    "path": "/Microsoft-Server-ActiveSync/default.eas",
    "query": "Cmd=Ping&User=philipp.kahr&DeviceId=9c731cc0e9fe8798783cb57e5977edd5e&DeviceType=Outlook&CorrelationID=<empty>;&ClientId=MJKNJEDEUJUUUID&cafeReqId=896c4201-8uue-489c-b193-f1e73aa6287a;"
  }
}
{
  "drop": "-",
  "iis": {
    "access": {
      "time": "2019-10-20 16:56:06"
    }
  },
  "http": {
    "request": {
      "method": "POST"
    },
    "response": {
      "status_code": 401
    }
  },
  "source": {
    "address": "212.166.112.250"
  },
  "user_agent": {
    "original": "MS-WebServices/1.0"
  },
  "url": {
    "path": "/EWS/Exchange.asmx",
    "query": "&CorrelationID=<empty>;&ClientId=LYBSLCQWEAJUYAMAAAA&cafeReqId=b45739f5-8de8-4e3a-1q2w-5054f4627012;"
  }
}

Log format 2

2019-10-20 15:13:56 POST /mapi/emsmdb/ [email protected]&CorrelationID=<empty>;&ClientId=FBLAXEWQEITNKQWERFFLG&ClientRequestInfo=R:{F66F1938-D99D-43F6-QWER-93FEF763BBCF}:22444;CI:{B2939AE0-65A7-1234-BD08-4FF5F68CA673}:19;RT:Execute&cafeReqId=bac0dc43-d123-4d15-8c47-436651764660; EXAMPLE\philippkahr 84.112.190.213 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.4849;+Pro) ClientId=FBLAXEWQWERNKCPORFFLG;MapiContext=MAPIAAAAAPaj+99+yf7d79/u1/rK8t/u7u7/MbyxP7N9a+MtYazgrOKuYu8Bw0zAAAAAAA=;MapiSequence=22364-xw+vIA==;X-BackEndCookie=3c5a09e2-u8u8-4451-b1b6-21dd29813d3f=u56Lnp2ejJqBnsibxseeyJzSy8jMydLLxsrN0sedyJzSz87OmpudysnLmZvKgYHNz87G0s7O0s7Gq87Kxc7MxcrJ 200
2019-10-20 17:12:43 POST /mapi/nspi/ [email protected]&CorrelationID=<empty>;&ClientId=ZHVWIKUEGTTTDJQWER&ClientRequestInfo=R:{264C1704-1234-4352-8UUU-4F77E047F94E}:243;CI:{335BABFC-9UUU-448D-B995-EEBB1FCF65E9}:5;RT:PING&cafeReqId=5155eda1-1234-4f6f-asdf-018dc5d163be; EXAMPLE\philippkahr 212.166.112.250 Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Lync+16.0.4849;+Pro) ClientId=ZHVWIKUEGTTTDJQWER;MapiContext=MAPIAAAAAPaj+87+yqweR6trr0v/P9tvp3f3M/cfyyvDA9a+Mvo29i7qDsYezGcMBAAAAAAA=;MapiSequence=56-3asd4YQ==;X-BackEndCookie=asd82dc1-9160-1234-96f2-b5366b14d319=u56Lnp2ejJqBzc3KzJvKncfSncmbzdTZErw/H0p3LzpvSmsuZxsaax5rMnsfOgYHNz87G0s123s7Gq87Ixc/Kxc7P 200

Grok filter 2

I am not quite sure if `url.param` is the best field to place this information `ClientId=FBLAXEWQWERNKCPORFFLG;MapiContext=MAPIAAAAAPaj+99+yf7d79/u1/rK8t/u7u7/MbyxP7N9a+MtYazgrOKuYu8Bw0zAAAAAAA=;MapiSequence=22364-xw+vIA==;X-BackEndCookie=3c5a09e2-u8u8-4451-b1b6-21dd29813d3f=u56Lnp2ejJqBnsibxseeyJzSy8jMydLLxsrN0sedyJzSz87OmpudysnLmZvKgYHNz87G0s7O0s7Gq87Kxc7MxcrJ`

%{TIMESTAMP_ISO8601:iis.access.time} %{WORD:http.request.method} %{URIPATH:url.path} %{NOTSPACE:url.query} %{NOTSPACE:user.name} %{IPORHOST:source.address} %{NOTSPACE:user_agent.original} %{NOTSPACE:url.param} %{NUMBER:http.response.status_code:long}

{
  "iis": {
    "access": {
      "time": "2019-10-20 15:13:56"
    }
  },
  "http": {
    "request": {
      "method": "POST"
    },
    "response": {
      "status_code": 200
    }
  },
  "source": {
    "address": "84.112.190.213"
  },
  "user": {
    "name": "EXAMPLE\\philippkahr"
  },
  "user_agent": {
    "original": "Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Outlook+16.0.4849;+Pro)"
  },
  "url": {
    "path": "/mapi/emsmdb/",
    "param": "ClientId=FBLAXEWQWERNKCPORFFLG;MapiContext=MAPIAAAAAPaj+99+yf7d79/u1/rK8t/u7u7/MbyxP7N9a+MtYazgrOKuYu8Bw0zAAAAAAA=;MapiSequence=22364-xw+vIA==;X-BackEndCookie=3c5a09e2-u8u8-4451-b1b6-21dd29813d3f=u56Lnp2ejJqBnsibxseeyJzSy8jMydLLxsrN0sedyJzSz87OmpudysnLmZvKgYHNz87G0s7O0s7Gq87Kxc7MxcrJ",
    "query": "[email protected]&CorrelationID=<empty>;&ClientId=FBLAXEWQEITNKQWERFFLG&ClientRequestInfo=R:{F66F1938-D99D-43F6-QWER-93FEF763BBCF}:22444;CI:{B2939AE0-65A7-1234-BD08-4FF5F68CA673}:19;RT:Execute&cafeReqId=bac0dc43-d123-4d15-8c47-436651764660;"
  }
}
{
  "iis": {
    "access": {
      "time": "2019-10-20 17:12:43"
    }
  },
  "http": {
    "request": {
      "method": "POST"
    },
    "response": {
      "status_code": 200
    }
  },
  "source": {
    "address": "212.166.112.250"
  },
  "user": {
    "name": "EXAMPLE\\philippkahr"
  },
  "user_agent": {
    "original": "Microsoft+Office/16.0+(Windows+NT+10.0;+Microsoft+Lync+16.0.4849;+Pro)"
  },
  "url": {
    "path": "/mapi/nspi/",
    "param": "ClientId=ZHVWIKUEGTTTDJQWER;MapiContext=MAPIAAAAAPaj+87+yqweR6trr0v/P9tvp3f3M/cfyyvDA9a+Mvo29i7qDsYezGcMBAAAAAAA=;MapiSequence=56-3asd4YQ==;X-BackEndCookie=asd82dc1-9160-1234-96f2-b5366b14d319=u56Lnp2ejJqBzc3KzJvKncfSncmbzdTZErw/H0p3LzpvSmsuZxsaax5rMnsfOgYHNz87G0s123s7Gq87Ixc/Kxc7P",
    "query": "[email protected]&CorrelationID=<empty>;&ClientId=ZHVWIKUEGTTTDJQWER&ClientRequestInfo=R:{264C1704-1234-4352-8UUU-4F77E047F94E}:243;CI:{335BABFC-9UUU-448D-B995-EEBB1FCF65E9}:5;RT:PING&cafeReqId=5155eda1-1234-4f6f-asdf-018dc5d163be;"
  }
}

Log format 3

2019-10-20 16:56:08 GET /mapi/healthcheck.htm - - 84.112.190.213 - - 200
2019-10-20 17:08:56 GET /ews/healthcheck.htm - - 84.112.190.213 - - 200

Grok filter 3

Here I am opposed to the same issue as in my first grok filter, saving the `-` char into a field called `drop`.

%{TIMESTAMP_ISO8601:iis.access.time} %{WORD:http.request.method} %{URIPATH:url.path} %{NOTSPACE:drop} %{NOTSPACE:drop} %{IPORHOST:source.address} %{NOTSPACE:drop} %{NOTSPACE:drop} %{NUMBER:http.response.status_code:long}

{
  "drop": "-",
  "iis": {
    "access": {
      "time": "2019-10-20 16:56:08"
    }
  },
  "http": {
    "request": {
      "method": "GET"
    },
    "response": {
      "status_code": 200
    }
  },
  "source": {
    "address": "84.112.190.213"
  },
  "url": {
    "path": "/mapi/healthcheck.htm"
  }
}
{
  "drop": "-",
  "iis": {
    "access": {
      "time": "2019-10-20 17:08:56"
    }
  },
  "http": {
    "request": {
      "method": "GET"
    },
    "response": {
      "status_code": 200
    }
  },
  "source": {
    "address": "84.112.190.213"
  },
  "url": {
    "path": "/ews/healthcheck.htm"
  }
}

Log format 4

2019-10-20 17:15:59 POST /owa/service.svc action=UpdateItem&ID=-53&AC=1&CorrelationID=547186e9-1234-4fc4-8522-74e03e586ade_157159175977353;&ClientId=WXDQHKBEIJBWICQWER&cafeReqId=6d21836a-0d43-4f74-a87a-92dae8a858f2; philippkahr 212.166.112.250 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/67.0.3396.99+Safari/537.36 X-BackEndCookie=S-1-5-21-3544562028-792812758-8888637587-34534=u56Lnp2ejJqBys+dzsvLy87SnZ3NyNLLy56b0p3PzsrSm8iZzpvNzszQwe7urzAYHNz87G0s7O0s7Gq87Ixc7KxcvG;+GUEST_LANGUAGE_ID=en_US;+ClientId=WXDQHKBEIJBWICQWER;+NSC_MC_DBT_PXB_NJU_BBB=ffffffffc3aiuiJu045525d5f4f58455e445a4a42378b;+cadata=C+SckAuIUL75SqEyLu0kFPjuGqzdVuiOWHNYkLLP9ER2hJ55TJn+ecExFj3GqEbiNTHyiOYTWea/2Ng7v8i9oI==;+cadataTTL=zlVJmWMTJLjERDh+k2Gasdw==;+cadataKey=DFE9oh67Q6jE4vwuTZgxqFMcFHjTsi7+63BhHoVp8WXPi0o8Tt5iOm+ZUVv0riqS3ZbD/ssg6cGADNh7asdhEuyAVTmVRQ9XI4Lz5qoJ2lZyDLfqeVSP8JFWdbyc4/AHTYGJiasdjdanIjqClBYKtHK0+isB218FiLTfKpafu32IblX5I1kIGgP13iwckM1q/64HpsNWIYLCYpxHXQkhhqz/de82E5iqZtppoUz7h/nJYvHxUzbzL17B7pNMKWcdHeQtmrfShkgJtxtV6To6JZCaTv1+XW+GT0TfL7XqM8cW/P8xqeZBtJmWR7EH4kjLKVbqwaDsz3ciMV3mw7yGYVFSvxMVmg==;+cadataIV=GTeJ1Y8mgpjK62v4zBogGgbcadgNGZCVvIHtwsobr4DiIyLsVoIwRVTJVsLTYZaetwPy6x6jMgxnXNiA0nVdaIJcNN59y6/A5nc8DPl1ykSlYgWPkc4ahxdOaDzg54dagzZtnGEfeiJ2P97RyRqO5eC5YGXg2MNjKV6mYw10IZw+b6/tU/dVMBW5RZHoWPrzJtAREu4aa1575648zEAOZPVoQOIo6y7+sT7aCxQ321TDS8OKRQvWJI1lH+AE68ypPDNay9+97abhBZ2tWlJDQ+dNfg572n4R10QjChXYDasdagZBZadqwo7KwITnyqucj/THdqA76IT3bFFFTZZWTtuOUO3OLcRsIKqkwV5zqQ==;+cadataSig=u3FddXDDeB54SEXDjCE4Nx/ph/wBatPGZGZZGHRNRNh6olZs+rl1Muj5ZZ9ecg9Paax6MPydDiwbgNdePWzWNnZ++VV/lZ+yFyNof3NWfdy46nRVMxsd3eUxzgUxLzup1KLd6fE2WPPJbpymo888ufWhTxcRxR5Px2boEgH8TxOznMWlV3UWlU0M7lQKpUZNamGELYP5JzDt+3NryqS21J8dxC759EaXBZBZWIOur74Unhjz3yxp2uZJ2GUkvU0FRzjTnzXM78UCHyNACSXVdixs8nU01ftVjez1uvDA6HYJrbImMV9FHjwpFqwKvOU5Gl/XLdNXkrY6FT5jS99u+U8nI0K+0AuXjgxcg==;+AppcacheVer=15.0.1473.3:en-usbase;+UC=da16c272c89c49df98dbdda7d20d7ec8;+X-OWA-CANARY=Ss4dZFGvM0GAuvmEKZCGATGUWONEUUOsBd5p0PK9VnA6xuaK4HoW9v0hxOhg4vPtc10DBY. 200

Grok filter 4

I have the same issue here, that I do not know if it's ok to put the weird string into `url.params`

%{TIMESTAMP_ISO8601:iis.access.time} %{WORD:http.request.method} %{URIPATH:url.path} %{NOTSPACE:url.query} %{USERNAME:user.name} %{IPORHOST:source.address} %{NOTSPACE:user_agent.original} %{NOTSPACE:url.param} %{NUMBER:http.response.status_code:long}

{
  "iis": {
    "access": {
      "time": "2019-10-20 17:15:59"
    }
  },
  "http": {
    "request": {
      "method": "POST"
    },
    "response": {
      "status_code": 200
    }
  },
  "source": {
    "address": "212.166.112.250"
  },
  "user": {
    "name": "philippkahr"
  },
  "user_agent": {
    "original": "Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/67.0.3396.99+Safari/537.36"
  },
  "url": {
    "path": "/owa/service.svc",
    "param": "X-BackEndCookie=S-1-5-21-3544562028-792812758-8888637587-34534=u56Lnp2ejJqBys+dzsvLy87SnZ3NyNLLy56b0p3PzsrSm8iZzpvNzszQwe7urzAYHNz87G0s7O0s7Gq87Ixc7KxcvG;+GUEST_LANGUAGE_ID=en_US;+ClientId=WXDQHKBEIJBWICQWER;+NSC_MC_DBT_PXB_NJU_BBB=ffffffffc3aiuiJu045525d5f4f58455e445a4a42378b;+cadata=C+SckAuIUL75SqEyLu0kFPjuGqzdVuiOWHNYkLLP9ER2hJ55TJn+ecExFj3GqEbiNTHyiOYTWea/2Ng7v8i9oI==;+cadataTTL=zlVJmWMTJLjERDh+k2Gasdw==;+cadataKey=DFE9oh67Q6jE4vwuTZgxqFMcFHjTsi7+63BhHoVp8WXPi0o8Tt5iOm+ZUVv0riqS3ZbD/ssg6cGADNh7asdhEuyAVTmVRQ9XI4Lz5qoJ2lZyDLfqeVSP8JFWdbyc4/AHTYGJiasdjdanIjqClBYKtHK0+isB218FiLTfKpafu32IblX5I1kIGgP13iwckM1q/64HpsNWIYLCYpxHXQkhhqz/de82E5iqZtppoUz7h/nJYvHxUzbzL17B7pNMKWcdHeQtmrfShkgJtxtV6To6JZCaTv1+XW+GT0TfL7XqM8cW/P8xqeZBtJmWR7EH4kjLKVbqwaDsz3ciMV3mw7yGYVFSvxMVmg==;+cadataIV=GTeJ1Y8mgpjK62v4zBogGgbcadgNGZCVvIHtwsobr4DiIyLsVoIwRVTJVsLTYZaetwPy6x6jMgxnXNiA0nVdaIJcNN59y6/A5nc8DPl1ykSlYgWPkc4ahxdOaDzg54dagzZtnGEfeiJ2P97RyRqO5eC5YGXg2MNjKV6mYw10IZw+b6/tU/dVMBW5RZHoWPrzJtAREu4aa1575648zEAOZPVoQOIo6y7+sT7aCxQ321TDS8OKRQvWJI1lH+AE68ypPDNay9+97abhBZ2tWlJDQ+dNfg572n4R10QjChXYDasdagZBZadqwo7KwITnyqucj/THdqA76IT3bFFFTZZWTtuOUO3OLcRsIKqkwV5zqQ==;+cadataSig=u3FddXDDeB54SEXDjCE4Nx/ph/wBatPGZGZZGHRNRNh6olZs+rl1Muj5ZZ9ecg9Paax6MPydDiwbgNdePWzWNnZ++VV/lZ+yFyNof3NWfdy46nRVMxsd3eUxzgUxLzup1KLd6fE2WPPJbpymo888ufWhTxcRxR5Px2boEgH8TxOznMWlV3UWlU0M7lQKpUZNamGELYP5JzDt+3NryqS21J8dxC759EaXBZBZWIOur74Unhjz3yxp2uZJ2GUkvU0FRzjTnzXM78UCHyNACSXVdixs8nU01ftVjez1uvDA6HYJrbImMV9FHjwpFqwKvOU5Gl/XLdNXkrY6FT5jS99u+U8nI0K+0AuXjgxcg==;+AppcacheVer=15.0.1473.3:en-usbase;+UC=da16c272c89c49df98dbdda7d20d7ec8;+X-OWA-CANARY=Ss4dZFGvM0GAuvmEKZCGATGUWONEUUOsBd5p0PK9VnA6xuaK4HoW9v0hxOhg4vPtc10DBY.",
    "query": "action=UpdateItem&ID=-53&AC=1&CorrelationID=547186e9-1234-4fc4-8522-74e03e586ade_157159175977353;&ClientId=WXDQHKBEIJBWICQWER&cafeReqId=6d21836a-0d43-4f74-a87a-92dae8a858f2;"
  }
}

KV Processor

I would also have one additional request/suggestion. Maybe using the KV Processor from elasticsearch could help with splitting the data inside url.query into something more useful. I tested it with the simulate API and my first example, which worked flawlessly. I did not do any mapping to correct the fields I just dumped them inside kvsplitted. Here is my simulate pipeline:

Simulate API

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "_description",
    "processors": [
      {
        "grok":{
          "field": "message",
          "patterns":[
            "%{TIMESTAMP_ISO8601:iis.access.time} %{WORD:http.request.method} %{URIPATH:url.path} %{NOTSPACE:url.query} %{NOTSPACE:drop} %{IPORHOST:source.address} %{NOTSPACE:user_agent.original} %{NOTSPACE:drop} %{NUMBER:http.response.status_code:long}"
            ]
        },
        "kv": {
          "field_split": "&",
          "value_split": "=",
          "field": "url.query",
          "target_field": "kvsplitted",
          "ignore_failure": true
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "message": "2019-10-20 15:13:55 POST /Microsoft-Server-ActiveSync/default.eas Cmd=Ping&User=philipp.kahr&DeviceId=9c731cc0e9fe8798783cb57e5977edd5e&DeviceType=Outlook&CorrelationID=<empty>;&ClientId=MJKNJEDEUJUUUID&cafeReqId=896c4201-8uue-489c-b193-f1e73aa6287a; - 212.166.112.250 Outlook-iOS-Android/1.0 - 200"
      }
    }
  ]
}

Simulate API result

{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "drop" : "-",
          "iis" : {
            "access" : {
              "time" : "2019-10-20 15:13:55"
            }
          },
          "http" : {
            "request" : {
              "method" : "POST"
            },
            "response" : {
              "status_code" : 200
            }
          },
          "source" : {
            "address" : "212.166.112.250"
          },
          "message" : "2019-10-20 15:13:55 POST /Microsoft-Server-ActiveSync/default.eas Cmd=Ping&User=philipp.kahr&DeviceId=9c731cc0e9fe8798783cb57e5977edd5e&DeviceType=Outlook&CorrelationID=<empty>;&ClientId=MJKNJEDEUJUUUID&cafeReqId=896c4201-8uue-489c-b193-f1e73aa6287a; - 212.166.112.250 Outlook-iOS-Android/1.0 - 200",
          "kvsplitted" : {
            "DeviceType" : "Outlook",
            "User" : "philipp.kahr",
            "DeviceId" : "9c731cc0e9fe8798783cb57e5977edd5e",
            "ClientId" : "MJKNJEDEUJUUUID",
            "Cmd" : "Ping",
            "CorrelationID" : "<empty>;",
            "cafeReqId" : "896c4201-8uue-489c-b193-f1e73aa6287a;"
          },
          "user_agent" : {
            "original" : "Outlook-iOS-Android/1.0"
          },
          "url" : {
            "path" : "/Microsoft-Server-ActiveSync/default.eas",
            "query" : "Cmd=Ping&User=philipp.kahr&DeviceId=9c731cc0e9fe8798783cb57e5977edd5e&DeviceType=Outlook&CorrelationID=<empty>;&ClientId=MJKNJEDEUJUUUID&cafeReqId=896c4201-8uue-489c-b193-f1e73aa6287a;"
          }
        },
        "_ingest" : {
          "timestamp" : "2019-10-20T16:37:59.856569Z"
        }
      }
    }
  ]
}

[narph]

@kaiyan-sheng ,
In principle filebeat modules are intended to support default log formats of services which seems to be the case here as well (we currently support only W3C formats).
We should edit the filebeat/iis module documentation and specify this limitation.

I wonder what are the most common iis log formats users work with (besides W3C) , @sorantis , feel free to chime in if you have any ideas, but this issue sounds like a great starting point.

@philippkahr , the situation you are dealing with seems to be different than the initial conversation point (supporting NCSA and IIS logs). From the log lines provided, it seems that the log format is W3C.
I suggest creating a different issue to followup, correct me if I am wrong here and the subjects are related.

[sorantis]

@kaiyan-sheng @narph looks like W3C is the most common format for collecting logs.
According to the official documentation: the W3C Extended log file format is the preferred log type to use. Also, other IIS integration providers default to it.

[philippkahr]

Hi @narph

seems ok to me. I wanted to help and point to the other "weird" log formats that IIS comes up with. I will create an additional issue and link it to my comment. I needed it for my work anyway, so I implemented the grok patterns in our pipeline and they work :D. At least we know something ;).

[kaiyan-sheng]

@narph W3C is the most common format 👍 We can either add support for other "weird" log formats(@philippkahr thank you for your help!) or add more documentation on exactly what we support currently. WDYT?

[narph]

I would update the documentation in this case and specify that in this case we support W3C formats only, at the moment no NCSA or IIS.
I would like to follow up on https://github.com/elastic/beats/issues/14284 and test if we can improve the current pipeline and accommodate this use case for the W3C format.
Regarding the other formats you mentioned above ( NCSA or IIS) would be interesting to hear from users, if this is something they are interested.

[philippkahr]

My log files are only generated when using an exchange mail server. We use them quite often to troubleshoot any connectivity issues. I do not know how the community feels about it. Maybe start with a similar architecture like the cisco module?

- module: iis-generic
- module: iis-exchange
- module: iis...