Goaccess: How can I pass custom nginx log format to GoAccess

Created on 1 Apr 2018  路  28Comments  路  Source: allinurl/goaccess

My nginx custom log format is 

'$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time $upstream_addr $upstream_response_time $pipe'

How can I use this in goaccess.

Tried using https://github.com/stockrt/nginx2goaccess and got back 

./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time $upstream_addr $upstream_response_time $pipe'

- Generated goaccess config:

time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" %T %^ %^ %^

When I used the above result in goaccess, it was throwing 

| Token 'EAD' doesn't match specifier '%s'                  |
| Token 'EAD' doesn't match specifier '%s'                  |
                                             | Token 'EAD' doesn't match specifier '%s'                  |
                                             | Token 'EAD' doesn't match specifier '%s'                  |
                                             | Token 'EAD' doesn't match specifier '%s'                  |
                                             | Token 'EAD' doesn't match specifier '%s'                  |
                                             | Token 'EAD' doesn't match specifier '%s'                  |
                                             | Token 'EAD' doesn't match specifier '%s'                  |
                                             | Token 'OST' doesn't match specifier '%s'                  |
                                             | Token 'OST' doesn't match specifier '%s'                  |

Any idea what to do here? Thanks.

lodattime format question
Source
picture luckydev

Most helpful comment

@Hosting-Server-Team This works for the line you posted above:

goaccess access.log --log-format='%h %^[%d:%t %^] "%r" %s %b"%R" "%u"' --date-format=%d/%b/%Y --time-format=%T
picture allinurl on 11 Apr 2018
❤2

All 28 comments

Can you please post a few lines from your log?

allinurl picture allinurl on 1 Apr 2018

i got the same issue,

here is my log example:

192.168.1.98 - - [10/Apr/2018:16:52:30 +0900] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36" 0.043 - "-"

here is my nginx2goaccess output:

./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time $upstream_response_time "$upstream_addr"'

  • Generated goaccess config:

time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" %T %^ "%^"

run goaccess:

goaccess -f access.log --log-format='%h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" %T %^ "%^"' --date-format=%d/%b/%Y --time-format=%T

error output:

Parsed 1 lines producing the following errors:

Token 'ET' doesn't match specifier '%s'

Format Errors - Verify your log/date/time format

my guess:

ET in the error message comes from "GET / HTTP/1.1"

any idea?

xxxatt picture xxxatt on 10 Apr 2018

@xxxatt For the sample line you posted above, this works for me. Please make sure you are using the latest version.

goaccess access.log --log-format='%h %^[%d:%t %^] "%r" %s %b "%R" "%u" %T %^' --date-format=%d/%b/%Y --time-format=%T
allinurl picture allinurl on 10 Apr 2018
❤2

@allinurl

I am getting the below issue

**Parsed 10 lines producing the following errors:**

Token 'OST' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'

here is my log example:

31.215.201.138 forwarded for - - - [11/Apr/2018:10:24:48 +0000]  "GET / HTTP/1.1" 200 14769"https://www.google.ae/" "Mozilla/5.0 (iPad; CPU OS 11_2_6 like Mac OS X) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0 Mobile/15D100 Safari/604.1"

Generated goaccess config:

time-format %T
date-format %d/%b/%Y
log-format %v:%^ %h %^[%d:%t %^] "%r" %s %b "%R" "%u"

run goaccess:

goaccess -f /var/log/nginx/access.log -a > /var/www/html/report.html

Could you please help me on this ?

Hosting-Server-Team picture Hosting-Server-Team on 11 Apr 2018

@Hosting-Server-Team This works for the line you posted above:

goaccess access.log --log-format='%h %^[%d:%t %^] "%r" %s %b"%R" "%u"' --date-format=%d/%b/%Y --time-format=%T
allinurl picture allinurl on 11 Apr 2018
❤2

@allinurl

Still there is a same issue.

Parsed 10 lines producing the following errors:

Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'OST' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'
Token 'ET' doesn't match specifier '%s'

Format Errors - Verify your log/date/time format
Hosting-Server-Team picture Hosting-Server-Team on 11 Apr 2018

@Hosting-Server-Team Please note how many spaces you have between ] and ". It seems there are two spaces in there. try

goaccess access.log --log-format='%h %^[%d:%t %^]  "%r" %s %b"%R" "%u"' --date-format=%d/%b/%Y --time-format=%T

Also you don't have a space between the number of bytes and the referrer, is that right?

allinurl picture allinurl on 11 Apr 2018
👍1

Closing this. Feel free to reopen it if needed.

allinurl picture allinurl on 15 Apr 2018

the spaces between ] and " causes the problems, my problem is there is two spaces between them :)

xxxatt picture xxxatt on 17 Apr 2018

Hi @allinurl hope you can assist me, did not want to open a new issue as it's related.

My log format is

1.2.3.4 - - mydomain.com [25/Jul/2018:13:35:10 +0200] "GET /includes/fontawesome/5.1.0/webfonts/fa-brands-400.woff2 HTTP/1.1" 200 63376 "https://mydomain.com/includes/fontawesome/5.1.0/css/all.css" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36 DMBrowser/2.1 (BV)" "-"PORT:443 0.000 - . "GZIP:-"

I tried this log format for goaccess but it seems to do everything except count referrers.

log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"
mitchellkrogza picture mitchellkrogza on 25 Jul 2018

@mitchellkrogza Please try:

goaccess access.log --log-format='%h %^ %^ %v %^[%d:%t %^] "%r" %s %b "%R" "%u" "%^"%^ %T %^' --date-format=%d/%b/%Y --time-format=%T
allinurl picture allinurl on 26 Jul 2018

Thanks @allinurl still giving me a count of 0 on the referrers 馃

mitchellkrogza picture mitchellkrogza on 26 Jul 2018

Here's the exact log format I am using in nginx.conf

    log_format  redirects  '$remote_addr - $remote_user $host [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"'
                  'PORT:$server_port $request_time $upstream_response_time $pipe "GZIP:$gzip_ratio"';
mitchellkrogza picture mitchellkrogza on 26 Jul 2018

@mitchellkrogza For the line you posted above, the format I posted works fine. Feel free to attach a file with several samples lines directly from your access.log and I can take a look.

allinurl picture allinurl on 26 Jul 2018

Thanks @allinurl I just noticed now my version is GoAccess - 1.0.2 so just updated now to 1.2 and looked through my goaccess.conf file and found 馃槉 I had this line uncommented ignore-panel REFERRERS now it works 100% that I commented #ignore-panel REFERRERS it out. Thanks for your help and LOVE the new look since 1.02.

mitchellkrogza picture mitchellkrogza on 26 Jul 2018
👍1

Hi allinurl,
I am using le_json log format and goaccess version 1.2. I am getting following error

Token '{' doesn't match specifier '%h'

log_format le_json '{ "time": "$time_iso8601", "remote_addr": "$http_true_client_ip", "remote_user": "$remote_user", "body_bytes_sent": "$body_bytes_sent", "request_time": "$request_time", "status": "$status", "request": "$request", "request_method": "$request_method", "http_referrer": "$http_referer", "http_user_agent": "$http_user_agent" };';

Access log 

{ "time": "2018-10-16T06:25:05+00:00", "remote_addr": "1.2.2.3", "remote_user": "-", "body_bytes_sent": "24510", "request_time": "0.010", "status": "200", "request": "GET /?xyz HTTP/1.1", "request_method": "GET", "http_referrer": "-", "http_user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 12_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/16A366" };
dodoanji picture dodoanji on 17 Oct 2018

@dodoanji This should do it:

goaccess access.log --log-format='%^: "%dT%t+%^",%^: "%h",%^: "%e",%^: "%b",%^: "%T",%^: "%s",%^: "%r",%^:%^: "%R",%^: "%u",%^' --date-format=%Y-%m-%d --time-format=%T
allinurl picture allinurl on 18 Oct 2018

Hi, I got the issue blow:
Token '10/Apr/2018' doesn't match specifier '%d'
my log format is :

$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$request_time"

and log examle is : 

36.110.199.48 - - [17/Oct/2018:23:59:56 +0800] "GET / HTTP/1.1" 200 8895 "http://yw.xmbtym.cn/" "Mozilla/5.0 (Linux; Android 6.0.1; SOV33 Build/35.0.D.0.326) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.91 Mobile Safari/537.36 baiduboxapp/10" "-" "0.000"

I work on version1.2, can you give some help?

HeyCoff picture HeyCoff on 20 Oct 2018

@HeyCoff Is your machine on a different locale than English? e.g., locale -a I've seen similar errors when the log has English dates such as Apr but your locale is not set to English. You could try:

export LANG='en_US-utf-8'
allinurl picture allinurl on 20 Oct 2018
👍1

It works to me ,Thank you! @allinurl

HeyCoff picture HeyCoff on 21 Oct 2018

Thanks allinurl, Its working fine.

dodoanji picture dodoanji on 22 Oct 2018

@allinurl I also have the same problem. I also used the nginx2goaccess.sh to try to get the log format but no result at all or errors for me.

./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$server_name" $request_time "$http_x_forwarded_for"'

time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" %T "%^"

When I run this line I get Parsing... [0] [0/s] vistors adds like 2000 lines/s

goaccess -f /var/log/nginx/access.log --log-format='%h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" %T "%^"' --date-format=%d/%b/%Y --time-format=%T

example line
192.168.0.3 - - [24/Feb/2019:10:26:56 +0100] "GET /img/myimage_1.jpg HTTP/1.1" 200 17920 "-" "Googlebot-Image/1.0" "www.mywebsite.com" 0.023 "-"

nginx conf

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$server_name" $request_time "$http_x_forwarded_for"';

I use CentOs 7 with "yum install epel-release"

Installed Packages
Name        : goaccess
Arch        : x86_64
Version     : 1.2
Release     : 1.el7
Size        : 929 k
Repo        : installed
From repo   : epel

I just got this linke working on my other server with the same config... probley only need a reboot after install to get it working...

goaccess -f /var/log/nginx/access.log --log-format='%h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" %T "%^"' --date-format=%d/%b/%Y --time-format=%T

Zewy picture Zewy on 25 Feb 2019

@Zewy Please try this, should do it:

goaccess access.log --log-format='%h %^[%d:%t %^] "%r" %s %b "%R" "%u" "%v" %T %^' --date-format=%d/%b/%Y --time-format=%T
allinurl picture allinurl on 26 Feb 2019
👍1

@allinurl

Got it working with this line after server reboot.
goaccess -f /var/log/nginx/access.log --log-format='%h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%v" %T "^%"' --date-format=%d/%b/%Y --time-format=%T

Zewy picture Zewy on 26 Feb 2019

@allinurl could I get some help with processing a custom Nginx log format please? I'm using the nginx2goaccess.sh tool to convert the log format but continue to run into parse errors.

GoAccess version: 1.3

$ goaccess access.log
Parsed 1 linesproducing the following errors:

A valid date is required.

Format Errors - Verify your log/date/time format

Sample of access.log:

$ head -n 5 access.log
127.0.0.1 - - [2019-07-31T06:33:06+00:00] "localhost" "GET /health.php HTTP/1.1" 200 5 "-" "curl/7.35.0" 0.038
127.0.0.1 - - [2019-07-31T06:33:36+00:00] "localhost" "GET /health.php HTTP/1.1" 200 5 "-" "curl/7.35.0" 0.033
127.0.0.1 - - [2019-07-31T06:34:06+00:00] "localhost" "GET /health.php HTTP/1.1" 200 5 "-" "curl/7.35.0" 0.063
127.0.0.1 - - [2019-07-31T06:34:36+00:00] "localhost" "GET /health.php HTTP/1.1" 200 5 "-" "curl/7.35.0" 0.033
127.0.0.1 - - [2019-07-31T06:35:06+00:00] "localhost" "GET /health.php HTTP/1.1" 200 5 "-" "curl/7.35.0" 0.068

Nginx log conf:

  log_format host_time_combined '$remote_addr - $remote_user [$time_iso8601] '
                                      '"$host" "$request" $status $body_bytes_sent '
                                      '"$http_referer" "$http_user_agent" '
                                      '$request_time';

nginx2goaccess.sh output:

./nginx2goaccess.sh '$remote_addr - $remote_user [$time_iso8601] "$host" "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time'

- Generated goaccess config:

time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%^8601] "%v" "%r" %s %b "%R" "%u" %T

$ cat ~/.goaccessrc
time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%^8601] "%v" "%r" %s %b "%R" "%u" %T

Thanks for any help.

reprah picture reprah on 4 Aug 2019

@reprah 

goaccess access.log --log-format='%h %^[%dT%t+%^] "%v" "%r" %s %b "%R" "%u" %T' --date-format='%Y-%m-%d' --time-format=%T
allinurl picture allinurl on 5 Aug 2019

@allinurl Thanks. Thanks for your hard work on goaccess, it's been very useful.

reprah picture reprah on 8 Aug 2019

hi. can you help me with my logformat?
Logformat:

log_format all '$remote_addr - $remote_user <$http_cookie> <$sent_http_set_cookie> [$time_local] "$host" "$request" ' '$status $body_bytes_sent "$http_referer" '
' $request_body $request_uri $uri'
' "$http_user_agent" "$http_x_forwarded_for" -> $upstream_response_time -> $request_time' ;

log example

178.22.252.111 - - <-> [19/Jun/2020:10:59:15 +0300] "tryha.ua" "GET / HTTP/1.1" 200 4789 "-" - / /index.php "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36" "-" -> 0.100 -> 0.100

Bansh1990 picture Bansh1990 on 19 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

eroldru picture eroldru  路  3Comments
LoanDEV picture LoanDEV  路  3Comments
ArunDahiya1 picture ArunDahiya1  路  3Comments
ychunc picture ychunc  路  3Comments
canepa picture canepa  路  3Comments