Goaccess: nginx custom log format parsing ?

[centminmod]

Hi my custom nginx log format is as follows

log_format  main_ext '$remote_addr - $remote_user [$time_local] "$request" '
                         '$status $body_bytes_sent "$http_referer" '
                         '"$http_user_agent" "$http_x_forwarded_for" '
                         'rt=$request_time ua="$upstream_addr" '
                         'us="$upstream_status" ut="$upstream_response_time" '
                         'ul="$upstream_response_length" '
                         'cs=$upstream_cache_status' ;

with example access log entry below

IPADDRESS - - [22/Feb/2017:22:59:47 +0000] "GET / HTTP/1.1" 200 6050 "-" "Mozilla/5.0 (unknown-x86_64-linux-gnu) Siege/4.0.2" "-" rt=0.000 ua="-" us="-" ut="-" ul="-" cs=-

how would i define log-format for goaccess.conf ?

thanks

[centminmod]

i found https://github.com/stockrt/nginx2goaccess so

so that gave me

./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" '  '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for" ' 'rt=$request_time ua="$upstream_addr" ' 'us="$upstream_status" ut="$upstream_response_time" ' 'ul="$upstream_response_length" ' 'cs=$upstream_cache_status' 

- Generated goaccess config:

time-format %T
date-format %d/%b/%Y
log_format %h - %^ [%d:%t %^] "%r" 

so maybe ?

log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u"

[allinurl]

Can you attach a few lines from your access log? Also, are you using the x_forwarded_for field or are you getting the client from remote_addr? Thanks

[allinurl]

Closing this. Feel free to reopen it if needed.

[allinurl]

@azamatgithub Assuming you have a valid address.

goaccess access.log --log-format='%dT%t+%^\t%h\t%^\t"%U"\t"%q"\t"%^"\t%s\t%b\t%T\t%^' --date-format=%Y-%m-%d --time-format=%T --ignore-panel=BROWSERS --ignore-panel=OS --ignore-panel=REFERRING_SITES --http-method=no --http-protocol=no

[allinurl]

@azamatgithub I'd use the in-memory storage instead of on-disk.

[cassianotartari]

@azamatgithub Assuming you have a valid address.

goaccess access.log --log-format='%dT%t+%^\t%h\t%^\t"%U"\t"%q"\t"%^"\t%s\t%b\t%T\t%^' --date-format=%Y-%m-%d --time-format=%T --ignore-panel=BROWSERS --ignore-panel=OS --ignore-panel=REFERRING_SITES --http-method=no --http-protocol=no

I've tried this for this sample:

186.249.192.208 - - [20/Sep/2019:15:44:36 -0200] "GET /loja/teste.com.br/produto/busca?q=testecriando HTTP/2.0" 200 6121 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-" "marketplace.xxx.com" sn="xxx.com" rt=1.073 ua="unix:/var/run/php5-ftwo1.sock" us="200" ut="1.073" ul="33568" cs=-
201.33.202.111 - - [20/Sep/2019:15:44:36 -0200] "GET /lista-de-desejos/editar HTTP/2.0" 200 14562 "https://hidroazul.xxx.com/lista-de-desejos/editar" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" "-" "hidroazul.xxx.com" sn="xxx.com" rt=0.461 ua="unix:/var/run/php5-ftwo1.sock" us="200" ut="0.461" ul="73864" cs=-
201.16.160.153 - - [20/Sep/2019:15:44:36 -0200] "GET /admin/order/list?query=238299 HTTP/2.0" 200 51554 "https://yyyy.xxx.com/admin/order/list?query=235685" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" "-" "yyyy.xxx.com" sn="xxx.com" rt=41.050 ua="unix:/var/run/php5-ftwo3.sock" us="200" ut="40.743" ul="267944" cs=-
177.21.200.45 - - [20/Sep/2019:15:44:36 -0200] "GET /admin/order/list HTTP/2.0" 499 0 "https://yyyy.xxx.com/admin/order/262609" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.145 Safari/537.36 Vivaldi/2.6.1566.49" "-" "yyyy.xxx.com" sn="xxx.com" rt=3.009 ua="unix:/var/run/php5-ftwo3.sock" us="-" ut="-" ul="0" cs=-

But I've got:

Parsed 1 linesproducing the following errors:

Token '177.76.148.24 - - [20/Sep/2019:15:44:21 -0200] "GE' doesn't match specifier '%d'

Format Errors - Verify your log/date/time format

@allinurl can you help me?

This log format is from amplify docs

[cassianotartari]

I think I figure it out:

goaccess access.log --log-format='%h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" "%v" sn="%^" rt=%T ua="%^" us="%^" ut="%^" ul="%^" cs=%^' --date-format=%d/%b/%Y --time-format=%T

[allinurl]

@cassianotartari please try:

goaccess access.log --log-format='%h %^[%d:%t %^] "%r" %s %b "%R" "%u" %^ %v %^ %^=%T %^' --date-format=%d/%b/%Y --time-format=%T