Goaccess: Log format for HAProxy log files

Created on 17 Feb 2016 · 16Comments · Source: allinurl/goaccess

Hi i am trying to use the goaccess tool but it keeps saying invalid file format
so my file format is the default http log format for haproxy (http://www.haproxy.org/download/1.5/doc/configuration.txt)

Example :
frontend http-in
mode http
option httplog
log global
default_backend bck

    backend static
        server srv1 127.0.0.1:8000

>>> Feb  6 12:14:14 localhost \
      haproxy[14389]: 10.0.1.2:33317 [06/Feb/2009:12:14:14.655] http-in \
      static/srv1 10/0/30/69/109 200 2750 - - ---- 1/1/1/1/0 0/0 {1wt.eu} \
      {} "GET /index.html HTTP/1.1"

  Field   Format                                Extract from the example above
      1   process_name '[' pid ']:'                            haproxy[14389]:
      2   client_ip ':' client_port                             10.0.1.2:33317
      3   '[' accept_date ']'                       [06/Feb/2009:12:14:14.655]
      4   frontend_name                                                http-in
      5   backend_name '/' server_name                             static/srv1
      6   Tq '/' Tw '/' Tc '/' Tr '/' Tt*                       10/0/30/69/109
      7   status_code                                                      200
      8   bytes_read*                                                     2750
      9   captured_request_cookie                                            -
     10   captured_response_cookie                                           -
     11   termination_state                                               ----
     12   actconn '/' feconn '/' beconn '/' srv_conn '/' retries*    1/1/1/1/0
     13   srv_queue '/' backend_queue                                      0/0
     14   '{' captured_request_headers* '}'                   {haproxy.1wt.eu}
     15   '{' captured_response_headers* '}'                                {}
     16   '"' http_request '"'                      "GET /index.html HTTP/1.1"

how should i run the goaccess tool to make it work for my log file
currently i am running it with this and

goaccess --log-format='%Ci:%Cp [%t] %ft %b/%s %Tq/%Tw/%Tc/%Tr/%Tt %st %B %cc %cs %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r' -f ../haproxy.log

I get this error:

GoAccess - version 0.9.7 - Feb 17 2016 19:59:42
Config file: /usr/local/etc/goaccess.conf

Fatal error has occurred
Error occured at: src/parser.c - verify_formats - 1960
No time format was found on your conf file.

I just want to use this to analyze the number of connections per second on the load balancer

question

Source

amolkedar

Most helpful comment

The following should do it for the line you posted above:

goaccess -f access.log --log-format='%^]%^ %h:%^ [%d:%t.%^] %^/%^/%^/%^/%L/%^ %s %b %^"%r"' --date-format='%d/%b/%Y' --time-format='%H:%M:%S'

allinurl on 18 Feb 2016

👍3

All 16 comments

You need date-format and time-format as well.

Could you please post a few lines straight from your log so I can take a look?

allinurl on 18 Feb 2016

here is my log entry

Feb 16 06:51:40 machine-name haproxy[15248]: 10.0.10.10:52011 [16/Feb/2016:06:51:39.884] www-https~ www-backend/app-name-app5 257/0/1/190/448 200 2852 - - --VN 7/7/1/0/0 0/0 "POST /file/_fileJSON.php?t=cc045448ab902fef890 HTTP/1.1"

amolkedar on 18 Feb 2016

The following should do it for the line you posted above:

goaccess -f access.log --log-format='%^]%^ %h:%^ [%d:%t.%^] %^/%^/%^/%^/%L/%^ %s %b %^"%r"' --date-format='%d/%b/%Y' --time-format='%H:%M:%S'

allinurl on 18 Feb 2016

👍3

ok thanks that helped a lot and i can now get data from the logfile . but a side question is that what if i want to see the highest number of connections at any given time in the log file which is represented by
7/7/1/0/0
how can i analyze that data?

Feb 16 06:51:40 machine-name haproxy[15248]: 10.0.10.10:52011 [16/Feb/2016:06:51:39.884] www-https~ www-backend/app-name-app5 257/0/1/190/448 200 2852 - - --VN 7/7/1/0/0 0/0 "POST /file/_fileJSON.php?t=cc045448ab902fef890 HTTP/1.1"

amolkedar on 18 Feb 2016

There's no panel that will display those metrics. You 'could' append the whole string to the virtual host panel, however, I'm not sure if that would help.

goaccess -f access.log --log-format='%^]%^ %h:%^ [%d:%t.%^] %^/%^/%^/%^/%L/%^ %s %b %^ %^ %^ %v "%r"' --date-format='%d/%b/%Y' --time-format='%H:%M:%S'

allinurl on 18 Feb 2016

Thanks a lot. This is quite helpful atleast sends me the right direction :)

amolkedar on 18 Feb 2016

the report is generated using this log formate but I am not getting proper browser hit in the report.