goaccess not running in --real-time-html

[muagkov]

Hello, I have a problem with "--real-time-html"

when i use command:
root@pa:~# goaccess -f /var/log/apache2/access.log -a -o /var/www/html/index.html --real-time-html
i get in terminal the following:
root@pa:~# goaccess -f /var/log/apache2/access.log -a -o /var/www/html/index.html --real-time-html WebSocket server ready to accept new client connections

but on my index.html i have static page with "Last Updated" when i use a command, not real time page.

so when i use command root@pa:~# goaccess -f /var/log/apache2/access.log -a -o /var/www/html/index.html --real-time-html i have the same result like root@pa:~# goaccess -f /var/log/apache2/access.log -a -o /var/www/html/index.html

Please, help me to make it real time

[allinurl]

Can you please post the output of your browser's console. Thanks.

[muagkov]

after some period of time there is an error in browser's console :
WebSocket network error: The operation couldn’t be completed. Operation timed out

(index):1210 WebSocket connection to 'ws://104.199.XXX.XXX:7890/' failed: Error in connection establishment: net::ERR_CONNECTION_TIMED_OUT

[allinurl]

Sounds like it's not able to reach the server on 104.199.XXX.XXX:7890. Is this the right IP address where GoAccess is running? Also, since it listens on port 7890, please make sure the port is opened.

If you are opening the report through https, you will need to run goaccess with --ssl-cert=<cert.crt> and --ssl-key=<priv.key>.

[allinurl]

Were you able to output real-time?

[muagkov]

no, i wasn't :(

is this the right IP address where GoAccess is running?

• yes

Also, since it listens on port 7890, please make sure the port is opened.

• it is open and i changed to others ports - result the same

If you are opening the report through https, you will need to run goaccess with --ssl-cert= and --ssl-key=.

• I'm not use https

[allinurl]

Are you behind some sort of proxy? can you please try telnet'ing to that IP and that port? Also, can you run the following and post the output here?

lsof -i -n -P

[sajjadalis]

Hi,

I have same issue. Static html page won't update auto or even after refreshing. I checked console and it shows this error.

WebSocket connection to 'ws://sh***************.com:7890/' failed: Error in connection establishment: net::ERR_CONNECTION_TIMED_OUT

I used this command to generate real time report.
goaccess -f /srv/users/serverpilot/log/sh***************/sh***************_nginx.access.log --log-format=COMBINED -o /srv/users/serverpilot/apps/sh***************/public/report.html --real-time-html

I opened port 7890 with this command.
sudo ufw allow 7890
http://prntscr.com/eyw3f9

But still same issue. I'm using serverpilot.io and cloudflare. Its using nginx in front of apache on server. here is result for lsof -i -n -P

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME dhclient 507 root 6u IPv4 10989 0t0 UDP *:68 sshd 710 root 3u IPv4 34347 0t0 TCP *:22 (LISTEN) sshd 710 root 4u IPv6 34349 0t0 TCP *:22 (LISTEN) sshd 1052 root 3u IPv4 314476 0t0 TCP 205.***.***.***:22-> 182.185.**.***:12797 (ESTABLISHED) sp-agent 2003 root 4u IPv4 289138 0t0 TCP 205.***.***.***:3037 9->35.186.205.226:443 (ESTABLISHED) sp-agent 2003 root 6u IPv4 345957 0t0 TCP 205.***.***.***:3430 6->104.198.211.92:443 (ESTABLISHED) sp-stats 2373 root 4u IPv4 353412 0t0 TCP 205.***.***.***:3462 8->104.198.211.92:443 (ESTABLISHED) sshd 2411 root 3u IPv4 343883 0t0 TCP 205.***.***.***:22-> 182.185.**.***:1068 (ESTABLISHED) monitor 2493 root 3u IPv4 128167 0t0 TCP 205.***.***.***:5513 5->104.154.33.168:2010 (ESTABLISHED) sshd 2610 root 3u IPv4 348214 0t0 TCP 205.***.***.***:22-> 182.185.**.***:2418 (ESTABLISHED) sshd 2873 root 3u IPv4 353164 0t0 TCP 205.***.***.***:22-> 118.212.132.227:43551 (ESTABLISHED) sshd 2874 sshd 3u IPv4 353164 0t0 TCP 205.***.***.***:22-> 118.212.132.227:43551 (ESTABLISHED) sshd 2893 root 3u IPv4 353506 0t0 TCP 205.***.***.***:22-> 59.45.175.56:43464 (ESTABLISHED) sshd 2894 sshd 3u IPv4 353506 0t0 TCP 205.***.***.***:22-> 59.45.175.56:43464 (ESTABLISHED) master 3132 root 12u IPv4 23361 0t0 TCP 127.0.0.1:25 (LISTEN ) mysqld 3868 mysql 17u IPv4 25934 0t0 TCP 127.0.0.1:3306 (LIST EN) httpd 4857 root 3u IPv4 28033 0t0 TCP 127.0.0.1:81 (LISTEN ) nginx 5011 root 17u IPv4 35685 0t0 TCP *:80 (LISTEN) nginx 5011 root 18u IPv6 35686 0t0 TCP *:80 (LISTEN) nginx 29952 www-data 3u IPv4 353422 0t0 TCP 205.***.***.***:80-> 162.158.79.165:30112 (ESTABLISHED) nginx 29952 www-data 15u IPv4 353007 0t0 TCP 205.***.***.***:80-> 162.158.92.50:25877 (ESTABLISHED) nginx 29952 www-data 16u IPv4 353683 0t0 TCP 205.***.***.***:80-> 162.158.88.192:14037 (ESTABLISHED) nginx 29952 www-data 17u IPv4 35685 0t0 TCP *:80 (LISTEN) nginx 29952 www-data 18u IPv6 35686 0t0 TCP *:80 (LISTEN) nginx 29952 www-data 19u IPv4 353144 0t0 TCP 205.***.***.***:80-> 162.158.91.3:33829 (ESTABLISHED) nginx 29952 www-data 20u IPv4 353440 0t0 TCP 205.***.***.***:80-> 162.158.91.237:28868 (ESTABLISHED) nginx 29953 www-data 3u IPv4 353295 0t0 TCP 205.***.***.***:80-> 172.68.58.45:25656 (ESTABLISHED) nginx 29953 www-data 8u IPv4 353080 0t0 TCP 205.***.***.***:80-> 162.158.91.231:20428 (ESTABLISHED) nginx 29953 www-data 13u IPv4 353597 0t0 TCP 205.***.***.***:80-> 162.158.91.237:26638 (ESTABLISHED) nginx 29953 www-data 17u IPv4 35685 0t0 TCP *:80 (LISTEN) nginx 29953 www-data 18u IPv6 35686 0t0 TCP *:80 (LISTEN) nginx 29953 www-data 19u IPv4 353695 0t0 TCP 205.***.***.***:80-> 162.158.92.14:18796 (ESTABLISHED) nginx 29954 www-data 3u IPv4 353024 0t0 TCP 205.***.***.***:80-> 162.158.92.86:35243 (ESTABLISHED) nginx 29954 www-data 6u IPv4 353313 0t0 TCP 205.***.***.***:80-> 162.158.69.127:31977 (ESTABLISHED) nginx 29954 www-data 7u IPv4 353211 0t0 TCP 205.***.***.***:80-> 162.158.69.127:24322 (ESTABLISHED) nginx 29954 www-data 8u IPv4 353655 0t0 TCP 205.***.***.***:80-> 162.158.91.237:14710 (ESTABLISHED) nginx 29954 www-data 14u IPv4 353687 0t0 TCP 205.***.***.***:80-> 162.158.88.60:13097 (ESTABLISHED) nginx 29954 www-data 15u IPv4 353691 0t0 TCP 205.***.***.***:80-> 162.158.89.95:19092 (ESTABLISHED) nginx 29954 www-data 17u IPv4 35685 0t0 TCP *:80 (LISTEN) nginx 29954 www-data 18u IPv6 35686 0t0 TCP *:80 (LISTEN) nginx 29955 www-data 3u IPv4 353627 0t0 TCP 205.***.***.***:80-> 162.158.69.127:11895 (ESTABLISHED) nginx 29955 www-data 6u IPv4 353639 0t0 TCP 205.***.***.***:80-> 162.158.91.237:19781 (ESTABLISHED) nginx 29955 www-data 7u IPv4 353065 0t0 TCP 205.***.***.***:80-> 162.158.89.227:26824 (ESTABLISHED) nginx 29955 www-data 8u IPv4 353193 0t0 TCP 205.***.***.***:80-> 162.158.74.133:14209 (ESTABLISHED) nginx 29955 www-data 16u IPv4 353651 0t0 TCP 205.***.***.***:80-> 162.158.90.142:28079 (ESTABLISHED) nginx 29955 www-data 17u IPv4 35685 0t0 TCP *:80 (LISTEN) nginx 29955 www-data 18u IPv6 35686 0t0 TCP *:80 (LISTEN) httpd 29962 www-data 3u IPv4 28033 0t0 TCP 127.0.0.1:81 (LISTEN ) httpd 29963 www-data 3u IPv4 28033 0t0 TCP 127.0.0.1:81 (LISTEN ) httpd 29964 www-data 3u IPv4 28033 0t0 TCP 127.0.0.1:81 (LISTEN ) httpd 30053 www-data 3u IPv4 28033 0t0 TCP 127.0.0.1:81 (LISTEN ) sshd 32051 root 3u IPv4 292215 0t0 TCP 205.***.***.***:22-> 182.185.**.***:11091 (ESTABLISHED) sshd 32060 serverpilot 3u IPv4 292215 0t0 TCP 205.***.***.***:22-> 182.185.**.***:11091 (ESTABLISHED) sshd 32274 root 3u IPv4 297153 0t0 TCP 205.***.***.***:22-> 182.185.**.***:11580 (ESTABLISHED)

[allinurl]

@Key7i Did you run lsof -i -n -P while goaccess was running? It should show up on the list, e.g.,

goaccess 1157 root 30u IPv4 3093 0t0 TCP *:7890 (LISTEN)

Also, if the machine running goaccess is reachable from the outside world through 205.***.***.***, then you can try specifying it in the command line with --ws-url:

goaccess /srv/users/serverpilot/log/site/site_nginx.access.log --log-format=COMBINED -o /srv/users/serverpilot/apps/site/public/report.html --real-time-html --ws-url=ws://205.***.***.***:7890

[sajjadalis]

Thanks for quick response.

I used this command and it's working fine now.
goaccess /srv/users/serverpilot/log/site/site_nginx.access.log --log-format=COMBINED -o /srv/users/serverpilot/apps/site/public/report.html --real-time-html --ws-url=ws://205.***.***.***:7890

A stupid question maybe. How can i run this real-time report permanently? And after running this real-time command how can i keep using terminal?

Thanks for this great tool.

[allinurl]

Glad that worked :) Usually it's best to use the public IP with --ws-url rather than the hostname just because some weird stuff can occur with the host not pointing to the right machine. For instance,

--ws-url=ws://205.***.***.***:7890

Great question, from v1.2 you can pass --daemonize, goaccess will keep running and you will be able to use the terminal as usual.

@muagkov please take a look at my previous response since @Key7i had the same issue. Feel free to reopen it if needed.

[trancef0rmer]

I have the same issue. Tried all written above. nothing helped. Got the error : WebSocket connection to wss://ip:7890/ failed: WebSocket opening handshake timed out.
I'm not using https, no proxy, opened port
cmd: goaccess logfile.log -o public_html/index.html --real-time-report --ws-url=wss://host_ip

What else could I try? thank you

[allinurl]

@trancef0rmer If you are using the latest version, you can try without --ws-url

goaccess logfile.log -o public_html/index.html --real-time-html

[trancef0rmer]

I'm using v.1.2
with no --ws-url I got only the loading gif in the middle of the page

[allinurl]

@trancef0rmer what's the output from your browser's console?

[trancef0rmer]

It was Mixed Content. I tried over https.
With no --ws-url, It works just fine ofer http )). Thank you!
But how can I handle error with mixt content over https ?
Should I indicate the cert and the key ?

[allinurl]

@trancef0rmer You will need to use --ws-url=wss://host as you had it before and pass a --ssl-cert=<path/cert.crt> --ssl-key=<path/priv.key> or make sure everything runs on http.

[trancef0rmer]

It works !!
Thank you allinurl !!

[trancef0rmer]

Should we wait for some integration with grafana ? TY

[allinurl]

@trancef0rmer Glad that worked :)

Should there be an integration with grafana? What are you looking to do with goaccess and grafana?

[trancef0rmer]

Just asked ))
Thank you Gerardo !
GoAccess is really a great thing !

[allinurl]

@trancef0rmer np, always happy to help :)

[trancef0rmer]

One more question
It is a good idea to copy apache log format to goaccess config ?
In apache I have : "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\""
If so, why I'm getting : Token '+0300]' doesn't match specifier '%h' ?

[allinurl]

@trancef0rmer no, goaccess and apache use different log formats. Please take a look at the man page for details on the specifiers.

[trancef0rmer]

Hi, @allinurl !
Sorry about bothering you with stupid questions, but I can not get rid of Token '+0300]' doesn't match specifier '%h' error.
This is my log:
mydomain.com:443 187.136.215.148 - - [22/May/2018:11:07:00 +0300] "POST /email/?_task=mail&_action=refresh HTTP/1.1" 200 992 "https://mydomain.com/email/?_task=mail&_mbox=INBOX" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"
This is my log format:
log-format "%v:%^ %h %^ %^ [%d:%t %^] \"%r\" %s %b "%R" "%u""
time-format %T
date-format %d/%b/%Y
What am I missing ? Thank you !

[allinurl]

@trancef0rmer This should do it:

 goaccess access.log --log-format=VCOMBINED

[trancef0rmer]

Same thing (

[allinurl]

@trancef0rmer for the line you posted above, it does work for me. You can try:

goaccess access.log --log-format=VCOMBINED --no-global-config

Does it work parsing that single line?

[trancef0rmer]

thank you @allinurl
I have noticed that is some lines of my log file, virtualhost and port ( %v:%p ) are missing
Apache logs monit like that
142.115.381.157 - - [23/May/2018:08:58:45 +0300] "GET / HTTP/1.1" 301 437 "-" "Monit/5.9"
These lines don't allow me to parse log file.
There is a way to ignore them ?
Thank you

[allinurl]

@trancef0rmer You can try using --num-tests=0 to parse the log and try to ignore them.

[trancef0rmer]

thanks @allinurl, it works with --num-test-0.
However, I think it's not parsing correctly, because unique visitors panel and referring sites panel are not showing anything, and browsers and OSes are all unknown.

[allinurl]

@trancef0rmer can you please attach your access log with a substantial number of lines on it?

[trancef0rmer]

@allinurl
https://pastebin.com/raw/rmhx2q6s

[trancef0rmer]

I just set this custom
log-format "%v:%^ %h %^ %^ [%d:%t %^] \"%r\" %s %b "%R" "%u"", instead of CLF with VHOST
and referring, browsers and OS appeared.
I don't get what's wrong with that http://prntscr.com/jlmhlh

[allinurl]

This works perfectly fine for me with the log you posted above, please try:

# wget https://pastebin.com/raw/rmhx2q6s
# goaccess rmhx2q6s --log-format=VCOMBINED

[trancef0rmer]

@allinurl
Tried. It's working, but in real time html that panel is not working as I expected.
I mean
http://prntscr.com/jlnj48
Maybe I need more log data ?

[allinurl]

@trancef0rmer that's right, you only have one day worth of data. In a month and if you are not rotating your log, then your report will look similar to rt.goaccess.io.

[trancef0rmer]

@allinurl
That makes sense.
Finally I can enjoy reading my apache logs )
Thank you for help.
Nice tool !

[trancef0rmer]

Hello, @allinurl
I am using vlogger for splitting apache logs for each vhost, but the output log of that is missing %v. Is there a way for goaccess to sort logs by vhost only by reading access.log path (which contains vhost name) ? TY

[allinurl]

@trancef0rmer Please take a look at this https://github.com/allinurl/goaccess/issues/673

[trancef0rmer]

Thanks @allinurl,
I guess it is modifying access.log on the fly, before goaccess is reading it.
The difference is that my virtual host name is present in file path, not in file name, but I will try to modify the awk command. ty

[trancef0rmer]

Oh, I see now. It's FILENAME, anyway)

[allinurl]

@trancef0rmer you can definitely do it with awk. I believe there's an example in the man page where you can extract a field and place it somewhere else.