Goaccess: Real-time html report not reloading using Docker

Created on 7 Apr 2017  路  27Comments  路  Source: allinurl/goaccess

Hi,

here my goaccessconf file.
I can t make it _real time html with automatic refresh_. What am I missing ?

fyi, I m using the docker version of allinurl/goaccess

time-format %H:%M:%
date-format %d/%b/%

log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u" %T %^
config-dialog true
hl-header true
html-report-title dwl web stats
json-pretty-print true
no-color false
no-column-names false
no-csv-summary false
no-progress false
no-tab-scroll false
with-mouse true
addr <my public ip>
origin http://<my public ip>
real-time-html true
ws-url <my public ip>:<my port>
log-file /dwl/var/log/apache2/access.log
debug-file /srv/data/debug.log
config-file /srv/data/goaccess.conf
invalid-requests /srv/data/invalid.log
no-global-config true
agent-list false
with-output-resolver false
http-method yes
http-protocol yes
output-format /srv/data/index.html
no-query-string false
no-term-resolver false
444-as-404 false
4xx-to-unique-count false
all-static-files false
double-decode false
ignore-crawlers false
crawlers-only false
ignore-panel REFERRER
ignore-panel KEYPHRASE
real-os true
static-file .css
static-file .js
static-file .jpg
static-file .png
static-file .gif
static-file .ico
static-file .jpeg
static-file .pdf
static-file .txt
static-file .csv
static-file .zip
static-file .mp3
static-file .mp4
static-file .mpeg
static-file .mpg
static-file .exe
static-file .swf
static-file .woff
static-file .woff2
static-file .xls
static-file .xlsx
static-file .doc
static-file .docx
static-file .ppt
static-file .pptx
static-file .iso
static-file .gz
static-file .rar
static-file .svg
static-file .bmp
static-file .tar
static-file .tgz
static-file .tiff
static-file .tif
static-file .ttf
static-file .fl
geoip-database /srv/data/GeoLiteCity.dat
docker
Source
picture davask

All 27 comments

I'd try commenting out addrand origin just to be sure those are not the conflict options. Also, please take a look at your browser's console and see if you can spot some errors on the client side.

allinurl picture allinurl on 7 Apr 2017

I did it however I got this error in the console

Uncaught DOMException: Failed to construct 'WebSocket': The port 65535 is not allowed.
    at Object.setWebSocket (http://163.172.35.42:65534/:3396:16)
    at Object.initialize (http://163.172.35.42:65534/:3371:9)
    at window.onload (http://163.172.35.42:65534/:4847:11)

fyi, I access the report.html file through another container as I don t know how to access the report within the goaccess container

so port 65534 is one container
65535 is the one which link to 8080 in goaccess container

davask picture davask on 7 Apr 2017

In which port are you running GoAccess' WebSocket server? It seems like the issue is the URL to which the browser is trying to connect. You certainly don't want http in the WS constructor. So if you know the container's address and port where GoAccess is running, then I'd use that under ws-url. For instance,

ws-url ws://163.172.35.42:8080

Since GoAccess doesn't run a webserver to host the generated report, then you probably need another container serving the HTML report, but @julianxhokaxhiu probably can chime in on this, since I'm not entirely sure if a workaround exists.

allinurl picture allinurl on 7 Apr 2017

here my docker cmd:

docker run --name=goaccess --restart=always -d -p 65535:8080 \
    -v /path/to/log/apache2:/dwl/var/log/apache2 \
    -v /path/to/data:/srv/data \
    allinurl/goaccess

still the same error

davask picture davask on 7 Apr 2017

Did you replace your ws-url option in the config file? Also if you are using a different port, please make sure to add it in your goaccess' config file. Otherwise, try:

docker run --name=goaccess --restart=always -d -p 7890:7890 \
    -v /path/to/log/apache2:/dwl/var/log/apache2 \
    -v /path/to/data:/srv/data \
    allinurl/goaccess
allinurl picture allinurl on 7 Apr 2017

here the error I got now:

WebSocket connection to 'ws://163.172.35.42:7890/' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED

here the new conf file

time-format %H:%M:%S
date-format %d/%b/%Y
log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u" %T %^
config-dialog true
hl-header true
html-report-title dwl web stats
json-pretty-print true
no-color false
no-column-names false
no-csv-summary false
no-progress false
no-tab-scroll false
with-mouse true
real-time-html true
ws-url ws://163.172.35.42:7890
log-file /dwl/var/log/apache2/access.log
debug-file /srv/data/debug.log
config-file /srv/data/goaccess.conf
invalid-requests /srv/data/invalid.log
no-global-config true
agent-list false
with-output-resolver false
http-method yes
http-protocol yes
output-format /srv/data/report.html
no-query-string false
no-term-resolver false
444-as-404 false
4xx-to-unique-count false
all-static-files false
double-decode false
ignore-crawlers false
crawlers-only false
ignore-panel REFERRERS
ignore-panel KEYPHRASES
real-os true
static-file .css
static-file .js
static-file .jpg
static-file .png
static-file .gif
static-file .ico
static-file .jpeg
static-file .pdf
static-file .txt
static-file .csv
static-file .zip
static-file .mp3
static-file .mp4
static-file .mpeg
static-file .mpg
static-file .exe
static-file .swf
static-file .woff
static-file .woff2
static-file .xls
static-file .xlsx
static-file .doc
static-file .docx
static-file .ppt
static-file .pptx
static-file .iso
static-file .gz
static-file .rar
static-file .svg
static-file .bmp
static-file .tar
static-file .tgz
static-file .tiff
static-file .tif
static-file .ttf
static-file .flv
geoip-database /srv/data/GeoLiteCity.dat
davask picture davask on 7 Apr 2017

Please run the following to see your NAT tables and what Docker has done.

iptables -t nat -L -n
allinurl picture allinurl on 7 Apr 2017 
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER-INGRESS  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
CATTLE_PREROUTING  all  --  0.0.0.0/0            0.0.0.0/0
CATTLE_PREROUTING  all  --  0.0.0.0/0            0.0.0.0/0
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER-INGRESS  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match src-type LOCAL
CATTLE_POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0
MASQUERADE  tcp  --  172.17.0.3           172.17.0.3           tcp dpt:80
MASQUERADE  tcp  --  172.17.0.3           172.17.0.3           tcp dpt:22
MASQUERADE  tcp  --  172.17.0.5           172.17.0.5           tcp dpt:80
MASQUERADE  tcp  --  172.17.0.5           172.17.0.5           tcp dpt:22
MASQUERADE  tcp  --  172.17.0.6           172.17.0.6           tcp dpt:3306
MASQUERADE  tcp  --  172.17.0.7           172.17.0.7           tcp dpt:443
MASQUERADE  tcp  --  172.17.0.7           172.17.0.7           tcp dpt:80
MASQUERADE  tcp  --  172.17.0.7           172.17.0.7           tcp dpt:22
MASQUERADE  tcp  --  172.17.0.8           172.17.0.8           tcp dpt:3306
MASQUERADE  tcp  --  172.17.0.9           172.17.0.9           tcp dpt:443
MASQUERADE  tcp  --  172.17.0.9           172.17.0.9           tcp dpt:80
MASQUERADE  tcp  --  172.17.0.9           172.17.0.9           tcp dpt:22
MASQUERADE  tcp  --  172.17.0.10          172.17.0.10          tcp dpt:80
MASQUERADE  tcp  --  172.17.0.10          172.17.0.10          tcp dpt:22
MASQUERADE  tcp  --  172.17.0.12          172.17.0.12          tcp dpt:80
MASQUERADE  tcp  --  172.17.0.12          172.17.0.12          tcp dpt:22
MASQUERADE  tcp  --  172.17.0.14          172.17.0.14          tcp dpt:3306
MASQUERADE  tcp  --  172.17.0.15          172.17.0.15          tcp dpt:3306
MASQUERADE  tcp  --  172.17.0.16          172.17.0.16          tcp dpt:80
MASQUERADE  tcp  --  172.17.0.16          172.17.0.16          tcp dpt:22
MASQUERADE  tcp  --  172.17.0.17          172.17.0.17          tcp dpt:80
MASQUERADE  tcp  --  172.17.0.17          172.17.0.17          tcp dpt:22
MASQUERADE  tcp  --  172.17.0.19          172.17.0.19          tcp dpt:3306
MASQUERADE  tcp  --  172.17.0.20          172.17.0.20          tcp dpt:80
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:80
MASQUERADE  tcp  --  172.17.0.2           172.17.0.2           tcp dpt:22
MASQUERADE  tcp  --  172.17.0.4           172.17.0.4           tcp dpt:8080
MASQUERADE  tcp  --  172.17.0.4           172.17.0.4           tcp dpt:7890
MASQUERADE  tcp  --  172.17.0.21          172.17.0.21          tcp dpt:80
MASQUERADE  tcp  --  172.17.0.21          172.17.0.21          tcp dpt:22

Chain CATTLE_POSTROUTING (1 references)
target     prot opt source               destination
ACCEPT     all  --  10.42.0.0/16         169.254.169.250
MASQUERADE  tcp  --  10.42.0.0/16        !10.42.0.0/16         masq ports: 1024-65535
MASQUERADE  udp  --  10.42.0.0/16        !10.42.0.0/16         masq ports: 1024-65535
MASQUERADE  all  --  10.42.0.0/16        !10.42.0.0/16
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x31920 to:10.42.203.40
SNAT       all  -- !10.42.0.0/16         169.254.169.250      mark match 0x11a25 to:10.42.72.229
MASQUERADE  tcp  --  172.17.0.0/16        0.0.0.0/0            masq ports: 1024-65535
MASQUERADE  udp  --  172.17.0.0/16        0.0.0.0/0            masq ports: 1024-65535

Chain CATTLE_PREROUTING (2 references)
target     prot opt source               destination
DNAT       tcp  --  10.42.0.0/16         10.42.0.1            tcp dpt:53 to:169.254.169.250
DNAT       udp  --  10.42.0.0/16         10.42.0.1            udp dpt:53 to:169.254.169.250
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:FD:11:53:86:D7 MARK set 0x31920
MARK       all  -- !10.42.0.0/16         169.254.169.250      MAC 02:FD:11:82:8D:F2 MARK set 0x11a25

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:900 to:172.17.0.3:80
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:902 to:172.17.0.3:22
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:300 to:172.17.0.5:80
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:302 to:172.17.0.5:22
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:301 to:172.17.0.6:3306
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.17.0.7:443
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.17.0.7:80
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:2222 to:172.17.0.7:22
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:213 to:172.17.0.8:3306
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65043 to:172.17.0.9:443
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65040 to:172.17.0.9:80
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65042 to:172.17.0.9:22
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65021 to:172.17.0.10:80
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65022 to:172.17.0.10:22
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65031 to:172.17.0.12:80
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65032 to:172.17.0.12:22
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65033 to:172.17.0.14:3306
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65023 to:172.17.0.15:3306
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:64020 to:172.17.0.16:80
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:64022 to:172.17.0.16:22
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65011 to:172.17.0.17:80
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65012 to:172.17.0.17:22
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65013 to:172.17.0.19:3306
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:203 to:172.17.0.20:80
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:910 to:172.17.0.2:80
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:912 to:172.17.0.2:22
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65535 to:172.17.0.4:8080
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:7890 to:172.17.0.4:7890
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65534 to:172.17.0.21:80
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:65533 to:172.17.0.21:22

Chain DOCKER-INGRESS (2 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0
davask picture davask on 8 Apr 2017

First of all I would use a lower port number ( say 9000 ), inside the docker ( so set it in your goaccess conf file ).

Second I would use the same HTTP port you use to connect to the web interface, like 80 or 443, in order to avoid any firewall issue that may be living in your distribution.

Third, you should be aware that browsers sometimes locks you from using certain ports, because of their special needs. So really, go for 80 or 443 :)

julianxhokaxhiu picture julianxhokaxhiu on 8 Apr 2017

For my understanding:

  • what is 7890 port for ?
  • what is 8080 port for ?
  • How can I debug a ws url - ws://ip:port ?

FYI, here what I understood about the conf required.

I need 2 + x containers :

  • x containers from which I'll externalize logs and provide them to goaccess
  • 1 for goaccess which will access the logs I provide to it and generate the report.html
  • 1 to access the report.html over http(s)

Am I right ?

davask picture davask on 8 Apr 2017

To answer your suggestion @julianxhokaxhiu, To which port should I assign 9000 ? 7890 or 8080 ?
Am I missing something in the documentation ?

davask picture davask on 8 Apr 2017

You should really go for a reverse proxy logic honestly. See https://github.com/julianxhokaxhiu/vps-powered-by-docker as an example. You can use the goaccess module located here.

Then doesn't matter which port do you use. The traffic to the final browser goes towards port 80 or 443. Internally you can route as you like.

julianxhokaxhiu picture julianxhokaxhiu on 9 Apr 2017

You should really go for a reverse proxy logic honestly. - Actually that s what I do once I ve validated some new installation and That s what I did following your advises (but I ll definitly look at your reverse proxy)

Howver could you tell me:

  • How can I debug a ws url - ws://ip:port ?
davask picture davask on 10 Apr 2017

What do you mean my debugging sorry? If you want to debug your application behind WS connection, then you may use your preferred IDE.

If you want to know if it works, then it does or it does not. There are no gray shades.

julianxhokaxhiu picture julianxhokaxhiu on 10 Apr 2017
👍1

Just adding to @julianxhokaxhiu's reponse, you can also try using telnet from an external network, e.g.,

telnet <host> <port>
allinurl picture allinurl on 10 Apr 2017
👍1

@davask let me play locally with your config file and I'll post back.

allinurl picture allinurl on 11 Apr 2017

@allinurl thanks.
do you need me to send it again ?

davask picture davask on 11 Apr 2017

@davask Unless you already changed some settings, otherwise, feel free to update the one above.

allinurl picture allinurl on 11 Apr 2017

@allinurl you can go like that :)
thks

davask picture davask on 11 Apr 2017

@davask Please take a look at this section, I've added some instructions on how to run the docker image. Let me know how it goes.

allinurl picture allinurl on 12 Apr 2017
👍1

Is it possible to have a generic binding over ws-url or to auto-guess it from goaccess?

The way to do it, makes goaccess not really scalable :/

julianxhokaxhiu picture julianxhokaxhiu on 12 Apr 2017
👍1

@allinurl, I do agree with @julianxhokaxhiu

davask picture davask on 12 Apr 2017

@julianxhokaxhiu Upon opening the generated report in your browser, the report will try to establish a WebSocket connection to the host name (window.location.hostname) of the generated report. That solves the issue of not needing to use --ws-url for most cases.

Now in the case of docker, if you are serving the report from a different container (unless supervisor is being used to run the WebServer on the same container) then you probably need to set --ws-url since the WebSocket server is not running on the same container as window.location.hostname. Now to make this easier from Docker's side, it would be great though if you could simply override GoAccess' config file by passing command line options such as ws-url when docker is ran. e.g.,

docker run -d -p 7890:7890 allinurl/goaccess goaccess --ws-url=xzy --log-format=VCOMBINED etc...
allinurl picture allinurl on 12 Apr 2017

Yeah, that's part of the custom goaccess.conf file, so I would assume that who wants to use Web Sockets, also is willing to place his own custom config file. With that argument inside :)

julianxhokaxhiu picture julianxhokaxhiu on 12 Apr 2017

@julianxhokaxhiu That I agree.

@davask and @julianxhokaxhiu, I just verified this and even if you leave ws-url empty in v1.2, the generated report will successfully establish a connection to the WebSocket server.

This occurs since Docker will bind to 0.0.0.0:7890, which means that GoAccess WebSocket server is using port 7890 and reachable from 127.0.0.1 in addition to your host IP.

Also, if you ever need to run it on a different port, e.g.,

docker run --restart=always -d -p 8080:7890 ...

Then you can simply set the external port within ws-url. e.g., ws-url ws://localhost:8080 and keep GoAccess' internal port in your config file set to port 7890. That should do it.

allinurl picture allinurl on 13 Apr 2017
👍1

Nice, even better :D no black magic, just scalable awesomeness

julianxhokaxhiu picture julianxhokaxhiu on 13 Apr 2017

Hi,

Super, this work like a charm! thanks @allinurl

here my new working conf

time-format %H:%M:%S
date-format %d/%b/%Y
log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u" %T %^
config-dialog true
hl-header true
html-report-title dwl web stats
json-pretty-print true
no-color false
no-column-names false
no-csv-summary false
no-progress false
no-tab-scroll false
with-mouse true
real-time-html true
ws-url ws://<my.ip.server>:<my.server.port bind to container server port 7890>
log-file /path/to/log/apache2/access.log
debug-file /srv/data/debug.log
config-file /srv/data/goaccess.conf
invalid-requests /srv/data/invalid.log
no-global-config true
agent-list false
with-output-resolver false
http-method yes
http-protocol yes
output-format /srv/data/index.html
no-query-string false
no-term-resolver false
444-as-404 false
4xx-to-unique-count false
all-static-files false
double-decode false
ignore-crawlers false
crawlers-only false
ignore-panel REFERRERS
ignore-panel KEYPHRASES
real-os true
static-file .css
static-file .js
static-file .jpg
static-file .png
static-file .gif
static-file .ico
static-file .jpeg
static-file .pdf
static-file .txt
static-file .csv
static-file .zip
static-file .mp3
static-file .mp4
static-file .mpeg
static-file .mpg
static-file .exe
static-file .swf
static-file .woff
static-file .woff2
static-file .xls
static-file .xlsx
static-file .doc
static-file .docx
static-file .ppt
static-file .pptx
static-file .iso
static-file .gz
static-file .rar
static-file .svg
static-file .bmp
static-file .tar
static-file .tgz
static-file .tiff
static-file .tif
static-file .ttf
static-file .flv
davask picture davask on 13 Apr 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

LoanDEV picture LoanDEV  路  3Comments
Gill-Bates picture Gill-Bates  路  3Comments
Tassleh0ff picture Tassleh0ff  路  4Comments
ychunc picture ychunc  路  3Comments
canepa picture canepa  路  3Comments