Goaccess: Ability to filter dataset by fields or regex

[allinurl]

Add the ability to filter the results within the UI (Terminal & HTML) - e.g. filter by fields such as host, request, etc. then display only data matching that filter criteria, or enter a regex to match in the request and restrict display to only those matching entries.

Ideally this would spin up a new thread so multiple datasets can be analyzed at the same time. Each dataset should live on its own dashboard.

[aphorise]

Out of curiosity - is this functionality and those referenced / related intended for the TUI only?

[allinurl]

Good question, the original thought was to make these filters for the terminal, however, I didn't think much about having them available in the HTML output.

Not sure yet how this would work, perhaps allowing the user to set initial filters in the config file or since there are plans to have the HTML output be real-time, have some sort of subset filtering in the client side. Any thoughts?

[aphorise]

A small related note - I think most of the rapid requests that are coming in for additional functionality, aggregation and related UI - would be better grouped in a separate argument. For example:

--rui 'regex,average_files,average_hits,host_servers...'

for Rich-User-Interface. Thereafter and into the future it can be included as part of standard views if its common to most user expectations or perhaps adaptively enabled based on the log-file and the scheme therein that matches RUI options.

Regarding HTML - if you dont mind using jQuery & DataTables then for the specific purposes of sort / filter I'd recommend:
http://datatables.net/examples/api/regex.html

Its a 160 Kbyte addition in javascript but worth it for what it does.
This would also give us a footing into other light / efficient jQ based libraries for additional UI and eye-candy as required.

If however you do not wish to have such dependencies - then we have our work cut out :-D

[gitanupam]

Is there a way to see today which IP visited which pages? I know this ticket might help achieve that in the future , but until this feature is added, is there a workaround to do that today? BTW, thanks heaps for this tool.

[allinurl]

@gitanupam while this is implemented, grep or any other filtering tool would be your friend here. e.g.,

For real-time filtering:

# tail -f -n +0 access.log | grep --line-buffered '192.168.3.1' | goaccess --log-format=COMBINED -

or for static filtering, simply

# grep '192.168.3.1' access.log | goaccess --log-format=COMBINED -

[rauschma]

It’ll be great to have this! It would allow one to dig into a single day (to check how traffic varies during that day).

[Dadibom]

+1 for date/time range filtering

[BirkhoffLee]

+1

[hiaselhans]

+1

[Shagon94]

not sure if this feature was deprioritized but would love to have this feature, seeing as this was opened 5 years ago I wanted to check if this is still being worked on?

[allinurl]

@Shagon94 certainly still on top of the list. however, there's an outstanding issue with the on-disk storage that needs to be worked on before this. stay tuned though.

[dmaziuk]

FWIW I'd stick the data into sqlite3 table(s) and use sql to do the filtering. Better yet, add a mini-abstraction-layer so people with huge amount of logs could use a grown-up sql engine (postgres) for storage too.

[allinurl]

@dmaziuk agree on that. Stay tuned for the upcoming storage change.

[nkvname]

Any updates regarding this? Would be already good, if there is a possibility to toggle between daily, weekly and monthly stats. Going back to AWStats hurts.

[dmaziuk]

awstats? Luxury! I'm thinking wrapping our analog plus report magic setup in a docker container...

[allinurl]

@nkvname need to finish the on-disk storage replacement, this is second on the list though.

[domolicious]

This would be a fantastic enhancement! Is there an issue for the on-disk storage replacement, if so, can you please link it?

[allinurl]

@domolicious I agree, this would be of great value! Issue #1274 Stay tuned!

[impactivize-hq]

@allinurl awesome work so far. excited about the new release.

Will the new release help us to do the following ?

• filter data in a table using data from other table (for eg. can we filter the list of requested urls using ip address or date/time

Right now, the information is there but the contextual information is missing.

It would be useful to have all the fields in a drop-down to filter all tables in different contexts.

[allinurl]

@insidesmart This issue will add that feature, it won't be part of the upcoming release (v1.4). I've addressed the storage issue mentioned before, so after deploying 1.4 I can focus on this issue.

[manix]

+1

[pstranghoener]

@allinurl When will the time range filter function been released?

We could make a donation or sponsor that if it would help.