Owasp-modsecurity-crs: Use the "HTTP DATASET CSIC 2010" to measure the quality of our rules

[dune73]

I stumbled over this set here: http://www.isi.csic.es/dataset/

It seems to be a standard data set used in the scientific community to measure the effectiveness of a WAF. Payloads are tagged as benign and malicious.

[EDIT: Backup link: https://gitlab.fing.edu.uy/gsi/web-application-attacks-datasets]

[rodrigo-martinez]

Hi,
I've been working with machine learning techniques to detect/prevent attacks to Web Applications. In order to experiment with these techniques, one important input is datasets with requests tagged as valid or attack. In this research, I found two dataset that comply with these requirements: the one mention by @dune73 and one from the ECML/PKDD 2007 Challenge (http://www.lirmm.fr/pkdd2007-challenge/).

As a baseline, I work testing this datasets with a ModSecurity configure with the OWASP CRS (versión 2.2.9). I believe that is a good baseline to compare to. I obtain these results:

Dataset | Precision | Recall | False Positive Rate | False Negative Rate
-- | --: | --: | --: | --:
ECML/PKDD 2007 | 0.7 | 0.93 | 57.21% | 7.03%
CSIC 2010 | 0.5 | 0.343 | 23.93% | 65.68%

I have a couple of comments about these two dataset:
In the CSIC dataset, has classified as anomalous traffic, not only web attacks but also cases where for example there is a misspelled attribute name (p.e. instead of username -> username). These type of cases are classified as anomalous traffic, but they are not actually attacks.

In the case of the ECML/PKDD dataset, all url, attribute names and attribute values have been sanitized, replacing them with random strings (except in the cases of attack payloads).

Regards,
Rodrigo

[dune73]

Hey @rodrigo-martinez, thank you very much for chiming in and for this additional information. I think you hit the nail on the head with the tagging. And that's our big issue.

And the additional comments are very valuable.

Are you in a position to repeat your research with 3.0.2? We like to think of 3.0.2 as far better when it comes to false positives.

[rodrigo-martinez]

Hi @dune73,,
Yes, I have in my roadmap repeat the tests with 3.0.2, but I am with a couple of deadlines so I won’t do it right away. As soon as I do them, I communicate the results throughout this thread.

[spartantri]

This is a cool thing to automate for every PR :)

[csanders-git]

adding it to my calendar for next week to add to tests.

[rodrigo-martinez]

I'm working on repeting the test using the last version of CRS 3.0 for both dataset. I hope to publish here my results for next week.

[rodrigo-martinez]

Hi @dune73,
I've repeated the test for the ECML/PKDD 2007 dataset, using the Latest version of the owasp/modsecurity-crs docker image, for the different paranoia levels (1..4).

These are the preliminary results I've got, but I've some issues when executing the test that may affect this results:

Paranoia Level | Precision | Recall | False Positive Rate | False Negative Rate
-- | -- | -- | -- | --
1 | 0.498 | 0.585 | 15.04% | 41.48%
2 | 0.396 | 0.784 | 24.86% | 21.63%
3 | 0.327 | 0.905 | 33.43% | 9.54%
4 | 0.275 | 0.999 | 100.00% | 0.13%

As for the testing lab, I've implemented a Java client that parses the dataset and sends the requests to a server. In this case the server was a docker running the owasp/modsecuirty-crs image with minimal modifications (I added some scripts so the server always returns 200 OK in case that ModSecurity doesn't block the request).

The issue that I have during the test is that, when the CRS was configured with the paranoia level 4, most of the requests give one of the following errors:

• java.net.SocketException: Connection reset
• java.net.SocketException: Unexpected end of file from server

Do you know if there is some rule or something in the paranoia level 4 that could be blocking my IP after several errors?

On the other hand, do you know of other dataset containing full requests that could be used to this type of testing?

We are working on Machine Learning techniques to identify attacks, and is very important to have this type of datasets. In https://arxiv.org/abs/1803.05529 you could find one of our publications, where we present some of the models we are working on. Any feedback or comment is welcome. Right know our objectives are: first improve the models by using newer and bigger datasets; and secondly to integrate these ML models to ModSecurity with the OWASP CRS in a way where CRS rules and ML modules results are combined in order to improve detection and lowered the False Positives.

Regards,
Rodrigo

[dune73]

Thanks, @rodrigo-martinez. That looks really cool, on the other hand, your FP rate for PL1 puzzles me.

Connection resets: No, there should not be anything that drops your connection after a few alarms. Certainly not by default. Maybe an issue with the container. Seeing the error-log might help.

[523hu]

Hi, @rodrigo-martinez . I am doing an reseacher about detecting web attack by machine learning. But I cannot get the ECML/PKDD 2007 dataset online. I cannot complete the registration in the offical website. Could you send the dataset to me? My e-mail address [email protected] . Thank you very much!

[rodrigo-martinez]

Hi @523hu,
We have put forward a repository to keep all the datasets about web application attack together, we ask for the owners' approvals.

You could access the repository at: https://gitlab.fing.edu.uy/gsi/web-application-attacks-datasets

Our goal is to continue updating this repository as new dataset are available. Do not hesitate to ask any doubt or problem. We are also open to suggestions and new datasets with this characteristics to add to this repo.

We've been working on Machine Learning and web attacks for some time now, and we are very interesting to see what others are doing. If you like you could contact me at: [email protected]

Regards,
Rodrigo

[nileshyadav2004]

Hi, @rodrigo-martinez . I am doing an reseacher about detecting web attack by machine learning. But I cannot get the original ECML/PKDD 2007 dataset online. Could you send the dataset to me? My e-mail address [email protected] . Thanks a lot plz!

[nileshyadav2004]

original xml ECML/PKDD 2007 dataset files

[farouqBenarous]

Can anyone send me the dataset its not availible in their website : farouk.[email protected]

[csanders-git]

@dune73 ?

[dune73]

Hey @faroukcr7. If it's no longer there, then @rodrigo-martinez is probably the best contact to get it.

Otherwise, did you try the repository https://gitlab.fing.edu.uy/gsi/web-application-attacks-datasets that he mentioned?

[dneha19]

Can anyone send me the dataset its not available.
id : [email protected]

[rodrigo-martinez]

Hi @dneha19,
As already mention @dune73, the dataset is publicly available at: https://gitlab.fing.edu.uy/gsi/web-application-attacks-datasets

Did you have any issue to get it?

Regards,
Rodrigo

[nlyf]

Hi @rodrigo-martinez , @dneha19 ,@dune73!
provided site is not available now. Is there any saved archve?

[nlyf]

Now their link is availibe, do not worry

[github-actions[bot]]

This issue has been open 120 days with no activity. Remove the stale label or comment, or this will be closed in 14 days

[dune73]

Let's keep this open for a bit longer.