Thehive: Unable to create an alert with big file artifact through API

[Xumeiquer]

Request Type

Bug

Work Environment

| Question | Answer
|---------------------------|--------------------
| OS version (server) | CentOS
| OS version (client) | CentOS
| TheHive version / git hash | 65771593e234 (docker image ID)
| Package Type | RPM

Problem Description

Using TheHive4py I am unable to create an alert with "big" zip file as artifact. The size of the file I tried it is 3005914 bytes (~2.9M).

The error I received is:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/thehive4py/api.py", line 379, in create_alert
    raise AlertException("Alert create error: {}".format(e))
thehive4py.exceptions.AlertException: Alert create error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

On the other hand, I tried it with a smaller zip file 166 bytes as well and it worked fine.

Steps to Reproduce

import uuid
from thehive4py.models import Alert
from thehive4py.models import AlertArtifact
from thehive4py.api import TheHiveApi

sourceRef = uuid.uuid4().hex[0:6]
files = ["path/to/big/zip/file"]

api = TheHiveApi("http://X.X.X.X:9000", "<apiKey>", None, {})

artifacts = []
for file in files:
    artifacts.append(AlertArtifact(dataType="file", data=file))

template = api.get_case_template("XYZ")

alert = Alert(title="TEST - Alert",
            tlp=template["tlp"],
            tags=template["tags"],
            description=template["description"],
            type="Alert",
            source="ZZZ",
            artifacts=artifacts,
            sourceRef=sourceRef,
            severity=template["severity"])

api.create_alert(alert)

Complementary information

I couldn't grab any logs from the application.log because basically nothing shows up in logs.

[nadouani]

Hello, I'm expecting an error because of the document size limitation in Elasticsearch, but I'm not expecting a 'Connection reset by peer' error.

[Xumeiquer]

I have double check the elasticsearch default index size and it is 100MB so the issue is no because of elasticsearch, at least for now.

[nadouani]

Ok, is it possible to call the same API directly by hitting the the endpoint, without TheHive4py?

[Xumeiquer]

I have tried to create two alerts one without artifacts and another with one artifact. The artifact is a zip file of 1.2Mb

1.2M Jan 29 15:58 96a572e614724e179fafba219a47ab16.zip

I have built a file to simulate the alert json data. The json file is a little bit more big than the zip file (obvious).

1.6M Jan 29 16:31 alert.json

The first lines of alert.json file are:

{
  "title": "New Alert",
  "description": "N/A",
  "type": "external",
  "source": "instance1",
  "sourceRef": "alert-ref2",
  "artifacts": ["96a572e614724e179fafba219a47ab16.zip;application/zip;UEsDBBQAAAAAAEx/...."],
}

The first alert creation done as test is:

curl -XPOST -H 'Authorization: Bearer **API KEY**' -H 'Content-Type: application/json' http://192.168.1.10:9000/api/alert -vvvv -d '{
>   "title": "New Alert",
>   "description": "N/A",
>   "type": "external",
>   "source": "instance1",
>   "sourceRef": "alert-ref1"
> }'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 192.168.1.10...
* TCP_NODELAY set
* Connected to 192.168.1.10 (192.168.1.10) port 9000 (#0)
> POST /api/alert HTTP/1.1
> Host: 192.168.1.10:9000
> User-Agent: curl/7.61.1
> Accept: */*
> Authorization: Bearer **API KEY**
> Content-Type: application/json
> Content-Length: 126
>
* upload completely sent off: 126 out of 126 bytes
< HTTP/1.1 201 Created
< Set-Cookie: THE_HIVE_SESSION=<cookie data>; SameSite=Lax; Path=/; Secure; HTTPOnly
< Date: Tue, 29 Jan 2019 16:34:54 GMT
< Content-Type: application/json
< Content-Length: 457
<
* Connection #0 to host 192.168.1.10 left intact
{"createdBy":"admin","customFields":{},"lastSyncDate":1548779694233,"date":1548779694233,"source":"instance1","type":"external","follow":true,"title":"New Alert","artifacts":[],"status":"New","severity":2,"description":"N/A","tlp":2,"_id":"d348ef48bf78e88f0e3060f72d72a977","createdAt":1548779694232,"sourceRef":"alert-ref1","_type":"alert","_parent":null,"_routing":"d348ef48bf78e88f0e3060f72d72a977","_version":1,"id":"d348ef48bf78e88f0e3060f72d72a977"}

That means the API works fine so let's try with an artifact.

curl -XPOST -H 'Authorization: Bearer **API KEY**' -H 'Content-Type: application/json' http://192.168.1.10:9000/api/alert -vvvv -d @alert.json
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 192.168.1.10...
* TCP_NODELAY set
* Connected to 192.168.1.10 (192.168.1.10) port 9000 (#0)
> POST /api/alert HTTP/1.1
> Host: 192.168.1.10:9000
> User-Agent: curl/7.61.1
> Accept: */*
> Authorization: Bearer **API KEY**
> Content-Type: application/json
> Content-Length: 1676351
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< Date: Tue, 29 Jan 2019 16:38:34 GMT
< HTTP/1.1 413 Request Entity Too Large
< Date: Tue, 29 Jan 2019 16:38:34 GMT
< Connection: close
< Content-Type: text/plain; charset=UTF-8
< Content-Length: 69
<
* we are done reading and this is set to close, stop send
* Closing connection 0
A client error occurred on POST /api/alert : Request Entity Too Large

Same as yesterday there is no logs going out from TheHive.

[mdtro]

Are you running TheHive behind some sort of reverse proxy (ex. haproxy, nginx, etc...) ?

[nadouani]

The POST body seems to have a bigger size than allowed by play framework. There is a configuration that allows customising that option. I’ll take a look an tell you how to fix that

[mdtro]

The POST body seems to have a bigger size than allowed by play framework. There is a configuration that allows customising that option. I’ll take a look an tell you how to fix that

From TheHiveDocs: https://github.com/TheHive-Project/TheHiveDocs/blob/master/admin/configuration.md#5-entity-size-limit

[nadouani]

Thanks @mdtro, that’s the answer :)

[nadouani]

@Xumeiquer does the solution above solve your issue?

[Xumeiquer]

The TheHive is actually running behind proxy (traefik), but the test I showed you has been done directly to the TheHive docker instance so no proxy is in place.

Play framework is configured with the following values

play.http.parser.maxMemoryBuffer=1M
play.http.parser.maxDiskBuffer=100M

Reading the documentation you have pointed out I have changed maxMemoryBuffer from 1M to 10M and I test it again. After that change the alert was created by hitting the API from curl.

curl -XPOST -H 'Authorization: Bearer **API KEY**' -H 'Content-Type: application/json' http://192.168.1.10:9000/api/alert -vvvv -d @alert.json
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 192.168.1.10...
* TCP_NODELAY set
* Connected to 192.168.1.10 (192.168.1.10) port 9000 (#0)
> POST /api/alert HTTP/1.1
> Host: 192.168.1.10:9000
> User-Agent: curl/7.61.1
> Accept: */*
> Authorization: Bearer **API KEY**
> Content-Type: application/json
> Content-Length: 1676433
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< Date: Wed, 30 Jan 2019 09:58:48 GMT
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Set-Cookie: THE_HIVE_SESSION=<cookie data>; SameSite=Lax; Path=/; Secure; HTTPOnly
< Date: Wed, 30 Jan 2019 09:58:50 GMT
< Content-Type: application/json
< Content-Length: 856
<
* Connection #0 to host 192.168.1.10 left intact
{"createdBy":"admin","customFields":{},"_id":"58977333e1959a3c7ff2a4971d2830b5","date":1548842329338,"lastSyncDate":1548842329340,"source":"instance1","type":"external","follow":true,"title":"New Alert","createdAt":1548842329328,"status":"New","sourceRef":"alert-ref-1","artifacts":[{"dataType":"file","tlp":2,"tags":[],"message":"Test message","attachment":{"name":"96a572e614724e179fafba219a47ab16.zip","hashes":["8f93ec75a8cef7d4271a8a598a1930f11c5abce4899b20382000446a8fd393f8","2962da041ec8872558d6f88102f308867d10f429","7d66412c35864807625cc3445a42eefe"],"size":1257118,"contentType":"application/zip","id":"8f93ec75a8cef7d4271a8a598a1930f11c5abce4899b20382000446a8fd393f8"}}],"severity":2,"description":"N/A","tlp":2,"_type":"alert","_parent":null,"_routing":"58977333e1959a3c7ff2a4971d2830b5","_version":1,"id":"58977333e1959a3c7ff2a4971d2830b5"}/

Next test is through TheHive4py. I will post the results shortly.

[nadouani]

It's expected to work. You just need to make sure the fix the max allowed size in your proxy too.

[Xumeiquer]

It works!

Thanks for the help.

This issue can be close.