Thehive: Short Report is not shown on observables (3.0.8)
Request Type
Bug
Work Environment
| Question | Answer
|---------------------------|--------------------
| OS version (server) | Debian
| OS version (client) | Seven using Chrome Browser
| TheHive version / git hash | 3.0.6
| Package Type | DEB
Problem Description
Unfortunately the short-report is still not always shown on observables (observables-tab and tab of the observable itself).
crackytsi
All 53 comments
Can you provide the JSON definition of the observable? Just to see if it's a data issue or a display issue
nadouani
on 23 Mar 2018
How can I retrieve this definiton?
crackytsi
on 23 Mar 2018
curl -XGET -H 'Authorization: Bearer API_KEY' 'http://SERVER:PORT/api/case/artifact/OBSERVABLE_ID'
The OBSERVABLE_ID is the last part of the URL path of observable details page
nadouani
on 23 Mar 2018
{
"_type" : "case_artifact",
"updatedBy" : "user1",
"createdAt" : 1520514455787,
"createdBy" : "user1",
"status" : "Ok",
"data" : "192.168.200.2",
"updatedAt" : 1520514472316,
"_id" : "ea19684074eacd32ac5b1a087977fc30",
"sighted" : false,
"message" : "environment1",
"reports" : {
"HASDB_1_0" : {
"taxonomies" : [
{
"predicate" : "Net",
"level" : "info",
"value" : "\"192.168.200.0/28 NPS FCARAS123\"",
"namespace" : "HASDB"
}
]
}
},
"ioc" : false,
"tlp" : 2,
"dataType" : "ip",
"id" : "ea19684074eacd32ac5b1a087977fc30",
"tags" : [
"environment1"
],
"startDate" : 1520514455790
}
Hi,
As you might have noticed, this are analyzers that we use internal.
Strangely on the overview table, I can see Reports for 2 analyzers (HASDB and AssetDB), both Long reports are shown correctly.
crackytsi
on 23 Mar 2018
I'm taking a look :)
nadouani
on 23 Mar 2018
Well all looks good, the data (reports attribute) is on the observable, well formed too.
Do you have any javascript error on your browsers console?
nadouani
on 23 Mar 2018
There are 2 reports.... not just 1.
The problem seems to be that not always all reports are added to the data structure.
No, I have no javascript error.
crackytsi
on 23 Mar 2018
Wait, wait. Your observables should have 2 reports but it's definition contains only one instead of 2?
In the observable details page, you see both mini reports, and in observables list you see only one mini report?
Just to be sure we are talking about the same thing
nadouani
on 23 Mar 2018
Hi,
Right, my observable should have 2 reports, but the Definition retrieved by the API Call listes only one.
In the observable page I see 2 reports, but there is also only one mini-reports.
If I click on the analyzer, I see both Long reports.
crackytsi
on 23 Mar 2018
and both analyzer jobs are successful right?
nadouani
on 23 Mar 2018
This sounds like #409
nadouani
on 23 Mar 2018
Both Jobs were successfull.
yes it does, but this issue was solved and I'm using version 3.0.6 :-/
crackytsi
on 23 Mar 2018
Yes, I know, but it could has been solved partially, or you could have a corner case that has not been taken into account. The problem with this issue is that it's not easy to reproduce.
@To-om will take a look on it
nadouani
on 23 Mar 2018
If I can support you, let me know
The Case was created after upgrade to 3.0.6 so it should be relatively independend. :)
I can see this issue several times...
crackytsi
on 23 Mar 2018
Can you reproduce it by just running an analyzer or a given observable or does it occur only when you bulk run analyzers?
nadouani
on 23 Mar 2018
it seems to happen only during bulk Analysis (mark observable -> Action -> Analyse -> click more than one analyzer)
crackytsi
on 23 Mar 2018
Problem still exists.
If I start an analyzer from the observable alone, reports are shown. If I use the bulk option, it does not work.
crackytsi
on 12 Apr 2018
This issue is really weird, working on my side
nadouani
on 12 Apr 2018
I added a new observable and started multiple analyzer for this selected observable.
I can see in the details all long reports as expected, but no short report is shown.
If I try to get all the observables of the case I see this for the observable I added:
"tags" : [
"demo"
],
"_parent" : "AWKVDz4VufoPl6Jhh4eZ",
"reports" : {
"ANADNS_1_0" : {
"taxonomies" : []
}
},
"_routing" : "AWKVDz4VufoPl6Jhh4eZ",
"updatedAt" : 1523537680133,
"tlp" : 2,
"id" : "c7af432217a2504c6f372adbaa822484",
"updatedBy" : "usera",
"ioc" : false,
"status" : "Ok",
"dataType" : "ip",
"createdBy" : "usera",
"_id" : "c7af432217a2504c6f372adbaa822484",
"_type" : "case_artifact",
"startDate" : 1523537660073,
"message" : "",
"createdAt" : 1523537660070,
"_version" : 2,
"data" : "130.122.14.195",
"sighted" : false
},
I trust you :) but I don't know how to reproduce it :(
nadouani
on 12 Apr 2018
If I do the same manually I see this:
{
"tags" : [
"demo"
],
"message" : "",
"createdBy" : "user",
"tlp" : 2,
"_routing" : "AWKVDz4VufoPl6Jhh4eZ",
"_version" : 4,
"data" : "130.122.14.195",
"updatedAt" : 1523538152823,
"_parent" : "AWKVDz4VufoPl6Jhh4eZ",
"sighted" : false,
"dataType" : "ip",
"updatedBy" : "user",
"status" : "Ok",
"id" : "16314bc92809b55225f17a7141c95756",
"createdAt" : 1523538115500,
"reports" : {
"ANADNS_1_0" : {
"taxonomies" : [
{
"predicate" : "Net",
"namespace" : "ANADNS",
"level" : "info",
"value" : "\"removed\""
}
]
},
"OutManager_1_1" : {
"taxonomies" : [
{
"level" : "info",
"namespace" : "AM",
"value" : "\"removed\"",
"predicate" : "Server"
}
]
},
"COMPONENT_1_0" : {
"taxonomies" : [
{
"predicate" : "Net",
"namespace" : "COMPONENT",
"level" : "info",
"value" : "\"removed\""
}
]
}
},
"_type" : "case_artifact",
"ioc" : false,
"startDate" : 1523538115506,
"_id" : "16314bc92809b55225f17a7141c95756"
},
Could this be related to the observable type?
crackytsi
on 12 Apr 2018
No the thing is that jobs are asynchronous and the issue here is that there is a conflict that ends by overriding the observable.reports property, which should have been fixed, but seems to appear again
nadouani
on 12 Apr 2018
It seems to be related if an analysis failes in any kind...
crackytsi
on 13 Apr 2018
you mean, if you run like 3 jobs and one of them fails?
nadouani
on 13 Apr 2018
@crackytsi Doe this occur with the "Official" analyzers? All the examples you provided are related to your "private" analyzers. Just curiosity :)
nadouani
on 13 Apr 2018
Hmm, strange. I did some checks and I've noticed that the "official" analyzers did always result an empty artifacts result:
{
"full" : {
"category" : "News and Media"
},
"summary" : {
"taxonomies" : [
{
"namespace" : "Fortiguard",
"level" : "safe",
"value" : "\"News and Media\"",
"predicate" : "URLCat"
}
]
},
"artifacts" : [],
"success" : true
}
But my own developements usually result something like this:
"artifacts" : [
{
"type" : "ip",
"value" : "1.2.3.0/28"
}
],
crackytsi
on 17 Apr 2018
Hello,
what version of cortexutils do you have, and does your analyzer enable the "artifacts auto extraction"?
nadouani
on 17 Apr 2018
pip freeze | grep cortexutils
cortexutils==1.2.4
pip3 freeze | grep cortexutils
cortexutils==1.2.4
ExtractObservables=False
crackytsi
on 17 Apr 2018
ExtractObservables=False is the trick.
Update your enabled analyzers to make this option become true :)
nadouani
on 17 Apr 2018
Do you mean if this is disabled in Cortex-GUI but the analyzer results something TheHive behaves strange with the short-observable?
crackytsi
on 17 Apr 2018
Non, I'm just saying the artifacts = [] is a result of auto_extract_artifacts = False.
Nothing related to TheHive here. If you disable the extraction of artifacts by the analyzers, then the latters will not return a list of extracted observables
nadouani
on 17 Apr 2018
Doe this occur with the "Official" analyzers? All the examples you provided are related to your "private" analyzers. Just curiosity :)
Any answer about this question (asked above few days ago)?
nadouani
on 17 Apr 2018
Strange, as this option is disabled for my analyzers as well as the others in the frontend.
crackytsi
on 17 Apr 2018
Why is this strange, I don't get you. it's disabled by default in the global configuration (just below the proxies)
nadouani
on 17 Apr 2018
I'm still trying to find the reason why sometimes the short-reports disappear or even don't appear if I run multiple analyzers as bulk instead of starting them one by one.
So this artifacts topic seemed to be different... But on my configuration side everything is configured the same (as I see...)
crackytsi
on 17 Apr 2018
Honestly, the artifacts thing is really not related to the short reports issue.
Please do you have an answer to my question above :)
nadouani
on 17 Apr 2018
It seems to be related only to self-developed ones.
I tested with 4 analyzers:
If I start it on case a) the short reports of analyzer 1 and 2 are shown.
If I start it on case b) with the same IP the short reports of analyzer 3 and 4 are shown.
So it seems to be randomly which result are shown/not shown.
crackytsi
on 17 Apr 2018
I forgot to add that all use the same predicate but different namespaces.
If I do execute an api call api/case/artifact/_search:
Case a)
"Analyzer_1" : {
"taxonomies" : [
{
"level" : "info",
"predicate" : "Net",
"value" : "\"somevaule\"",
"namespace" : "Analyzer1"
}
]
},
"Analyzer_2" : {
"taxonomies" : [
{
"namespace" : "Analyzer2",
"level" : "info",
"value" : "\"someothervalue\"",
"predicate" : "Net"
}
]
}
},
Case b)
"reports" : {
"Analyzer_3" : {
"taxonomies" : [
{
"namespace" : "Analyzer3",
"level" : "info",
"predicate" : "Server",
"value" : "\"somevalue\""
}
]
},
"Analyzer_4" : {
"taxonomies" : [
{
"level" : "info",
"value" : "\"someothervalue\"",
"predicate" : "Net",
"namespace" : "Analyzer4"
}
]
}
},
But if I click on the observable I can see that in both cases all 4 analyzers were executed and I seen see all 4 long reports.
On the observable view, I also see only 2 short-reports instead of 4.
So I don't thing that something on the template is wrong...
crackytsi
on 17 Apr 2018
If you do the same thing using 4 or 5 public analyzers, do you reproduce the issue?
nadouani
on 17 Apr 2018
Hi,
I can reproduce it also with public analyzers.
It seems to depend on the response time. If there is a big delay between the different reports, it seems not to occur.
You should be able to reproduce it if you have lets say 4 dummy analyzers that always bring directly the result. If you execute them for some the short reports will appear for others not. Randomly.
crackytsi
on 18 Apr 2018
Btw: if I start an analysis again the logic could check if short Report / taxonomy is added and add it if missing.
Currently only long report gets added.
This would compensate the issue ;)
crackytsi
on 20 Apr 2018
This issue has been reproduced using 3 "fast" analyzers that just return the input
nadouani
on 3 May 2018
Unfortunately, this issue is still not fixed for us (running 3.0.9 currently). We can reproduce it when running several (three is already enough) analyzers at once. Note that most of our analyzers are home-grown and return results quickly.
@crackytsi Can you confirm the fix or do you still experience this issue as well?
Rolinh
on 22 May 2018
@Rolinh this is fixed in 3.0.10 that is not yet released
nadouani
on 22 May 2018
it will come in the next few days
nadouani
on 22 May 2018
@nadouani Ah, good to know, thanks. In which issue was this tracked? I missed it.
Rolinh
on 22 May 2018
In this one :)
We took 2 months to be able to reproduce and fine the reason, and it has been fixed in 30 minutes after reproducing it.
Most of the time has been spent on understanding that fast home made analyzers are required to reproduce the bug :)
nadouani
on 22 May 2018
In this one :)
:grin:
It is not labeled 3.0.10 though :wink:
Rolinh
on 23 May 2018
You're right, I was sure it had the correct milestone :(
nadouani
on 23 May 2018
Related issues
Passimist
路
4Comments
vi-or-die
路
4Comments
crackytsi
路
5Comments
rhaist
路
4Comments
Xumeiquer
路
6Comments