Cuckoo: Invalid JSON Report

[arnydo]

On some analysis I am receiving an invalid JSON report. There error received is:

SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 552683 column 14 of the JSON data

Appears to be due to certain strings returned from the file analysis is conflicting with the JSON format.

I have also attached the JSON report that threw this error.
badreport.zip

[jbremer]

That looks incorrect indeed. Most likely somehow the report.json file got corrupted. Do you see this often?

[arnydo]

A little further investigation and it looks like the bad reports have a duplicate item at the bottom of the file that breaks the JSON.

                {
                    "basename": "6fd5c28a56cfd308_SharedDataEvents-journal", 
                    "sha256": "6fd5c28a56cfd308531a02e0dca0d633992ab7ca1119bd92a7b2bac63d85be52", 
                    "dirname": "files"
                }, 
                {
                    "basename": "3e720d2969a7b751_SharedDataEvents-journal", 
                    "sha256": "3e720d2969a7b7517f344d16336a767ae34116d9014997b2e68cfa0067b7caf4", 
                    "dirname": "files"
                }, 
                {
                    "basename": "15bd0452f4f8bd93_SharedDataEvents-journal", 
                    "sha256": "15bd0452f4f8bd93e0172035944f4b71791283af598c4babf7ba4e3ae9a4aad0", 
                    "dirname": "files"
                }, 
                {
                    "basename": "72cc5375ad53900f_aum.log", 
                    "sha256": "72cc5375ad53900fc3d0d6a6c3074dfbaae70e37c666033e26eb953e0322d2a1", 
                    "dirname": "files"
                }
            ]
        }
    }
}            "basename": "72cc5375ad53900f_aum.log", 
                    "sha256": "72cc5375ad53900fc3d0d6a6c3074dfbaae70e37c666033e26eb953e0322d2a1", 
                    "dirname": "files"
                }
            ]
        }
    }
}

[guzmud]

Sorry to bother, but I couldn't find any more information on this kind of issues. It seems to happen quite often with various cuckoo installation, but I couldn't find why. Any more info on this point ?

[doomedraven]

would help more if you can share samples which generates that

[guzmud]

Doesn't matter, happened with various samples on various installs. I wonder if it's not some kind of concurrency in post-processing the report.json, hence the duplication of json data at the end, ending in an invalid json.

As of now my fix is to report.count('\n}') and if it's more than one, to cut after the first one. Ugly, hacky, working.

[doomedraven]

em it should be what someone modified something, nobody else have this problem so far. so you should debug what is wrong and how that data added there from where etc

[guzmud]

Nobody else has this problem except the guy just above and a few of my coworkers. Usually re-running the reporting is fine (which may explain why people did not complain + most people use the .html instead of the .json).

BTW my hack is working due to the json report being indented (which may be the buggy post-process, would be fun), which should lead to the last '\n}' being the only on in a properly dumped report.

[doomedraven]

well i mean in community nobody else reported that, so should be something on your side, it hard to fix something what we can't access to debug/investigate

[guzmud]

I get it do not worry. Most I can say is this :

• no custom plugins or homemade extensions
• almost vanilla configuration, no modification made to processing.conf or reporting.conf
• happened in various installs with various backends
• happened with various unrelated samples, rerunning the reporting produce a proper report

So doesnt' seem to be conf-related, sample-related, backend-related, homemade-stuff-related.

I think it's a freak case of concurrency on file access and appending data. And has I said, most of the community (as far as I know) is using the html reporting rather than the json reporting, those who are using the json are inclined to manage those problems without ticketing the issue on github (whether by trying hard the reporting until it's fixed, or hacking their way out of the issue), and I do not only spoke for myself regarding this issue.

[doomedraven]

can you put here how you execute processing part? cuckoo -d process a1 something like that?

[guzmud]

cuckoo --cwd /some/path/to/the/cwd process p%(process_num)d

(process_num being filled in by supervisor)

You think it may be concurrence between cuckoo process on the same sample/report ?

[doomedraven]

try that started by hand, just in case to see if supervisor mess something, who knows

[guzmud]

try that started by hand, just in case to see if supervisor mess something, who knows

Actually would make sense with the multi-multi-process of supervisor. You genius @doomedraven ! Thank you a lot. I'll make some tests and if it seems to be fine, I'll report back here.

[guzmud]

Well, seems we brainfucked on our side and forgot the incompatibility between the multiprocessing made by supervisor and the multiprocess created by the process_results option in the cuckoo.conf (in the documentation it is mentioned in the distributed cuckoo/proposed setup/configuration settings, but seems the case is similar for a single-instance Cuckoo with multiprocess cuckoo process made using supervisor or whatever else).

We'll wait and see but it seems to be that. Thank you again @doomedraven !

[doomedraven]

yey cool, glad that helped :)