Cuckoo: elasticsearch problem

[ness993]

Hy,
I was trying to find a problem but I can't.

Help me GitHub, you are my only hope!

my cucko star look slike theas :

2017-12-16 16:30:42,141 [cuckoo.core.startup] DEBUG: Imported modules...
[2017-12-16T16:30:42,223][DEBUG][o.e.a.a.i.t.p.TransportPutIndexTemplateAction] [FvmTES8] failed to put template [1_template]
org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping [cuckoo]: Could not convert [procmemory.include_in_root] to boolean
    at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:346) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:274) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService.validateAndAddTemplate(MetaDataIndexTemplateService.java:234) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService.access$200(MetaDataIndexTemplateService.java:63) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService$2.execute(MetaDataIndexTemplateService.java:172) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:640) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:270) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:195) [elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:130) [elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-6.1.0.jar:6.1.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) [?:?]
    at java.lang.Thread.run(Thread.java:844) [?:?]
Caused by: java.lang.IllegalArgumentException: Could not convert [procmemory.include_in_root] to boolean
    at org.elasticsearch.common.xcontent.support.XContentMapValues.nodeBooleanValue(XContentMapValues.java:433) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.TypeParsers.nodeBooleanValue(TypeParsers.java:59) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseNested(ObjectMapper.java:232) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:168) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:278) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:199) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:131) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:111) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:91) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:343) ~[elasticsearch-6.1.0.jar:6.1.0]
    ... 17 more
Caused by: java.lang.IllegalArgumentException: Failed to parse value [True] as only [true] or [false] are allowed.
    at org.elasticsearch.common.Booleans.parseBoolean(Booleans.java:73) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.common.xcontent.support.XContentMapValues.nodeBooleanValue(XContentMapValues.java:438) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.common.xcontent.support.XContentMapValues.nodeBooleanValue(XContentMapValues.java:431) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.TypeParsers.nodeBooleanValue(TypeParsers.java:59) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseNested(ObjectMapper.java:232) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:168) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:278) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:199) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:131) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:111) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:91) ~[elasticsearch-6.1.0.jar:6.1.0]
    at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:343) ~[elasticsearch-6.1.0.jar:6.1.0]
    ... 17 more
2017-12-16 16:30:42,238 [elasticsearch] WARNING: PUT http://127.0.0.1:9200/_template/1_template [status:400 request:0.081s]
2017-12-16 16:30:43,197 [pip.utils] DEBUG: lzma module is not available
2017-12-16 16:30:43,200 [pip.vcs] DEBUG: Registered VCS backend: git
2017-12-16 16:30:43,228 [pip.vcs] DEBUG: Registered VCS backend: hg
2017-12-16 16:30:43,313 [pip.vcs] DEBUG: Registered VCS backend: svn
2017-12-16 16:30:43,314 [pip.vcs] DEBUG: Registered VCS backend: bzr
Oops! Cuckoo failed in an unhandled exception!
Sometimes bugs are already fixed in the development release, it is therefore recommended to retry with the latest development release available https://github.com/cuckoosandbox/cuckoo
If the error persists please open a new issue at https://github.com/cuckoosandbox/cuckoo/issues

=== Exception details ===
Cuckoo version: 2.0.5
OS version: posix
OS release: Ubuntu 16.04 xenial
Python version: 2.7.12
Python implementation: CPython
Machine arch: x86_64
Modules: adium-theme-ubuntu:0.3.4 alembic:0.8.8 androguard:3.0.1 asn1crypto:0.23.0 beautifulsoup4:4.5.3 capstone:3.0.5rc2 cffi:1.11.2 chardet:2.3.0 click:6.6 colorama:0.3.7 cryptography:2.1.4 cuckoo:2.0.5.2 distorm3:3.3.0 django-extensions:1.6.7 django:1.8.4 dpkt:1.8.7 ecdsa:0.13 egghatch:0.2.1 elasticsearch:5.3.0 enum34:1.1.6 flask-sqlalchemy:2.1 flask:0.12.2 functools32:3.2.3.post2 html5lib:0.999 httpreplay:0.2.1 idna:2.6 ipaddress:1.0.18 itsdangerous:0.24 jdcal:1.0 jinja2:2.9.6 jsbeautifier:1.6.2 jsonschema:2.6.0 lxml:3.5.0 m2crypto:0.24.0 mako:1.0.7 markupsafe:1.0 mat:0.6.1 mutagen:1.31 olefile:0.43 oletools:0.51 openpyxl:2.3.0 pdfrw:0.2 peepdf:0.3.6 pefile2:1.2.11 pillow:3.2.0 pip:9.0.1 py:1.4.31 pycparser:2.18 pycrypto:2.6.1 pyelftools:0.24 pygobject:3.20.0 pymisp:2.4.54 pymongo:3.0.3 pyopenssl:17.5.0 pyrit:0.4.0 pytest:2.8.7 python-dateutil:2.4.2 python-editor:1.0.3 python-magic:0.4.12 pythonaes:1.0 pytz:2014.10 requests:2.13.0 scapy:2.3.2 setuptools:20.7.0 sflock:0.2.17 six:1.11.0 sqlalchemy:1.0.8 tlslite-ng:0.6.0 unicorn:1.0.1 unity-lens-photos:1.0 urllib3:1.22 vboxapi:1.0 virtualenv:15.0.1 volatility:2.6 wakeonlan:0.2.2 werkzeug:0.13 wheel:0.29.0 yara-python:3.6.3

2017-12-16 16:30:43,337 [cuckoo] ERROR: RequestError: TransportError(400, u'mapper_parsing_exception', u'Failed to parse mapping [cuckoo]: Could not convert [procmemory.include_in_root] to boolean')
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cuckoo/main.py", line 230, in main
    cuckoo_init(level, ctx)
  File "/usr/local/lib/python2.7/dist-packages/cuckoo/main.py", line 137, in cuckoo_init
    init_modules()
  File "/usr/local/lib/python2.7/dist-packages/cuckoo/core/startup.py", line 183, in init_modules
    module.init_once()
  File "/usr/local/lib/python2.7/dist-packages/cuckoo/reporting/elasticsearch.py", line 51, in init_once
    if not cls.apply_template():
  File "/usr/local/lib/python2.7/dist-packages/cuckoo/reporting/elasticsearch.py", line 76, in apply_template
    name=cls.template_name, body=json.dumps(template)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 73, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/indices.py", line 458, in put_template
    name), params=params, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 318, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 128, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 124, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
RequestError: TransportError(400, u'mapper_parsing_exception', u'Failed to parse mapping [cuckoo]: Could not convert [procmemory.include_in_root] to boolean')

my reporting module for el.search looks like :

[elasticsearch]
enabled = yes
# Comma-separated list of ElasticSearch hosts. Format is IP:PORT, if port is
# missing the default port is used.
# Example: hosts = 127.0.0.1:9200, 192.168.1.1:80
hosts = 127.0.0.1:9200
# Increase default timeout from 10 seconds, required when indexing larger
# analysis documents.
timeout = 1200
# Set to yes if we want to be able to search every API call instead of just
# through the behavioral summary.
calls = yes
# Index of this Cuckoo instance. If multiple Cuckoo instances connect to the
# same ElasticSearch host then this index (in Moloch called "instance") should
# be unique for each Cuckoo instance.
index = 1

[RicoVZ]

Hi ness993,

Thanks for posting this issue!

It looks like you are using Elasticsearch 6.x. Some things have changed in v6.x, rendering the template that Cuckoo currently uses to creates indexes unusable for Elasticsearch 6.x.

I have tested Cuckoo with your config and can confirm this issue starts as soon as I use 6.x. The template does still work on Elasticsearch 5.x.

Cuckoo currently uses types in the Elasticsearch mapping. Support for this has been partially removed in v6.x. According to this page: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html upgrades from 5.x to 6.x will allow you to still use existing indexes that use types, but creating new indexes with types is not possible anymore.

So, for now, if you want to use Elasticsearch with Cuckoo, you can only use Elasticsearch 5.x

[ness993]

Hi RicoVz,
Thanks for showing me this point.

I weel try ASAP.

[jbremer]

Any luck @ness993 ?

[mohlcyber]

Hi Guys, is there any progress on the Elasticsearch 6.x support?

[RicoVZ]

Hi mohlcyber, not yet, unfortunately. We are working on some other things at the moment. But we definitely have not forgotten yet.

[mohlcyber]

Excellent thanks for the prompt response.

[jbremer]

Closing for now, this issue will eventually be addressed.