Wazuh: "[ERROR] Couldn't decode line at syscheck database" in fim_migrate

[iasdeoupxe]

Bug report

OS
Debian 9/Stretch

Wazuh version
3.7.0 (previously 3.6.1)

Wazuh API version
3.7.0

Install type
manager

Install method
packages

Log sample

2018-11-10 16:32:20 [ERROR] Couldn't decode line at syscheck database.
Traceback (most recent call last):
  File "./fim_migrate", line 287, in <module>
    if not check_file_entry(agt[0], decoded[2], s):
  File "./fim_migrate", line 91, in check_file_entry
    msg = msg + cfile + b"';"
TypeError: cannot concatenate 'str' and 'NoneType' objects

Bug description

Following the migration steps in https://documentation.wazuh.com/current/user-manual/reference/tools/fim_migrate.html#fim-migrate (downloaded the fim_migrate today) the migration stops/fails at one agent with the log entry above (running in DEBUG mode).

When adding a:

    if cfile is None:
        print("cfile is None")

before (line 91) this:

    msg = msg + cfile + b"';"

it seems that "cfile" is None due to some reasons causing this failure.

[iasdeoupxe]

So i did a little bit more debugging and changed:

                logging.info("[{0}/{1}] Upgrading FIM database for agent '{2}'...".format(pos, total_agents, str(agt[0]).zfill(3)))
            with open(dbfile, 'rb') as syscheck:
                for line in syscheck:
                    if not line[0] == b'#':
                        decoded = _fim_decode(line)
                        if not _force:

into:

                logging.info("[{0}/{1}] Upgrading FIM database for agent '{2}'...".format(pos, total_agents, str(agt[0]).zfill(3)))
            with open(dbfile, 'rb') as syscheck:
                for line in syscheck:
                    if not line[0] == b'#':
                        print(line)
                        decoded = _fim_decode(line)
                        print(decoded)
                        if not _force:

After re-run the fim_migrate the following line is showing up which seems to be the problematic one:

!++4285:33256:0:0:829c192f512518213afeb2ab88dfffac:f571d66a3211fff4200891b5ff7320bebe6768f6:root:root:1532064424:134005:a44478ea6a17c7dd7deeb4ff0692b34d5f1039ff94d39b65e92aa02fbcf7312e!:::::::::: !1532681200 /etc/init.d/elasticsearch.dpkg-new

[albertomn86]

Hi @iasdeoupxe

thank you for reporting this issue and also for the researching. As you already discovered, the problem is caused by some wrong entries in the database.

We've already updated the script to discard these entries and continue with the migration. This fix is in this branch: https://github.com/wazuh/wazuh/blob/fix-migration-tool/tools/migration/fim_migrate.py
Pull request: https://github.com/wazuh/wazuh/pull/1867

You can check if this solves your issue.

We are still investigating what caused the wrong entries in the database.

Best regards.

[iasdeoupxe]

Hi @albertomn86,

thanks for this great background information. I was already able to test the new fim_migrate tool which has now finished the migration successfully. Running it in debug mode (-d) showed additional similar lines, all from an ELK installation with the .dpkg-new file ending:

2018-11-12 18:49:37 [ERROR] Couldn't decode line at syscheck database.
2018-11-12 18:49:37 [DEBUG] Error parsing line: !++4285:33256:0:0:829c192f512518213afeb2ab88dfffac:f571d66a3211fff4200891b5ff7320bebe6768f6:root:root:1532064424:134005:a44478ea6a17c7dd7deeb4ff0692b34d5f1039ff94d39b65e92aa02fbcf7312e!:::::::::: !1532681200 /etc/init.d/elasticsearch.dpkg-new

2018-11-12 18:49:37 [ERROR] Couldn't decode line at syscheck database.
2018-11-12 18:49:37 [DEBUG] Error parsing line: !++3009:33200:0:109:4105ce09af151d74bd80ba609ec459ee:484fe15d3cabe25823b1aa090415fdbec6d48506:root:elasticsearch:1532064424:134001:d12a63cdeb65dbdbbd9c3ebea459d83c5d69d9b471fe24e95539d55346bf3aa5!:::::::::: !1532681201 /etc/elasticsearch/jvm.options.dpkg-new

2018-11-12 18:49:37 [ERROR] Couldn't decode line at syscheck database.
2018-11-12 18:49:37 [DEBUG] Error parsing line: !++6380:33200:0:109:91c16b0c9fa76e450be9d209810ce017:94eef98ab6f48f4b4b8729b405876492df21ae4a:root:elasticsearch:1532064424:134000:dfb9dfad8ab87f0b5789291f9797c6ed55c0868f315496e4c3a035cf879f05c9!:::::::::: !1532681201 /etc/elasticsearch/log4j2.properties.dpkg-new

2018-11-12 18:49:37 [ERROR] Couldn't decode line at syscheck database.
2018-11-12 18:49:37 [DEBUG] Error parsing line: !++1613:33200:0:0:2f22b01002e8e30fef4b38a4d94d3b7a:1dae1d69c03cfc228878679c55ec700a6484f9bf:root:root:1532064424:134003:46c410bafb031ff5cc4f78e204e30e0bac5e23c7aae1c7b937458feba6209265!:::::::::: !1532681207 /etc/default/elasticsearch.dpkg-new

If there is anything else i could provide (information on the file, environment details etc.) please let me know.

[albertomn86]

Hi @iasdeoupxe,

your contribution was really helpful to solve this issue.

The problem with the incorrect part in the database entries was probably caused by the new message format introduced in the agents in version 3.4 if these agents reported to a manager with a lower version.

We have updated the migration script so that it automatically deletes that part and inserts the entry normally.

Best regards.