Misp: Users get error 0 when they attempt to sync barncat MISP to their own local MISP

[bambenek]

A few users have reported when they try to sync their own MISP instances with Barncat, they get error code 0. Want to make sure they can only pull events and not push with whatever resolution there is.

If you would like to report a bug, please fill the template bellow

Work environment

| Questions | Answers
|---------------------------|--------------------
| Type of issue | Bug
| OS version (server) | Ubuntu
| OS version (client) |
| PHP version | 7.0
| MISP version / git hash | 2.4.70
| Browser | If applicable

Expected behavior

Actual behavior

Steps to reproduce the behavior

Logs, screenshots, configuration dump, ...

[lolnate]

As a user, I have added the instance and the test connection works properly. In the jobs section of my MISP instance it says "Error: got response code 0". I can't find what this means, and I don't see anything in any of the logs on my server. A connection test to the remote server works fine:

Local version: 2.4.70
Remote version: 2.4.70
Status: OK
Compatiblity: Compatible
POST test: Received sent package

[BurntToast-DFIR]

I also receive this error. Exactly the same as that shown above.

[iglocska]

Sorry for all the delays, looking into it now.

[iglocska]

So after looking into this: This is caused by a low timeout, at least on the Barncat instance (seems to be set to 30 seconds).

The problem is that the sync starts with a negotiation phase where the full event index containing all event metadata is fetched to decide on what actually gets synchronised. This, considering the amount of data fetched (all event fields, creator and orwner organisation, tags, galaxies) whilst dealing with 200k+ events is actually quite a massive fetch and serialisation task.

So in order to resolve this, I pushed a new patch that includes a parameter called "minimal" for the event index. This makes MISP only fetch the event id, uuid, timestamp and published state, meaning that the data amount and fetch workload is heavily reduced. The synchronisation mechanism now tries to request the minimal version of the index.

Please make sure that you update your instances with this new patch and tell your users to do the same and the issue should be resolved. Additionally, make sure you raise the timeout timer to something a tad higher. Let me know if this has resolved it!

[bambenek]

Where is that timeout defined? php.ini?

On 4/13/2017 2:01 PM, Andras Iklody wrote:
>

So after looking into this: This is caused by a low timeout, at least
on the Barncat instance (seems to be set to 30 seconds).

The problem is that the sync starts with a negotiation phase where the
full event index containing all event metadata is fetched to decide on
what actually gets synchronised. This, considering the amount of data
fetched (all event fields, creator and orwner organisation, tags,
galaxies) whilst dealing with 200k+ events is actually quite a massive
fetch and serialisation task.

So in order to resolve this, I pushed a new patch that includes a
parameter called "minimal" for the event index. This makes MISP only
fetch the event id, uuid, timestamp and published state, meaning that
the data amount and fetch workload is heavily reduced. The
synchronisation mechanism now tries to request the minimal version of
the index.

Please make sure that you update your instances with this new patch
and tell your users to do the same and the issue should be resolved.
Additionally, make sure you raise the timeout timer to something a tad
higher. Let me know if this has resolved it!

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/MISP/MISP/issues/2098#issuecomment-293992586, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AD7YX7Qxs4ndjKmcQ_JZ83uO2ClSx9Xxks5rvnEbgaJpZM4Mv4cy.

[iglocska]

Yup!

[iglocska]

I should also work on the error message "code 0" is about as useful as a solar powered flashlight.

[bambenek]

Which precise variable, i see nothing at 30 seconds in the config

On 4/13/2017 2:04 PM, Andras Iklody wrote:
>

Yup!

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/MISP/MISP/issues/2098#issuecomment-293993165, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AD7YX5O1ti9QElff28uRislIFS9Dni02ks5rvnGmgaJpZM4Mv4cy.

[iglocska]

Should be

max_execution_time

[iglocska]

Argh, not my day. The above patch wasn't working quite as intended, could you git pull again? Sorry about that.

[bambenek]

done

On 4/13/2017 2:32 PM, Andras Iklody wrote:
>

Argh, not my day. The above patch wasn't working quite as intended,
could you git pull again? Sorry about that.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/MISP/MISP/issues/2098#issuecomment-293999918, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AD7YX58yQ7yPNX0dju2rLimAExAw1xCqks5rvnhCgaJpZM4Mv4cy.

[iglocska]

Just tested and I can now pull from you, yay!

Fixed a few more edge-case issues, think we'll do another tagged bugfix-release tomorrow. This has been productive, thanks for the help!

[BurntToast-DFIR]

I'll try on Monday. Thanks for the swift resolution.
On Thu, 13 Apr 2017 at 21:45, Andras Iklody [e-mail]
wrote:

Just tested and I can now pull from you, yay!

Fixed a few more edge-case issues, think we'll do another tagged
bugfix-release tomorrow. This has been productive, thanks for the help!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/MISP/MISP/issues/2098#issuecomment-294016802, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AL4mDB1o7c6Rynf-MjjoQRAzoWVsiWOWks5rvolcgaJpZM4Mv4cy
.

[bambenek]

No worries... I'll try to make it over there at some point where we can
just do a bunch of this stuff in the same room and call it a beer and
hackathon at some point considering I'm giving you all the edge cases :)

On 4/13/2017 3:45 PM, Andras Iklody wrote:
>

Just tested and I can now pull from you, yay!

Fixed a few more edge-case issues, think we'll do another tagged
bugfix-release tomorrow. This has been productive, thanks for the help!

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/MISP/MISP/issues/2098#issuecomment-294016802, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AD7YX7W08mmdThoxjZS3nvPwUqoiVj-Xks5rvolcgaJpZM4Mv4cy.

[iglocska]

That would be fantastic! Your use-case is super interesting and the best
real-life stress test we've had so far. Really appreciate all the work you
do!

On Thu, Apr 13, 2017 at 10:47 PM, John Bambenek [e-mail]
wrote:

No worries... I'll try to make it over there at some point where we can
just do a bunch of this stuff in the same room and call it a beer and
hackathon at some point considering I'm giving you all the edge cases :)

On 4/13/2017 3:45 PM, Andras Iklody wrote:
>

Just tested and I can now pull from you, yay!

Fixed a few more edge-case issues, think we'll do another tagged
bugfix-release tomorrow. This has been productive, thanks for the help!

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/MISP/MISP/issues/2098#issuecomment-294016802, or
mute the thread
AD7YX7W08mmdThoxjZS3nvPwUqoiVj-Xks5rvolcgaJpZM4Mv4cy>.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/MISP/MISP/issues/2098#issuecomment-294017339, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADf6wKcHX07W77k2hkzoBtaTytHKRUnvks5rvontgaJpZM4Mv4cy
.

[adulau]

@bambenek there is one hackathon foreseen (and you should come over ;-) https://hackathon.hack.lu/ with MISP project and many others like TheHive.

[bambenek]

That's a little too soon for me to leave the country, $newbaby won't
even be 2 months old yet.

On 04/14/2017 01:53 AM, Alexandre Dulaunoy wrote:
>

@bambenek https://github.com/bambenek there is one hackathon
foreseen (and you should come over ;-) https://hackathon.hack.lu/ with
MISP project and many others like TheHive.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MISP/MISP/issues/2098#issuecomment-294102537, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AD7YXwhP4_TigBPQ8EmNGXBUsnkZJFO5ks5rvxfpgaJpZM4Mv4cy.

[iglocska]

The only acceptable excuse ;)

Massive congratulations!

[bambenek]

Thanks... I'll get over there soon enough :)

On 04/14/2017 10:37 AM, Andras Iklody wrote:
>

The only acceptable excuse ;)

Massive congratulations!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MISP/MISP/issues/2098#issuecomment-294175778, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AD7YX-zPeIUEyCdqW3Bg47U2uklYkqo2ks5rv5KsgaJpZM4Mv4cy.

[adulau]

@bambenek If you come to Luxembourg at some point, feel free to stop at our office ;-)

[bambenek]

It's on my agenda, just gotta find an excuse to get there... and I'm
good at finding excuses :)

On 04/14/2017 10:48 AM, Alexandre Dulaunoy wrote:
>

@bambenek https://github.com/bambenek If you come to Luxembourg at
some point, feel free to stop at our office ;-)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MISP/MISP/issues/2098#issuecomment-294177914, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AD7YX6Kq3Bt7Tqw2Kmo8w7NS4qCF5Efhks5rv5VjgaJpZM4Mv4cy.

[BurntToast-DFIR]

Fixed for me by upgrading to 2.4.72. Thanks guys. You are, as ever, freaking awesome.

[iglocska]

Excellent news! B-)

[adulau]

Then we will close the issue as it seems resolved ;-) Feel free to reopen or create a new issue if there another related issue.