misp use a lot of memory when you use search on event controller.
same issue when we search with curl
| Questions | Answers
|---------------------------|--------------------
| Type of issue | Bug, memory usage
| OS version (server) | Debian, ubuntu
| PHP version | 7.0,
| MISP version / git hash | 2.4.81
returns an event matching the search quickly
request time out because memory usage
sample :
r = misp.search(uuid='59313678-1068-434e-9974-42f502de0b81')
```bash
curl -H "Authorization: xxx" -H "Accept: application/json" -X get http://misp.local/events/restSearch/json/null/null/null/null/nul/null/null/null/null/null/null/null/59313678-1068-434e-9974-42f502de0b81
### Logs, screenshots, configuration dump, ...
```bash
Traceback (most recent call last):
File "test.py", line 34, in <module>
r = misp.search(uuid='59313678-1068-434e-9974-42f502de0b81')
File "/usr/local/lib/python3.4/dist-packages/pymisp-2.4.81-py3.4.egg/pymisp/api.py", line 1013, in search
return self.__query(session, 'restSearch/download', query, controller, async_callback)
File "/usr/local/lib/python3.4/dist-packages/pymisp-2.4.81-py3.4.egg/pymisp/api.py", line 838, in __query
return self._check_response(response)
File "/usr/local/lib/python3.4/dist-packages/pymisp-2.4.81-py3.4.egg/pymisp/api.py", line 213, in _check_response
raise PyMISPError('Unknown error: {}'.format(response.text))
pymisp.exceptions.PyMISPError: Unknown error: <br />
<b>Fatal error</b>: Allowed memory size of 134217728 bytes exhausted (tried to allocate 4096 bytes) in <b>/var/www/html/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php</b> on line <b>470</b><br />
<br />
``SQL
Query SELECTUser.id,User.password,User.org_id,User.server_id,User.email,User.autoalert,User.authkey,User.invited_by,User.gpgkey,User.certif_public,User.nids_sid,User.termsaccepted,User.newsread,User.role_id,User.change_pw,User.contactalert,User.disabled,User.expiration,User.current_login,User.last_login,User.force_logout,User.date_created,User.date_modified,Role.id,Role.name,Role.created,Role.modified,Role.perm_add,Role.perm_modify,Role.perm_modify_org,Role.perm_publish,Role.perm_delegate,Role.perm_sync,Role.perm_admin,Role.perm_audit,Role.perm_full,Role.perm_auth,Role.perm_site_admin,Role.perm_regexp_access,Role.perm_tagger,Role.perm_template,Role.perm_sharing_group,Role.perm_tag_editor,Role.perm_sighting,Role.default_role,Role.perm_object_template, (CASE WHEN (Role.perm_add+Role.perm_modify+Role.perm_publish= 3) THEN '3' WHEN (Role.perm_add+Role.perm_modify_org= 2) THEN '2' WHEN (Role.perm_add= 1) THEN '1' ELSE '0' END) ASRole__permission,Organisation.id,Organisation.name,Organisation.date_created,Organisation.date_modified,Organisation.description,Organisation.type,Organisation.nationality,Organisation.sector,Organisation.created_by,Organisation.uuid,Organisation.contacts,Organisation.local,Organisation.landingpage,Server.id,Server.url,Server.push_rulesFROMmisp.usersASUserLEFT JOINmisp.rolesASRoleON (User.role_id=Role.id) LEFT JOINmisp.organisationsASOrganisationON (User.org_id=Organisation.id) LEFT JOINmisp.serversASServerON (User.server_id=Server.id) WHEREUser.authkey= 'xxxx' LIMIT 1
Query SELECTUser.id,User.password,User.org_id,User.server_id,User.email,User.autoalert,User.authkey,User.invited_by,User.gpgkey,User.certif_public,User.nids_sid,User.termsaccepted,User.newsread,User.role_id,User.change_pw,User.contactalert,User.disabled,User.expiration,User.current_login,User.last_login,User.force_logout,User.date_created,User.date_modified,Role.id,Role.name,Role.created,Role.modified,Role.perm_add,Role.perm_modify,Role.perm_modify_org,Role.perm_publish,Role.perm_delegate,Role.perm_sync,Role.perm_admin,Role.perm_audit,Role.perm_full,Role.perm_auth,Role.perm_site_admin,Role.perm_regexp_access,Role.perm_tagger,Role.perm_template,Role.perm_sharing_group,Role.perm_tag_editor,Role.perm_sighting,Role.default_role,Role.perm_object_template, (CASE WHEN (Role.perm_add+Role.perm_modify+Role.perm_publish= 3) THEN '3' WHEN (Role.perm_add+Role.perm_modify_org= 2) THEN '2' WHEN (Role.perm_add= 1) THEN '1' ELSE '0' END) ASRole__permission,Organisation.id,Organisation.name,Organisation.date_created,Organisation.date_modified,Organisation.description,Organisation.type,Organisation.nationality,Organisation.sector,Organisation.created_by,Organisation.uuid,Organisation.contacts,Organisation.local,Organisation.landingpage,Server.id,Server.url,Server.push_rulesFROMmisp.usersASUserLEFT JOINmisp.rolesASRoleON (User.role_id=Role.id) LEFT JOINmisp.organisationsASOrganisationON (User.org_id=Organisation.id) LEFT JOINmisp.serversASServerON (User.server_id=Server.id) WHEREUser.authkey= 'xxxx' LIMIT 1
Query SELECTAdminSetting.id,AdminSetting.setting,AdminSetting.valueFROMmisp.admin_settingsASAdminSettingWHEREsetting= 'clean_db' LIMIT 1
Query SELECTAdminSetting.id,AdminSetting.setting,AdminSetting.valueFROMmisp.admin_settingsASAdminSettingWHEREsetting= 'db_version'
Query SELECTRole.id,Role.name,Role.created,Role.modified,Role.perm_add,Role.perm_modify,Role.perm_modify_org,Role.perm_publish,Role.perm_delegate,Role.perm_sync,Role.perm_admin,Role.perm_audit,Role.perm_full,Role.perm_auth,Role.perm_site_admin,Role.perm_regexp_access,Role.perm_tagger,Role.perm_template,Role.perm_sharing_group,Role.perm_tag_editor,Role.perm_sighting,Role.default_role,Role.perm_object_template, (CASE WHEN (Role.perm_add+Role.perm_modify+Role.perm_publish= 3) THEN '3' WHEN (Role.perm_add+Role.perm_modify_org= 2) THEN '2' WHEN (Role.perm_add= 1) THEN '1' ELSE '0' END) ASRole__permissionFROMmisp.rolesASRoleWHERERole.id= 1 LIMIT 1
Query SELECTRegexp.id,Regexp.regexp,Regexp.replacement,Regexp.typeFROMmisp.regexpASRegexpWHERE 1 = 1 ORDER BYidASC
Query SELECTRegexp.id,Regexp.regexp,Regexp.replacement,Regexp.typeFROMmisp.regexpASRegexpWHERE 1 = 1 ORDER BYidASC
Query SELECTShadowAttribute.event_id,ShadowAttribute.event_org_idFROMmisp.shadow_attributesASShadowAttributeWHEREShadowAttribute.event_org_id= 1 ANDShadowAttribute.deleted= '0' ORDER BYShadowAttribute.event_idDESC,ShadowAttribute.typeASC
Quit
Connect misp@localhost as anonymous on misp
Query SET NAMES utf8
Query SELECTRegexp.id,Regexp.regexp,Regexp.replacement,Regexp.typeFROMmisp.regexpASRegexpWHERE 1 = 1 ORDER BYidASC
Query SELECTUser.id,User.password,User.org_id,User.server_id,User.email,User.autoalert,User.authkey,User.invited_by,User.gpgkey,User.certif_public,User.nids_sid,User.termsaccepted,User.newsread,User.role_id,User.change_pw,User.contactalert,User.disabled,User.expiration,User.current_login,User.last_login,User.force_logout,User.date_created,User.date_modified,Role.id,Role.name,Role.created,Role.modified,Role.perm_add,Role.perm_modify,Role.perm_modify_org,Role.perm_publish,Role.perm_delegate,Role.perm_sync,Role.perm_admin,Role.perm_audit,Role.perm_full,Role.perm_auth,Role.perm_site_admin,Role.perm_regexp_access,Role.perm_tagger,Role.perm_template,Role.perm_sharing_group,Role.perm_tag_editor,Role.perm_sighting,Role.default_role,Role.perm_object_template, (CASE WHEN (Role.perm_add+Role.perm_modify+Role.perm_publish= 3) THEN '3' WHEN (Role.perm_add+Role.perm_modify_org= 2) THEN '2' WHEN (Role.perm_add= 1) THEN '1' ELSE '0' END) ASRole__permission,Organisation.id,Organisation.name,Organisation.date_created,Organisation.date_modified,Organisation.description,Organisation.type,Organisation.nationality,Organisation.sector,Organisation.created_by,Organisation.uuid,Organisation.contacts,Organisation.local,Organisation.landingpage,Server.id,Server.url,Server.push_rulesFROMmisp.usersASUserLEFT JOINmisp.rolesASRoleON (User.role_id=Role.id) LEFT JOINmisp.organisationsASOrganisationON (User.org_id=Organisation.id) LEFT JOINmisp.serversASServerON (User.server_id=Server.id) WHEREUser.authkey= 'xxxx' LIMIT 1
Query SELECTUser.id,User.password,User.org_id,User.server_id,User.email,User.autoalert,User.authkey,User.invited_by,User.gpgkey,User.certif_public,User.nids_sid,User.termsaccepted,User.newsread,User.role_id,User.change_pw,User.contactalert,User.disabled,User.expiration,User.current_login,User.last_login,User.force_logout,User.date_created,User.date_modified,Role.id,Role.name,Role.created,Role.modified,Role.perm_add,Role.perm_modify,Role.perm_modify_org,Role.perm_publish,Role.perm_delegate,Role.perm_sync,Role.perm_admin,Role.perm_audit,Role.perm_full,Role.perm_auth,Role.perm_site_admin,Role.perm_regexp_access,Role.perm_tagger,Role.perm_template,Role.perm_sharing_group,Role.perm_tag_editor,Role.perm_sighting,Role.default_role,Role.perm_object_template, (CASE WHEN (Role.perm_add+Role.perm_modify+Role.perm_publish= 3) THEN '3' WHEN (Role.perm_add+Role.perm_modify_org= 2) THEN '2' WHEN (Role.perm_add= 1) THEN '1' ELSE '0' END) ASRole__permission,Organisation.id,Organisation.name,Organisation.date_created,Organisation.date_modified,Organisation.description,Organisation.type,Organisation.nationality,Organisation.sector,Organisation.created_by,Organisation.uuid,Organisation.contacts,Organisation.local,Organisation.landingpage,Server.id,Server.url,Server.push_rulesFROMmisp.usersASUserLEFT JOINmisp.rolesASRoleON (User.role_id=Role.id) LEFT JOINmisp.organisationsASOrganisationON (User.org_id=Organisation.id) LEFT JOINmisp.serversASServerON (User.server_id=Server.id) WHEREUser.authkey= 'xxxx' LIMIT 1
Query SELECTAdminSetting.id,AdminSetting.setting,AdminSetting.valueFROMmisp.admin_settingsASAdminSettingWHEREsetting= 'clean_db' LIMIT 1
Query SELECTAdminSetting.id,AdminSetting.setting,AdminSetting.valueFROMmisp.admin_settingsASAdminSettingWHEREsetting= 'db_version'
Query SELECTRole.id,Role.name,Role.created,Role.modified,Role.perm_add,Role.perm_modify,Role.perm_modify_org,Role.perm_publish,Role.perm_delegate,Role.perm_sync,Role.perm_admin,Role.perm_audit,Role.perm_full,Role.perm_auth,Role.perm_site_admin,Role.perm_regexp_access,Role.perm_tagger,Role.perm_template,Role.perm_sharing_group,Role.perm_tag_editor,Role.perm_sighting,Role.default_role,Role.perm_object_template, (CASE WHEN (Role.perm_add+Role.perm_modify+Role.perm_publish= 3) THEN '3' WHEN (Role.perm_add+Role.perm_modify_org= 2) THEN '2' WHEN (Role.perm_add= 1) THEN '1' ELSE '0' END) ASRole__permissionFROMmisp.rolesASRoleWHERERole.id= 1 LIMIT 1
Query SELECTRegexp.id,Regexp.regexp,Regexp.replacement,Regexp.typeFROMmisp.regexpASRegexpWHERE 1 = 1 ORDER BYidASC
Query SELECTShadowAttribute.event_id,ShadowAttribute.event_org_idFROMmisp.shadow_attributesASShadowAttributeWHEREShadowAttribute.event_org_id= 1 ANDShadowAttribute.deleted= '0' ORDER BYShadowAttribute.event_idDESC,ShadowAttribute.typeASC
Quit
Connect misp@localhost as anonymous on misp
Query SET NAMES utf8
Query SELECTUser.id,User.password,User.org_id,User.server_id,User.email,User.autoalert,User.authkey,User.invited_by,User.gpgkey,User.certif_public,User.nids_sid,User.termsaccepted,User.newsread,User.role_id,User.change_pw,User.contactalert,User.disabled,User.expiration,User.current_login,User.last_login,User.force_logout,User.date_created,User.date_modified,Role.id,Role.name,Role.created,Role.modified,Role.perm_add,Role.perm_modify,Role.perm_modify_org,Role.perm_publish,Role.perm_delegate,Role.perm_sync,Role.perm_admin,Role.perm_audit,Role.perm_full,Role.perm_auth,Role.perm_site_admin,Role.perm_regexp_access,Role.perm_tagger,Role.perm_template,Role.perm_sharing_group,Role.perm_tag_editor,Role.perm_sighting,Role.default_role,Role.perm_object_template, (CASE WHEN (Role.perm_add+Role.perm_modify+Role.perm_publish= 3) THEN '3' WHEN (Role.perm_add+Role.perm_modify_org= 2) THEN '2' WHEN (Role.perm_add= 1) THEN '1' ELSE '0' END) ASRole__permission,Organisation.id,Organisation.name,Organisation.date_created,Organisation.date_modified,Organisation.description,Organisation.type,Organisation.nationality,Organisation.sector,Organisation.created_by,Organisation.uuid,Organisation.contacts,Organisation.local,Organisation.landingpage,Server.id,Server.url,Server.push_rulesFROMmisp.usersASUserLEFT JOINmisp.rolesASRoleON (User.role_id=Role.id) LEFT JOINmisp.organisationsASOrganisationON (User.org_id=Organisation.id) LEFT JOINmisp.serversASServerON (User.server_id=Server.id) WHEREUser.authkey= 'xxxx' LIMIT 1
Query SELECTUser.id,User.password,User.org_id,User.server_id,User.email,User.autoalert,User.authkey,User.invited_by,User.gpgkey,User.certif_public,User.nids_sid,User.termsaccepted,User.newsread,User.role_id,User.change_pw,User.contactalert,User.disabled,User.expiration,User.current_login,User.last_login,User.force_logout,User.date_created,User.date_modified,Role.id,Role.name,Role.created,Role.modified,Role.perm_add,Role.perm_modify,Role.perm_modify_org,Role.perm_publish,Role.perm_delegate,Role.perm_sync,Role.perm_admin,Role.perm_audit,Role.perm_full,Role.perm_auth,Role.perm_site_admin,Role.perm_regexp_access,Role.perm_tagger,Role.perm_template,Role.perm_sharing_group,Role.perm_tag_editor,Role.perm_sighting,Role.default_role,Role.perm_object_template, (CASE WHEN (Role.perm_add+Role.perm_modify+Role.perm_publish= 3) THEN '3' WHEN (Role.perm_add+Role.perm_modify_org= 2) THEN '2' WHEN (Role.perm_add= 1) THEN '1' ELSE '0' END) ASRole__permission,Organisation.id,Organisation.name,Organisation.date_created,Organisation.date_modified,Organisation.description,Organisation.type,Organisation.nationality,Organisation.sector,Organisation.created_by,Organisation.uuid,Organisation.contacts,Organisation.local,Organisation.landingpage,Server.id,Server.url,Server.push_rulesFROMmisp.usersASUserLEFT JOINmisp.rolesASRoleON (User.role_id=Role.id) LEFT JOINmisp.organisationsASOrganisationON (User.org_id=Organisation.id) LEFT JOINmisp.serversASServerON (User.server_id=Server.id) WHEREUser.authkey= 'xxxx' LIMIT 1
Query SELECTAdminSetting.id,AdminSetting.setting,AdminSetting.valueFROMmisp.admin_settingsASAdminSettingWHEREsetting= 'clean_db' LIMIT 1
Query SELECTAdminSetting.id,AdminSetting.setting,AdminSetting.valueFROMmisp.admin_settingsASAdminSettingWHEREsetting= 'db_version'
Query SELECTRole.id,Role.name,Role.created,Role.modified,Role.perm_add,Role.perm_modify,Role.perm_modify_org,Role.perm_publish,Role.perm_delegate,Role.perm_sync,Role.perm_admin,Role.perm_audit,Role.perm_full,Role.perm_auth,Role.perm_site_admin,Role.perm_regexp_access,Role.perm_tagger,Role.perm_template,Role.perm_sharing_group,Role.perm_tag_editor,Role.perm_sighting,Role.default_role,Role.perm_object_template, (CASE WHEN (Role.perm_add+Role.perm_modify+Role.perm_publish= 3) THEN '3' WHEN (Role.perm_add+Role.perm_modify_org= 2) THEN '2' WHEN (Role.perm_add= 1) THEN '1' ELSE '0' END) ASRole__permissionFROMmisp.rolesASRoleWHERERole.id= 1 LIMIT 1
Query SELECTRegexp.id,Regexp.regexp,Regexp.replacement,Regexp.typeFROMmisp.regexpASRegexpWHERE 1 = 1 ORDER BYidASC
Query SELECTRegexp.id,Regexp.regexp,Regexp.replacement,Regexp.typeFROMmisp.regexpASRegexpWHERE 1 = 1 ORDER BYidASC
Query SELECTShadowAttribute.event_id,ShadowAttribute.event_org_idFROMmisp.shadow_attributesASShadowAttributeWHEREShadowAttribute.event_org_id= 1 ANDShadowAttribute.deleted= '0' ORDER BYShadowAttribute.event_idDESC,ShadowAttribute.typeASC
Query SELECTAttribute.event_idFROMmisp.attributesASAttributeLEFT JOINmisp.eventsASEventON (Attribute.event_id=Event.id) WHERE ((((Attribute.uuid= '59313678-1068-434e-9974-42f502de0b81') OR (Event.uuid= '59313678-1068-434e-9974-42f502de0b81'))) AND (Attribute.deleted= '0')) GROUP BYAttribute.event_idORDER BYAttribute.event_idDESC
Query SELECTSharingGroup.idFROMmisp.sharing_groupsASSharingGroupWHERE 1 = 1
Query SELECTSharingGroup.*,Organisation.id,SharingGroup.idFROMmisp.sharing_groupsASSharingGroupLEFT JOINmisp.organisationsASOrganisationON (SharingGroup.org_id=Organisation.id) WHERE 1 = 1
Query SELECTEvent.id,Event.orgc_id,Event.org_id,Event.date,Event.threat_level_id,Event.info,Event.published,Event.uuid,Event.attribute_count,Event.analysis,Event.timestamp,Event.distribution,Event.proposal_email_lock,Event.user_id,Event.locked,Event.publish_timestamp,Event.sharing_group_id,Event.disable_correlation,User.email,ThreatLevel.name,Org.id,Org.name,Org.uuid,Orgc.id,Orgc.name,Orgc.uuid,User.id,ThreatLevel.idFROMmisp.eventsASEventLEFT JOINmisp.usersASUserON (Event.user_id=User.id) LEFT JOINmisp.threat_levelsASThreatLevelON (Event.threat_level_id=ThreatLevel.id) LEFT JOINmisp.organisationsASOrgON (Event.org_id=Org.id) LEFT JOINmisp.organisationsASOrgcON (Event.orgc_id=Orgc.id) WHEREEvent.id= 436
Query SELECTAttribute.id,Attribute.type,Attribute.category,Attribute.to_ids,Attribute.uuid,Attribute.event_id,Attribute.distribution,Attribute.timestamp,Attribute.comment,Attribute.sharing_group_id,Attribute.deleted,Attribute.disable_correlation,Attribute.object_id,Attribute.object_relation, (CASE WHENAttribute.value2= '' THENAttribute.value1ELSE CONCAT(Attribute.value1, '|',Attribute.value2) END) ASAttribute__valueFROMmisp.attributesASAttributeWHEREAttribute.deleted= '0' ANDAttribute.event_id= (436)
Query SELECTAttributeTag.id,AttributeTag.attribute_id,AttributeTag.event_id,AttributeTag.tag_idFROMmisp.attribute_tagsASAttributeTagWHEREAttributeTag.attribute_idIN (82983, 82984, 82985, 82986, 82987, 82988, 82989, 82990, 82991, 82992, 82993, 82994, 82995, 82996, 82997, 82998, 82999, 83000, 83001, 83002, 83003, 83004, 83005, 83006, 83007, 83008, 83009, 83010, 83011, 83012, 83013, 83014, 83015, 83016, 83017, 83018, 83019, 83020, 83021, 83022, 83023, 83024, 83025, 83026, 83027, 83028, 83029, 83030, 83031, 83032, 83033, 83034, 83035, 83036, 83037, 83038, 83039, 83040, 83041, 83042, 83043, 83044)
Query SELECTShadowAttribute.id,ShadowAttribute.type,ShadowAttribute.category,ShadowAttribute.to_ids,ShadowAttribute.uuid,ShadowAttribute.event_uuid,ShadowAttribute.event_id,ShadowAttribute.old_id,ShadowAttribute.comment,ShadowAttribute.org_id,ShadowAttribute.proposal_to_delete,ShadowAttribute.timestamp, (CASE WHENShadowAttribute.value2= '' THENShadowAttribute.value1ELSE CONCAT(ShadowAttribute.value1, '|',ShadowAttribute.value2) END) ASShadowAttribute__valueFROMmisp.shadow_attributesASShadowAttributeWHEREdeleted= 0 ANDShadowAttribute.event_id= (436)
Query SELECT *, id,Object.event_idFROMmisp.objectsASObjectWHEREObject.deleted= '0' ANDObject.event_id= (436)
Query SELECTEventTag.id,EventTag.event_id,EventTag.tag_idFROMmisp.event_tagsASEventTagWHEREEventTag.event_id= (436)
Query SELECTTag.id,Tag.name,Tag.colour,Tag.exportable,Tag.org_id,Tag.hide_tagFROMmisp.tagsASTagWHERETag.id= 1 ANDexportable= 1
Query SELECTCorrelation.event_idFROMmisp.correlationsASCorrelationWHERECorrelation.1_event_id= (436) GROUP BYCorrelation.event_idORDER BYCorrelation.event_idDESC
Query SELECTEvent.id,Event.date,Event.threat_level_id,Event.info,Event.published,Event.uuid,Event.analysis,Event.timestamp,Event.distribution,Event.org_id,Event.orgc_id,Org.id,Org.name,Org.uuid,Orgc.id,Orgc.name,Orgc.uuidFROMmisp.eventsASEventLEFT JOINmisp.organisationsASOrgON (Event.org_id=Org.id) LEFT JOINmisp.organisationsASOrgcON (Event.orgc_id=Orgc.id) WHEREEvent.idIN (680, 670, 623, 570, 403) ORDER BYEvent.dateDESC
Query SELECTCorrelation.* FROMmisp.correlationsASCorrelationWHERECorrelation.1_event_id= 436 LIMIT 5000
Query SELECTShadowAttributeCorrelation.* FROMmisp.shadow_attribute_correlationsASShadowAttributeCorrelationWHEREShadowAttributeCorrelation.1_event_id= 436 LIMIT 5000
Query SELECTSighting.id,Sighting.attribute_id,Sighting.event_id,Sighting.org_id,Sighting.date_sighting,Sighting.uuid,Sighting.source,Sighting.type,Organisation.id,Organisation.uuid,Organisation.nameFROMmisp.sightingsASSightingLEFT JOINmisp.organisationsASOrganisationON (Sighting.org_id=Organisation.id) WHERESighting.event_id= 436
Query SELECTWhitelist.nameFROMmisp.whitelistASWhitelistWHERE 1 = 1
Query SELECTEvent.id,Event.orgc_id,Event.org_id,Event.date,Event.threat_level_id,Event.info,Event.published,Event.uuid,Event.attribute_count,Event.analysis,Event.timestamp,Event.distribution,Event.proposal_email_lock,Event.user_id,Event.locked,Event.publish_timestamp,Event.sharing_group_id,Event.disable_correlation,User.email,ThreatLevel.name,Org.id,Org.name,Org.uuid,Orgc.id,Orgc.name,Orgc.uuid,User.id,ThreatLevel.idFROMmisp.eventsASEventLEFT JOINmisp.usersASUserON (Event.user_id=User.id) LEFT JOINmisp.threat_levelsASThreatLevelON (Event.threat_level_id=ThreatLevel.id) LEFT JOINmisp.organisationsASOrgON (Event.org_id=Org.id) LEFT JOINmisp.organisationsASOrgcON (Event.orgc_id=Orgc.id) WHERE 1 = 1
Query SELECTAttribute.id,Attribute.type,Attribute.category,Attribute.to_ids,Attribute.uuid,Attribute.event_id,Attribute.distribution,Attribute.timestamp,Attribute.comment,Attribute.sharing_group_id,Attribute.deleted,Attribute.disable_correlation,Attribute.object_id,Attribute.object_relation, (CASE WHENAttribute.value2= '' THENAttribute.value1ELSE CONCAT(Attribute.value1, '|',Attribute.value2) END) ASAttribute__valueFROMmisp.attributesASAttributeWHEREAttribute.deleted= '0' ANDAttribute.event_idIN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, ...)
Query SELECTUser.id,User.password,User.org_id,User.server_id,User.email,User.autoalert,User.authkey,User.invited_by,User.gpgkey,User.certif_public,User.nids_sid,User.termsaccepted,User.newsread,User.role_id,User.change_pw,User.contactalert,User.disabled,User.expiration,User.current_login,User.last_login,User.force_logout,User.date_created,User.date_modifiedFROMmisp.usersASUserWHEREUser.authkey= 'xxxx' LIMIT 1
Query START TRANSACTION
Query INSERT INTOmisp.logs(org,model,model_id,email,action,title,change,created,user_id,description`) VALUES ('SYSTEM', 'User', 0, 'SYSTEM', 'auth_fail', 'Failed authentication using API key (xxxx)', '', '2017-10-12 12:20:48', 0, '')
Query COMMIT
Quit
There is a lot of duplicate query
IN in SQL uses a lot of load and is generally not to be used.
### for the curl :
uuid not in the response
response :
```json
{"response":[{
"Event": {
"id": "1",
"orgc_id": "2",
"org_id": "1",
"date": "2015-10-16",
"threat_level_id": "3",
"info": "Targeted Malware Attacks against NGO Linked to Attacks on Burmese Government Websites - Citizen Lab",
"published": true,
"uuid": "56caad17-c7d0-4005-bc41-51299062e56a",
"attribute_count": "36",
"analysis": "2",
"timestamp": "1466509649",
"distribution": "1",
"proposal_email_lock": false,
"locked": false,
"publish_timestamp": "1498133423",
"sharing_group_id": "0",
"disable_correlation": false,
"Org": {
"id": "1",
"name": "ORGNAME",
"uuid": "5947dc31-c300-40d4-825a-1d2d00000000"
},
"Orgc": {
"id": "2",
"name": "CUDESO",
"uuid": "56c42374-fdb8-4544-a218-41ffc0a8ab16"
},
"RelatedEvent": [
{
"Event": {
"id": "711",
"date": "2015-10-16",
"threat_level_id": "2",
"info": "OSINT Targeted Malware Attacks against NGO Linked to Attacks on Burmese Government Websites by Citizen Labs",
"published": true,
"uuid": "56240d98-a524-4386-8e43-8371950d210b",
"analysis": "2",
"timestamp": "1450794987",
"distribution": "1",
"org_id": "1",
"orgc_id": "4",
"Org": {
"id": "1",
"name": "ORGNAME",
"uuid": "5947dc31-c300-40d4-825a-1d2d00000000"
},
"Orgc": {
"id": "4",
"name": "CthulhuSPRL.be",
"uuid": "55f6ea5f-fd34-43b8-ac1d-40cb950d210f"
}
}
}
.....
5 events
}
],
"Galaxy": [],
"Object": [],
"Tag": [
{
"id": "1",
"name": "tlp:white",
"colour": "#00d622",
"exportable": true,
"hide_tag": false
}
]
}
}]}
the log for the crul request is very large
here a sample :
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 776 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 776 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 776
Query SELECT `GalaxyCluster`.`id`, `GalaxyCluster`.`uuid`, `GalaxyCluster`.`type`, `GalaxyCluster`.`value`, `GalaxyCluster`.`tag_name`, `GalaxyCluster`.`description`, `GalaxyCluster`.`galaxy_id`, `GalaxyCluster`.`source`, `GalaxyCluster`.`authors`, `GalaxyCluster`.`version`, `Galaxy`.`id`, `Galaxy`.`uuid`, `Galaxy`.`name`, `Galaxy`.`type`, `Galaxy`.`description`, `Galaxy`.`version`, `Galaxy`.`icon` FROM `misp`.`galaxy_clusters` AS `GalaxyCluster` LEFT JOIN `misp`.`galaxies` AS `Galaxy` ON (`GalaxyCluster`.`galaxy_id` = `Galaxy`.`id`) WHERE `GalaxyCluster`.`tag_name` = 'misp-galaxy:threat-actor=\"Packrat\"' LIMIT 1
Query SELECT `GalaxyElement`.`id`, `GalaxyElement`.`galaxy_cluster_id`, `GalaxyElement`.`key`, `GalaxyElement`.`value` FROM `misp`.`galaxy_elements` AS `GalaxyElement` WHERE `GalaxyElement`.`galaxy_cluster_id` = (3996)
Query SELECT `Tag`.`id` FROM `misp`.`tags` AS `Tag` WHERE `Tag`.`name` = 'misp-galaxy:threat-actor=\"Packrat\"' LIMIT 1
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (777) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` = (15) ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 777 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 777 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 777
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (778) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IN (794, 786, 783, 764, 742, 715, 685, 632, 623, 561, 551, 534, 524, 520, 486, 484, 474, 469, 433, 421, 408, 394, 367, 349, 345, 335, 285, 206, 177, 173, 165, 158, 156, 150, 143, 116, 65) ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 778 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 778 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 778
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (779) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IS NULL ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 779 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 779 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 779
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (780) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IN (585, 384, 254, 180) ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 780 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 780 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 780
Query SELECT `GalaxyCluster`.`id`, `GalaxyCluster`.`uuid`, `GalaxyCluster`.`type`, `GalaxyCluster`.`value`, `GalaxyCluster`.`tag_name`, `GalaxyCluster`.`description`, `GalaxyCluster`.`galaxy_id`, `GalaxyCluster`.`source`, `GalaxyCluster`.`authors`, `GalaxyCluster`.`version`, `Galaxy`.`id`, `Galaxy`.`uuid`, `Galaxy`.`name`, `Galaxy`.`type`, `Galaxy`.`description`, `Galaxy`.`version`, `Galaxy`.`icon` FROM `misp`.`galaxy_clusters` AS `GalaxyCluster` LEFT JOIN `misp`.`galaxies` AS `Galaxy` ON (`GalaxyCluster`.`galaxy_id` = `Galaxy`.`id`) WHERE `GalaxyCluster`.`tag_name` = 'misp-galaxy:tool=\"PlugX\"' LIMIT 1
Query SELECT `GalaxyElement`.`id`, `GalaxyElement`.`galaxy_cluster_id`, `GalaxyElement`.`key`, `GalaxyElement`.`value` FROM `misp`.`galaxy_elements` AS `GalaxyElement` WHERE `GalaxyElement`.`galaxy_cluster_id` = (4644)
Query SELECT `Tag`.`id` FROM `misp`.`tags` AS `Tag` WHERE `Tag`.`name` = 'misp-galaxy:tool=\"PlugX\"' LIMIT 1
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (781) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IN (787, 740) ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 781 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 781 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 781
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (782) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IS NULL ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 782 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 782 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 782
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (783) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IN (786, 778, 764, 742, 632, 561, 551, 534, 524, 520, 486, 484, 474, 469, 433, 421, 408, 394, 349, 345, 335, 285, 206, 177, 173, 116) ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 783 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 783 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 783
Query SELECT `GalaxyCluster`.`id`, `GalaxyCluster`.`uuid`, `GalaxyCluster`.`type`, `GalaxyCluster`.`value`, `GalaxyCluster`.`tag_name`, `GalaxyCluster`.`description`, `GalaxyCluster`.`galaxy_id`, `GalaxyCluster`.`source`, `GalaxyCluster`.`authors`, `GalaxyCluster`.`version`, `Galaxy`.`id`, `Galaxy`.`uuid`, `Galaxy`.`name`, `Galaxy`.`type`, `Galaxy`.`description`, `Galaxy`.`version`, `Galaxy`.`icon` FROM `misp`.`galaxy_clusters` AS `GalaxyCluster` LEFT JOIN `misp`.`galaxies` AS `Galaxy` ON (`GalaxyCluster`.`galaxy_id` = `Galaxy`.`id`) WHERE `GalaxyCluster`.`tag_name` = 'misp-galaxy:ransomware=\"Uiwix Ransomware\"' LIMIT 1
Query SELECT `GalaxyElement`.`id`, `GalaxyElement`.`galaxy_cluster_id`, `GalaxyElement`.`key`, `GalaxyElement`.`value` FROM `misp`.`galaxy_elements` AS `GalaxyElement` WHERE `GalaxyElement`.`galaxy_cluster_id` = (3883)
Query SELECT `Tag`.`id` FROM `misp`.`tags` AS `Tag` WHERE `Tag`.`name` = 'misp-galaxy:ransomware=\"Uiwix Ransomware\"' LIMIT 1
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (784) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IS NULL ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 784 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 784 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 784
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (785) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IN (730, 680, 666, 648, 552, 486, 474, 433, 407, 403, 337, 335, 264, 261, 223, 126) ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 785 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 785 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 785
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (786) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IN (794, 783, 778, 764, 742, 632, 561, 551, 544, 534, 524, 520, 486, 484, 474, 469, 460, 433, 421, 408, 394, 349, 347, 345, 335, 285, 264, 206, 177, 173, 116) ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 786 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 786 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 786
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (787) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IN (781, 740, 731, 729, 711, 654, 645, 625, 593, 561, 528, 482, 468, 462, 453, 443, 419, 417, 395, 385, 363, 358, 346, 309, 291, 290, 287, 279, 261, 158, 128, 123, 118, 61, 3, 1) ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 787 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 787 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 787
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (788) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IN (730, 716, 681, 563, 552, 391, 178) ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 788 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 788 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 788
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (789) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IS NULL ORDER BY `Event`.`date` DESC
Query SELECT `Correlation`.* FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = 789 LIMIT 5000
Query SELECT `ShadowAttributeCorrelation`.* FROM `misp`.`shadow_attribute_correlations` AS `ShadowAttributeCorrelation` WHERE `ShadowAttributeCorrelation`.`1_event_id` = 789 LIMIT 5000
Query SELECT `Sighting`.`id`, `Sighting`.`attribute_id`, `Sighting`.`event_id`, `Sighting`.`org_id`, `Sighting`.`date_sighting`, `Sighting`.`uuid`, `Sighting`.`source`, `Sighting`.`type`, `Organisation`.`id`, `Organisation`.`uuid`, `Organisation`.`name` FROM `misp`.`sightings` AS `Sighting` LEFT JOIN `misp`.`organisations` AS `Organisation` ON (`Sighting`.`org_id` = `Organisation`.`id`) WHERE `Sighting`.`event_id` = 789
Query SELECT `GalaxyCluster`.`id`, `GalaxyCluster`.`uuid`, `GalaxyCluster`.`type`, `GalaxyCluster`.`value`, `GalaxyCluster`.`tag_name`, `GalaxyCluster`.`description`, `GalaxyCluster`.`galaxy_id`, `GalaxyCluster`.`source`, `GalaxyCluster`.`authors`, `GalaxyCluster`.`version`, `Galaxy`.`id`, `Galaxy`.`uuid`, `Galaxy`.`name`, `Galaxy`.`type`, `Galaxy`.`description`, `Galaxy`.`version`, `Galaxy`.`icon` FROM `misp`.`galaxy_clusters` AS `GalaxyCluster` LEFT JOIN `misp`.`galaxies` AS `Galaxy` ON (`GalaxyCluster`.`galaxy_id` = `Galaxy`.`id`) WHERE `GalaxyCluster`.`tag_name` = 'misp-galaxy:ransomware=\"WannaCry\"' LIMIT 1
Query SELECT `GalaxyElement`.`id`, `GalaxyElement`.`galaxy_cluster_id`, `GalaxyElement`.`key`, `GalaxyElement`.`value` FROM `misp`.`galaxy_elements` AS `GalaxyElement` WHERE `GalaxyElement`.`galaxy_cluster_id` = (3595)
Query SELECT `Tag`.`id` FROM `misp`.`tags` AS `Tag` WHERE `Tag`.`name` = 'misp-galaxy:ransomware=\"WannaCry\"' LIMIT 1
Query SELECT `Correlation`.`event_id` FROM `misp`.`correlations` AS `Correlation` WHERE `Correlation`.`1_event_id` = (790) GROUP BY `Correlation`.`event_id` ORDER BY `Correlation`.`event_id` DESC
Query SELECT `Event`.`id`, `Event`.`date`, `Event`.`threat_level_id`, `Event`.`info`, `Event`.`published`, `Event`.`uuid`, `Event`.`analysis`, `Event`.`timestamp`, `Event`.`distribution`, `Event`.`org_id`, `Event`.`orgc_id`, `Org`.`id`, `Org`.`name`, `Org`.`uuid`, `Orgc`.`id`, `Orgc`.`name`, `Orgc`.`uuid` FROM `misp`.`events` AS `Event` LEFT JOIN `misp`.`organisations` AS `Org` ON (`Event`.`org_id` = `Org`.`id`) LEFT JOIN `misp`.`organisations` AS `Orgc` ON (`Event`.`orgc_id` = `Orgc`.`id`) WHERE `Event`.`id` IN (614, 187, 96) ORDER BY `Event`.`date` DESC
Experiencing the same issue, the load of mysqld has drastically increased since the upgrade to 2.4.81. A lot of searches ends up with a HTTP 500, an empty reply or a long delay in response.
Questions | Answers
-- | --
Type of issue | Bug, memory usage
OS version (server) | Ubuntu 16.04.2
PHP version | 7.1.6
MISP version / git hash | 2.4.81
I have the same issue...
I followed an advice and applied the following setting in /etc/php/7.1/apache2/php.ini, but the problem is still relevant:
; Maximum amount of memory a script may consume (128MB)
; http://php.net/memory-limit
memory_limit = 4096M
I believe that this is a really important problem. Because of this issue, it's difficult to integrate other TI platforms and utilities with MISP.
Please, pay attention to this problem.
Will have a look asap at what could have caused the difference, odd.
Hmph, I am not seeing it on my end, could you test with the latest commit? Also, how many attributes do you have in your instance?
(list attributes, look at the count at the bottom of the page)
@iglocska,
I have 1696193 attributes in my instance. I use latest release version v2.4.81 (77c9d2864e6a6f85e5c1026bf777e1a5cb3019e3).
Python code that reproduces the problem (PyMISP v2.4.81):
from pymisp import PyMISP
from keys import misp_key, misp_url
misp_obj = PyMISP(misp_url, misp_key, True, 'json', debug=True)
event = misp_obj.search(uuid='030ce6ec-f7bd-4abc-a965-5a6b02ab6d28')
Error message from /var/www/MISP/app/tmp/logs/error.log:
2017-10-31 12:12:47 Error: [InternalErrorException] Internal Server Error
Request URL: /events/restSearch/json/null/null/null/null/nul/null/null/null/null/null/null/null/030ce6ec-f7bd-4abc-a965-5a6b02ab6d28
Stack Trace:
#0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Error/ErrorHandler.php(212): ErrorHandler::handleFatalError(1, 'Allowed memory ...', '/var/www/MISP/a...', 277)
#1 /var/www/MISP/app/Lib/cakephp/lib/Cake/Core/App.php(970): ErrorHandler::handleError(1, 'Allowed memory ...', '/var/www/MISP/a...', 277, Array)
#2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Core/App.php(943): App::_checkFatalError()
#3 [internal function]: App::shutdown()
#4 {main}
Then I've updated my MISP instance to latest commit (v2.4.82 b7b09b7cdb92bf15e2bc9853f10f121fccf26d32) and PyMISP to v2.4.81.2 and the problem with search event by its UUID was successfully solved! I also tried to search by attributes and it was successfully executed too.
So for me the problem was solved by updating my MISP instance to latest commit.
Awesome, there was a bugged version indeed that was resolved a short while
ago, was hoping it was this. Thanks for the feedback!
On Oct 31, 2017 4:26 PM, "Alex Bolshakov" notifications@github.com wrote:
@iglocska https://github.com/iglocska,
I have 1696193 attributes in my instance. I use latest release version
v2.4.81 (77c9d28
https://github.com/MISP/MISP/commit/77c9d2864e6a6f85e5c1026bf777e1a5cb3019e3
).
Python code that reproduces the problem (PyMISP v2.4.81):from pymisp import PyMISP
from keys import misp_key, misp_url
misp_obj = PyMISP(misp_url, misp_key, True, 'json', debug=True)
event = misp_obj.search(uuid='030ce6ec-f7bd-4abc-a965-5a6b02ab6d28')Error message from /var/www/MISP/app/tmp/logs/error.log:
2017-10-31 12:12:47 Error: [InternalErrorException] Internal Server Error
Request URL: /events/restSearch/json/null/null/null/null/nul/null/null/null/null/null/null/null/030ce6ec-f7bd-4abc-a965-5a6b02ab6d28
Stack Trace:0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Error/ErrorHandler.php(212): ErrorHandler::handleFatalError(1, 'Allowed memory ...', '/var/www/MISP/a...', 277)
1 /var/www/MISP/app/Lib/cakephp/lib/Cake/Core/App.php(970): ErrorHandler::handleError(1, 'Allowed memory ...', '/var/www/MISP/a...', 277, Array)
2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Core/App.php(943): App::_checkFatalError()
3 [internal function]: App::shutdown()
4 {main}
Then I've updated my MISP instance to latest commit (v2.4.82 b7b09b7
https://github.com/MISP/MISP/commit/b7b09b7cdb92bf15e2bc9853f10f121fccf26d32)
and PyMISP to v2.4.81.2 and the problem with search event by its UUID was
successfully solved! I also tried to search by attributes and it was
successfully executed too.So for me the problem was solved by updating my MISP instance to latest
commit.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MISP/MISP/issues/2554#issuecomment-340798569, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADf6wPqJ4jLh9s3GK6D7fs9KocKthAFoks5sxzwkgaJpZM4P27rJ
.
Hi,
I am having the same issue and updated the MISP install to latest commit as suggested in this thread, but still get an error while only while searching events. For example when I search for "locky" or "alienvault" in the event search, get an error: "An Internal Error Has Occurred.
Error: An Internal Error Has Occurred."
And the error in /var/log/apache2/misp.local_error.log is:
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/Database/Mysql.php on line 277, referer: https://
Appreciate any help.
Your max memory is set to the default value in PHP, please raise it to
something higher - you will often deal with data sizes that exceed 128 megs
when exporting in MISP.
Just change max_memory in php.ini
On Mon, Nov 27, 2017 at 6:30 PM, Spj28 notifications@github.com wrote:
Hi,
I am having the same issue and updated the MISP install to latest commit
as suggested in this thread, but still get an error while searching the
events. I am searching for "Locky" in the event search. The error in web
interface is "An Internal Error Has Occurred.
Error: An Internal Error Has Occurred."And error in /var/log/apache2/misp.local_error.log is:
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried
to allocate 20480 bytes) in /var/www/MISP/app/Lib/cakephp/
lib/Cake/Model/Datasource/Database/Mysql.php on line 277, referer:
https:///events/indexAppreciate any help.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MISP/MISP/issues/2554#issuecomment-347258896, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADf6wBQZnZ1rUDT0Qq4ZVyPTCxosXr7Pks5s6vG7gaJpZM4P27rJ
.
Thanks @iglocska. Increasing the memory helped.
I suppose this has been solved. Feel free to reopen if this is not the case.
Hi there, my MISP instance also started to reproduce the problems mentioned above.
The instance's version is 2.4.119 and so far it was working fine during ReST searches.
I think this bug is still present somewhere in the system.
My instance currently stores 219243 attributes, which shouldn't be a problem, right?
For the search I'm using the pymisp.PyMISP.search(type_attributes=['ip-src', 'ip-dst', 'domain|ip', 'ip-src|port', 'ip-dst|port'], values="an-ip-addr") method.
220k is peanuts, so indeed it should work, that is weird. will try to reproduce it.
Just to understand a bit more about your instance: how large is the largest event that you have (in #attributes)?
ok, went back to the drawing board. It should hopefully work out of the box for you - we cannot account for every variable so with some headroom it should work fine on the use-cases we see.
However, we now have some new tools that you can use to adjust the parameters yourself:
in administration->server settings -> MISP you now have two new settings:
MISP.default_attribute_memory_coefficient: this controls the number of attributes that can go into 1MB of memory (we guestimate 80 by default, but if you add a bunch of tags to each attribute in general then you might want to go a bit lower).
MISP.default_event_memory_divisor: this controls the ratio of MB alotted for each attribute. Meaning that if you have 80 set in the previous setting and 2 set in this setting, then each 40 attributes in an event will require an extra MB of memory according to the internal clustering engine.
Let us know if this solves it for you.