Misp: Python library using the MISP Rest API

[WakandaKing]

I want to use the PyMISP for MISP Rest API, I have followed the install instructions as specified in https://github.com/MISP/PyMISP . My problem starts at the part where test if the URL and API ley are correct
"
cd examples
python3 last.py -l 10
"
I get the errors below:

root@ubuntu:/PyMISP/examples# python3 last.py -l 10
Traceback (most recent call last):
File "/home/userA/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/home/userA/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/home/userA/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/home/userA/.local/lib/python3.5/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/home/userA/.local/lib/python3.5/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.5/ssl.py", line 377, in wrap_socket
_context=self)
File "/usr/lib/python3.5/ssl.py", line 752, in __init__
self.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 988, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 633, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/userA/.local/lib/python3.5/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/home/userA/.local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/userA/.local/lib/python3.5/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ubuntu', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/pymisp-2.4.80-py3.5.egg/pymisp/api.py", line 113, in __init__
response = self.get_recommended_api_version()
File "/usr/local/lib/python3.5/dist-packages/pymisp-2.4.80-py3.5.egg/pymisp/api.py", line 1120, in get_recommended_api_version
response = session.get(url)
File "/home/userA/.local/lib/python3.5/site-packages/requests/sessions.py", line 521, in get
return self.request('GET', url, *kwargs)
File "/home/userA/.local/lib/python3.5/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, *send_kwargs)
File "/home/userA/.local/lib/python3.5/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/home/userA/.local/lib/python3.5/site-packages/requests/adapters.py", line 506, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='ubuntu', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "last.py", line 41, in
misp = init(misp_url, misp_key)
File "last.py", line 15, in init
return PyMISP(url, key, misp_verifycert, 'json')
File "/usr/local/lib/python3.5/dist-packages/pymisp-2.4.80-py3.5.egg/pymisp/api.py", line 127, in __init__
raise PyMISPError('Unable to connect to MISP ({}). Please make sure the API key and the URL are correct (http/https is required): {}'.format(self.root_url, e))
pymisp.exceptions.PyMISPError: Unable to connect to MISP (https://ubuntu/). Please make sure the API key and the URL are correct (http/https is required): HTTPSConnectionPool(host='ubuntu', port=443): Max retries exceeded with url: /servers/getPyMISPVersion.json (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))
root@ubuntu:/PyMISP/examples# exit

I don't know what the problem is, because i have copied the URL and API key as it is on the web interface

[FloatingGhost]

I don't know what the problem is
"certificate verify failed"

Boy I do wonder what the problem is.

Now you know the keys.py file you set up to run last.py?

You see the setting that says misp_verifycert?

Set it to false.

[iglocska]

I also got the hint from @rommelfs that it may be insufficient voltage from the wall. ;)

[WakandaKing]

I just did that and excuted "python3 last.py -l 10" again,
Im now getting this :

root@ubuntu:/PyMISP/examples# exit python3 last.py -l 10 exit [K python3 last.py -l 10
/home/userA/.local/lib/python3.5/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
/home/userA/.local/lib/python3.5/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
/home/userA/.local/lib/python3.5/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
Traceback (most recent call last):
File "last.py", line 43, in
download_last(misp, args.last, args.output)
File "last.py", line 19, in download_last
result = m.download_last(last)
File "/usr/local/lib/python3.5/dist-packages/pymisp-2.4.80-py3.5.egg/pymisp/api.py", line 1077, in download_last
return self.search(last=last)
File "/usr/local/lib/python3.5/dist-packages/pymisp-2.4.80-py3.5.egg/pymisp/api.py", line 1013, in search
return self.__query(session, 'restSearch/download', query, controller, async_callback)
File "/usr/local/lib/python3.5/dist-packages/pymisp-2.4.80-py3.5.egg/pymisp/api.py", line 838, in __query
return self._check_response(response)
File "/usr/local/lib/python3.5/dist-packages/pymisp-2.4.80-py3.5.egg/pymisp/api.py", line 208, in _check_response
response.raise_for_status()
File "/home/userA/.local/lib/python3.5/site-packages/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://ubuntu/events/restSearch/download
root@ubuntu:/PyMISP/examples# exit

[iglocska]

Check /var/www/MISP/app/tmp/error.log for the reason for the 500. Just try it again, and check the output of error.log right after.

Please make sure you paste it here!

[WakandaKing]

i have doen as you said , here is the error.log

2017-09-28 13:18:14 Error: Fatal Error (1): Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in [/var/www/MISP/app/Lib/cakephp/lib/Cake/Utility/Hash.php, line 65]
2017-09-28 13:18:14 Error: [InternalErrorException] Internal Server Error
Request URL: /events/restSearch/download
Stack Trace:

0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Error/ErrorHandler.php(212): ErrorHandler::handleFatalError(1, 'Allowed memory ...', '/var/www/MISP/a...', 65)

1 /var/www/MISP/app/Lib/cakephp/lib/Cake/Core/App.php(970): ErrorHandler::handleError(1, 'Allowed memory ...', '/var/www/MISP/a...', 65, Array)

2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Core/App.php(943): App::_checkFatalError()

3 [internal function]: App::shutdown()

4 {main}

[iglocska]

You are using the default settings of PHP which is 128MB of memory. Raise it in your php.ini to something more fitting for 2017 - 1GB should be a bare minimum imho. The diagnostics page should give you some more guidance on the settings (administration -> server settings -> diagnostics)

[WakandaKing]

Thanks @iglocska :+1: it worked , it does show the events on the terminal i just dont know where it puts them (where it stores them ) when i execute python3 last.py -l 10

[FloatingGhost]

it does show the events on the terminal

i just dont know where it puts them

Sometimes words escape me

[adulau]

@WakandaKing Great. Now everything is fine. We let you dig into your deepest part of your computer to find the events ;-)

[WakandaKing]

Lol @FloatingGhost Im sorry for confusing you, The events get printed on the screen when i execute the command, but i am not sure if they do get stored in a file that i can later view.

[FloatingGhost]

I'll take "what is file redirection" for $1000 please alex

[adulau]

~~
What was the significant file system improvement introduced by "UNIX Time-Sharing System" as described by D. M. Ritchie and K. Thompson in 1974?
~~
or
~~
What was the last word in the "The Treachery of Images" from the famous Belgian painter René Magritte?
~~

[iglocska]

I think this may be a case for some forensics on your drive. ;)