Elastalert: ImportError: failed to find libmagic. Check your installation

[lucas-rudd]

I believe this may be an issue with the magic package used by thehive4py/api.py

After installing elastalert v 0.1.3.6 I get the following stacktrace when I try to run an elastalert test with a folder of rules

Traceback (most recent call last):
  File "/home/elastalert/.local/bin/elastalert", line 11, in <module>
    load_entry_point('elastalert==0.1.36', 'console_scripts', 'elastalert')()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2691, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2322, in load
    return self.resolve()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2328, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/elastalert/.local/lib/python2.7/site-packages/elastalert/elastalert.py", line 21, in <module>
    from alerts import DebugAlerter
  File "/home/elastalert/.local/lib/python2.7/site-packages/elastalert/alerts.py", line 31, in <module>
    from thehive4py.api import TheHiveApi
  File "/home/elastalert/.local/lib/python2.7/site-packages/thehive4py/api.py", line 7, in <module>
    import magic
  File "/home/elastalert/.local/lib/python2.7/site-packages/magic.py", line 181, in <module>
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation
RMTMACSIERRA01:infrastructure-cm lucas.rudd$ docker run --rm -it -e ES_HOST=search-pss-pf-prod-es-51-gcm4nbuhxjuo6telgbszkp3i7u.us-east-1.es.amazonaws.com -e AWS_PROFILE=asurion-soluto-nonprod.dev -v ~/.aws:/home/elastalert/.aws  -v ~/Desktop/rules/:/home/elastalert/rules --entrypoint elastalert asurionpss/elastalert:test --verbose --es_debug
Traceback (most recent call last):
  File "/home/elastalert/.local/bin/elastalert", line 11, in <module>
    load_entry_point('elastalert==0.1.36', 'console_scripts', 'elastalert')()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2691, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2322, in load
    return self.resolve()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2328, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/elastalert/.local/lib/python2.7/site-packages/elastalert/elastalert.py", line 21, in <module>
    from alerts import DebugAlerter
  File "/home/elastalert/.local/lib/python2.7/site-packages/elastalert/alerts.py", line 31, in <module>
    from thehive4py.api import TheHiveApi
  File "/home/elastalert/.local/lib/python2.7/site-packages/thehive4py/api.py", line 7, in <module>
    import magic
  File "/home/elastalert/.local/lib/python2.7/site-packages/magic.py", line 181, in <module>
    raise ImportError('failed to find libmagic.  Check your installation')
ImportError: failed to find libmagic.  Check your installation

After manually installing libmagic the issue persists.

Has anyone else experienced this issue?

[synhershko]

Here too

[synhershko]

This worked for me:

pip uninstall python-magic
pip install python-magic-bin==0.4.14

[lsoumille]

@synhershko Your solution doesn't work for me because python-magic requirement is >=0.4.15. Here is the error message:

pkg_resources.DistributionNotFound: The 'python-magic>=0.4.15' distribution was not found and is required by elastalert

[primeroz]

I just installed libmagic and it fixed that error. I am running an alpine container

[eliasbrange]

+1. Our elastalert suddenly started failing in our CI due to libmagic not being installed. Might want to add it to the docs that it is required. Even better, don't try to import it unless a Hive rule is configured.

[wuxinwei]

+1. I got the same problem when I was run elastalert at alpine container

[wuxinwei]

@primeroz , Can you tell me that how do you solve this problem, just install libmagic when build docker image?

[eliasbrange]

@primeroz , Can you tell me that how do you solve this problem, just install libmagic when build docker image?

We've just added apk add libmagic.

[primeroz]

@wuxinwei as @eliasbrange mentioned just apk add libmagic

[wuxinwei]

@eliasbrange ,thank you, the solution which you mentioned solved my problem

[Shenmiu]

This worked for me:

pip uninstall python-magic
pip install python-magic-bin==0.4.14

[skatakar]

Observed the same issue while running elastalert
1)D:\elastalert>elastalert-test-rule "D:\elastalert\example_rules\example_frequenc
y.yaml"
Traceback (most recent call last):
File "D:\Python27\Scripts\elastalert-test-rule-script.py", line 11, in >
load_entry_point('elastalert==0.1.36', 'console_scripts', 'elastalert-test-r
ule')()
File "D:\Python27\Lib\site-packagespkg_resources__init__.py", line 480, in l
oad_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "D:\Python27\Lib\site-packagespkg_resources__init__.py", line 2693, in
load_entry_point
return ep.load()
File "D:\Python27\Lib\site-packagespkg_resources__init__.py", line 2324, in
load
return self.resolve()
File "D:\Python27\Lib\site-packagespkg_resources__init__.py", line 2330, in
resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "D:\Python27\Lib\site-packages\elastalert\test_rule.py", line 20, in ule>
import elastalert.config
File "D:\Python27\Lib\site-packages\elastalert\config.py", line 9, in

import alerts

File "D:\Python27\Lib\site-packages\elastalertalerts.py", line 31, in >
from thehive4py.api import TheHiveApi
File "D:\Python27\Lib\site-packages\thehive4pyapi.py", line 7, in
import magic
File "D:\Python27\Lib\site-packagesmagic.py", line 181, in
raise ImportError('failed to find libmagic. Check your installation')
ImportError: failed to find libmagic. Check your installation

D:\elastalert>

C:\Users\u3xlbes>elastalert --start NOW --verbose
Traceback (most recent call last):
File "D:\Python27\Scripts\elastalert-script.py", line 11, in
load_entry_point('elastalert==0.1.36', 'console_scripts', 'elastalert')()
File "D:\Python27\Lib\site-packagespkg_resources__init__.py", line 480, in l
oad_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "D:\Python27\Lib\site-packagespkg_resources__init__.py", line 2693, in
load_entry_point
return ep.load()
File "D:\Python27\Lib\site-packagespkg_resources__init__.py", line 2324, in
load
return self.resolve()
File "D:\Python27\Lib\site-packagespkg_resources__init__.py", line 2330, in
resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "D:\Python27\Lib\site-packages\elastalert\elastalert.py", line 21, in dule>
from alerts import DebugAlerter
File "D:\Python27\Lib\site-packages\elastalertalerts.py", line 31, in >
from thehive4py.api import TheHiveApi
File "D:\Python27\Lib\site-packages\thehive4pyapi.py", line 7, in
import magic
File "D:\Python27\Lib\site-packagesmagic.py", line 181, in
raise ImportError('failed to find libmagic. Check your installation')
ImportError: failed to find libmagic. Check your installation

C:\Users\u3xlbes>

we are using python 2.7.15 . any workaround for this issue?

[skatakar]

Do we have any workaround or solution for this issue?

[eliasbrange]

Do we have any workaround or solution for this issue?

Check out https://github.com/pidydx/libmagicwin64 for installing libmagic on windows.

[skatakar]

Do we have any workaround or solution for this issue?

Check out https://github.com/pidydx/libmagicwin64 for installing libmagic on windows.

I have downloaded all 4 files which are mentioned in the above link and placed under C:\WindowsSystem32.. But when I try to import magic or import liblogic, its giving same error.

import magic
Traceback (most recent call last):
File "", line 1, in
File "D:\Python27\Lib\site-packagesmagic.py", line 181, in
raise ImportError('failed to find libmagic. Check your installation')
ImportError: failed to find libmagic. Check your installation
import libmagic
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named libmagic

Do we need to run/configure any thing after download those 4 files?

we have python 32bit on windows 2012 64 bit

platform.architecture()
('32bit', 'WindowsPE')

[MickaelHUMPHREYS]

Hi,

I had the same problem.
I solved it by adding this requirement : python-magic
pip install python-magic==0.4.15

[Ogagnon]

This worked for me:

pip uninstall python-magic
pip install python-magic-bin==0.4.14

I love you.

[efinal]

This worked for me:
pip uninstall python-magic
pip install python-magic-bin==0.4.14

it works thank you

[andresjbotello]

This worked for me:

pip uninstall python-magic
pip install python-magic-bin==0.4.14

This has worked for me, thanks!!!

[yustiks]

I have the same issue (Linux mint), I tried to remove magic by this command:
pip uninstall python-magic
and then reinstall, but it didn't work.
What I did in the end: I cloned the github code: https://github.com/ahupp/python-magic
into the local folder to the directory where the actual project is, and then renamed the folder to 'magic'. After that it worked.

[lobinator]

This worked for me:
pip uninstall python-magic
pip install python-magic-bin==0.4.14

Worked for me too. Note that I uninstalled libmagic in addition to python-magic.

[roykingtree]

neither solution worked for me

[Lebostein]

What is this for a crap? I get this error on Windows and Mac OS.

[saginadir]

Solved it by running

brew install libmagic

[richardbalwane]

Use the method by @synhershko
i.e, 1. pip uninstall python-magic
2. pip install python-magic-bin
Except, don't include the python-magic-bin version!

[vvvlad]

Solved it by running

brew install libmagic

In my case this is what helped (MacOS obviously...)

[Bhairab123]

In my case, I had to install pip install python-magic-bin only. If I uninstall then I get following error:-
"pkg_resources.DistributionNotFound: The 'python-magic>=0.4.15' distribution was not found and is required by elastalert"

[shasan-marsdd]

This worked for me:

pip uninstall python-magic
pip install python-magic-bin==0.4.14

this issue happens on any python software running on mac. I don't fully understand why python-magic doesn't fix this or if this issue shouldn't be fixed because it's a mac only issue. It isn't an issue on any containers containing linux.

[kushan-gunasekera]

Installation

The current stable version of python-magic is available on PyPI and can be installed by running pip install python-magic.

Other sources:

PyPI: http://pypi.python.org/pypi/python-magic/
GitHub: https://github.com/ahupp/python-magic
This module is a simple wrapper around the libmagic C library, and that must be installed as well:

Debian/Ubuntu

$ sudo apt-get install libmagic1

Windows

You'll need DLLs for libmagic. @julian-r has uploaded a version of this project that includes binaries to PyPI: https://pypi.python.org/pypi/python-magic-bin/0.4.14

Other sources of the libraries in the past have been File for Windows . You will need to copy the file magic out of [binary-zip]\share\misc, and pass its location to Magic(magic_file=...).

If you are using a 64-bit build of python, you'll need 64-bit libmagic binaries which can be found here: https://github.com/pidydx/libmagicwin64. Newer version can be found here: https://github.com/nscaife/file-windows.

OSX

When using Homebrew: brew install libmagic
When using macports: port install file

Ref:- https://pypi.org/project/python-magic/

[Qmando]

thehive4py is gone now, I'll remove python-magic from the dependencies