Misp: [master] RHEL/CentOS 7.x/8.x Install Issues

[SteveClement]

Main install Guide for RHEL/CentOS

Permissions

Revise the permission mess and consolidate in 1 function:

https://misp.github.io/MISP/INSTALL.rhel7/#5-set-file-permissions
AND
https://misp.github.io/MISP/INSTALL.rhel7/#7-apache-configuration
Section: # Since SELinux is enabled, we need to allow httpd to write to certain directories

• [ ] Fixed

Diagnostics Page

php-cli

Cli
Issues reading PHP settings. This could be due to the test script not being readable.

• [x] Fixed

python lief

lief:… Please install lief, documentation here: https://github.com/lief-project/LIEF

• [x] Fixed

Fix: Manual compilation of lief
Fix: Manual compilation of pyLIEF and add it to the venv path, BUT, it now will work on the CLI but NOT when called via PHP.
Fix: permissions: sudo chcon -t httpd_sys_script_exec_t $PATH_TO_MISP/app/files/scripts/lief/build/api/python/lief.so

GnuPG

GnuPG installation and settings…FAIL: Failed to load GnuPG

This is most likely due to a bug in GnuPG 2.0.22
This is an unknown issue at this moment in time, more details eventually here: https://github.com/MISP/MISP/issues/4796

• [ ] Fixed

ToDo: check what CLI/Python scripts want to get launched withing PHP.

ZeroMQ

ZeroMQ settings…ZeroMQ script not running.

If the permissions are wrong on ps, it will not work, restore them: sudo /sbin/restorecon -v /usr/bin/ps

• [x] Fixed

ToDo: Works, some times, reproduce non-working states.

Workers

Can: stop/start individual workers
Cannot: restart all workers after initial install

• [x] Fixed

Potential fix: sudo chcon -t httpd_sys_script_exec_t $PATH_TO_MISP/app/Console/worker/start.sh
ToDo: Check why some workers are ok with this fix, others are not...

Emailing

Add and test:

setsebool -P httpd_can_sendmail on

• [ ] Fixed

Redis

2019-06-24 17:22:38 Error: [RedisException] Permission denied
Request URL: /servers/serverSettings/diagnostics
Stack Trace:
#0 /var/www/MISP/app/Lib/Tools/PubSubTool.php(35): Redis->connect('localhost', 6379)
#1 /var/www/MISP/app/Model/AppModel.php(1840): PubSubTool->initTool()
#2 /var/www/MISP/app/Model/AppModel.php(1831): AppModel->loadPubSubTool()
#3 /var/www/MISP/app/Model/Server.php(4251): AppModel->getPubSubTool()
#4 /var/www/MISP/app/Controller/ServersController.php(1009): Server->zmqDiagnostics(2)
#5 [internal function]: ServersController->serverSettings('diagnostics')
#6 /var/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(499): ReflectionMethod->invokeArgs(Object(ServersController), Array)
#7 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction(Object(CakeRequest))
#8 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke(Object(ServersController), Object(CakeRequest))
#9 /var/www/MISP/app/webroot/index.php(92): Dispatcher->dispatch(Object(CakeRequest), Object(CakeResponse))
#10 {main}

[saeeda12]

I am experiencing all of these issues currently - is there something I can test additionally? My workers are showing as not running in the GUI even after I start them again via CLI.

[iwitz]

Do you have an error when starting the workers? Are the processes healthy after you've started them ( ps -e | grep php$ ) ?
I'm using RHEL 7 and the workers are properly detected by MISP. I'm starting the workers with systemd though, systemctl status makes debbuging a bit easier

[saeeda12]

@iwitz Hello, this is my output after starting the workers again. But still in the UI, it says Issues prevent jobs from being processed. Please resolve them below. next to all the workers.

sudo -u apache /usr/bin/scl enable rh-php71 /var/www/MISP/app/Console/worker/start.sh
Stopping workers
Stopping 23031 ... Done
Stopping 23012 ... Done
Stopping 23049 ... Done
Stopping 22994 ... Done
Stopping the Scheduler Worker ... Done

Creating workers
Starting worker ... Done

Creating workers
Starting worker ... Done

Creating workers
Starting worker ... Done

Creating workers
Starting worker ... Done

Creating the scheduler workers
Starting scheduler worker ... Done

ps -e | grep php$
31814 pts/2    00:00:00 php
31832 pts/2    00:00:00 php
31851 pts/2    00:00:00 php
31869 pts/2    00:00:00 php
31888 pts/2    00:00:00 php

[saeeda12]

Running systemctl I found that the mariadb.service has failed. I enabled and started it again, but still seeing the same results.

[iwitz]

@SteveClement If I remember correctly in order for MISP to be able to read the PHP settings I added the following in /etc/opt/rh/rh-php71/php-fpm.d/www.conf :

env[PATH] = /opt/rh/rh-php71/root/usr/bin:/opt/rh/rh-php71/root/usr/sbin:/opt/rh/rh-python36/root/usr/bin
env[LD_LIBRARY_PATH] = /opt/rh/rh-php71/root/usr/lib64:/opt/rh/rh-python36/root/usr/lib64
env[MANPATH] = /opt/rh/rh-php71/root/usr/share/man:/opt/rh/rh-python36/root/usr/share/man
env[PKG_CONFIG_PATH] = /opt/rh/rh-python36/root/usr/lib64/pkgconfig
env[XDG_DATA_DIRS] = /opt/rh/rh-python36/root/usr/share

Then I suppose you'd have to restart php-fpm although I did this quite a while ago.
Does it work for you ?

@saeeda12 Did you follow the installation procedure to install mariadb ? Theoretically the service should be called rh-mariadb102-mariadb.service. If so :

• Is MariaDB properly installed and configured ? Can you connect to it using scl enable rh-mariadb102 mysql -u root -p ? Can you connect to it using the credentials you created for MISP ?
• Do you have any extra information as to why the mariadb service is failing in journalctl -xe or with systemctl status rh-mariadb102-mariadb.service ?

[SteveClement]

@iwitz thanks, I will try this on my test setup.

[saeeda12]

@iwitz I followed the exact instructions per https://github.com/MISP/MISP/blob/2.4/docs/xINSTALL.centos7.md but the service is just called mariadb.service.

Running sudo scl enable mariadb mysql -u root -p returned:
Unable to open /etc/scl/conf/mariadb - but when I navigated to that directory, only rh-php71 and rh-python36 is listed.

Running sudo systemctl status mariadb.service gives:

 sudo systemctl status mariadb.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2019-02-07 11:23:40 EST; 23h ago
  Process: 3824 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=0/SUCCESS)
  Process: 3823 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=1/FAILURE)
  Process: 3793 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
 Main PID: 3823 (code=exited, status=1/FAILURE)

Feb 07 11:23:39 misp systemd[1]: Starting MariaDB database server...
Feb 07 11:23:39 misp mariadb-prepare-db-dir[3793]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Feb 07 11:23:39 misp mariadb-prepare-db-dir[3793]: If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Feb 07 11:23:40 misp systemd[1]: Started MariaDB database server.
Feb 07 11:23:40 misp mysqld_safe[3823]: 190207 11:23:40 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Feb 07 11:23:40 misp mysqld_safe[3823]: 190207 11:23:40 mysqld_safe A mysqld process already exists
Feb 07 11:23:40 misp systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
Feb 07 11:23:40 misp systemd[1]: Unit mariadb.service entered failed state.
Feb 07 11:23:40 misp systemd[1]: mariadb.service failed.

[iwitz]

@saeeda12 Sorry, I thought it was the SCL mariadb package that was installed in the procedure but it isn't.

Can you make sure that there is no mysql process running then try to start mariadb again ? Seems to me like A mysqld process already exists is what is preventing mariadb from starting properly.

systemctl stop mariadb
ps -e | grep -i mysql
ps -e | grep -i mariadb

Then kill these processes ( assuming you do not have any data in your database yet ; otherwise back your data up first ) and try to systemctl start mariadb again.
If mariadb starts properly this way, run mysql_secure_installation then proceed to the rest of the installation procedure from the line # Additionally, it is probably a good idea to make the database server listen on localhost only.
If it fails, can you copy the output of /var/log/mariadb/mariadb.log here ?

[iwitz]

@SteveClement I tried to remove the /etc/opt/rh/rh-php71/php-fpm.d/www.conf modification I mentioned earlier to see what issues it fixes exactly ; it fixes the following :

• Cli Issues reading PHP settings. This could be due to the test script not being readable.
• PHP CLI Version (>7.0.16 recommended): Unknown (Issues determining version)
• PyMISP:… Not installed or version outdated.
• Current libraries status…Some of the libraries related to STIX are not installed. Make sure that all libraries listed below are correctly installed.

However it causes issue #4134 ; I'm working on it and will create a PR for what I come up with.

[saeeda12]

@iwitz Thanks! So mysqld and mysqld_safe were running, so I killed them (sudo kill -9 [pid]), then started mariadb again.

Then I ran sudo mysql_secure_installation and went through the steps one by one. Then went through the below steps:

# Additionally, it is probably a good idea to make the database server listen on localhost only
echo [mysqld] |sudo tee /etc/my.cnf.d/bind-address.cnf
echo bind-address=127.0.0.1 |sudo tee -a /etc/my.cnf.d/bind-address.cnf
sudo systemctl restart mariadb.service

Then I checked systemctl again and didn't see any failed processes/ services - mariadb was loaded, active and running. So I started the workers. But in the UI, it's still showing as not running for all of them.

[liedekef]

For php to detect php, python, redis and such: you just need to adjust your php.ini to account for the scl-paths. Put the following in /etc/opt/rh/rh-php71/php-fpm.d/www.conf:
env[PATH] =/opt/rh/rh-redis32/root/usr/bin:/opt/rh/rh-python36/root/usr/bin:/opt/rh/rh-php71/root/usr/bin:/usr/local/bin:/usr/bin:/bin
then restart: systemctl restart rh-php71-php-fpm

[saeeda12]

Thanks @liedekef - I made the changes, but I'm still seeing the original errors (can't read php CLI, etc.) in the Diagnostics page unfortunately.

@iwitz What were the changes you made to your www.conf file? My environment variables look like the below:

; Pass environment variables like LD_LIBRARY_PATH. All $VARIABLEs are taken from
; the current environment.
; Default Value: clean env
;env[HOSTNAME] = $HOSTNAME
;env[PATH] = /opt/rh/rh-redis32/root/usr/bin:/opt/rh/rh-python36/root/usr/bin:/opt/rh/rh-php71/root/usr/bin:/usr/local/bin:/usr/bin:/bin
;env[TMP] = /tmp
;env[TMPDIR] = /tmp
;env[TEMP] = /tmp

[liedekef]

What I see is that your PATH setting is commented out, that should not be of course ...
So remove the '";", restart your php-fpm and retry. If it still doesn't work then: do a su to the webuser, scl-enable php in a bash-shell and type php then. It might be you have php-permission issues (but my guess is the path-thing should solve it).

[saeeda12]

@liedekef Thank you! Doing that updated the Diagnostics page to now show that PHP CLI Version (>7.0.16 recommended): 7.1.8 (Up to date).

But under PHP Extensions for Cli it says redis:... Not loaded. I ran sudo -u apache /usr/bin/scl enable rh-php71 in a bash shell, but not sure if that's the full command I need to use?

[liedekef]

That means that it either can't load the redis extension, or it isn't loaded for the command-line version. I'm guessing the latter one, so check in /etc/opt/rh/rh-php71/php.d that there's something like 99-redis.ini (which should be a symlink to ../php-fpm.d/redis.ini , which in turn loads redis.so as an extension).

[saeeda12]

Hi @liedekef , apologies for the delayed response.

I do see 99-redis.ini -> ../php-fpm.d/redis.ini in the php.d directory. I fixed some permissions in the php-fpm.d directory and now Cli for redis is showing as OK.

[shanto12]

anyone experiencing these issues in ubuntu?

[SteveClement]

@shanto12 can you please be more specific?
There are many to chose from ;)

But in theory, no, if you use Ubuntu 18.04 and our install script. Everything works as expected.

[rohit725]

I followed the instructions as on installation guide but misp-module.service is not working.

[root@misp misp-modules]# systemctl status misp-modules
● misp-modules.service - MISP modules
Loaded: loaded (/etc/systemd/system/misp-modules.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2019-05-16 14:20:37 IST; 4s ago
Process: 6419 ExecStart=/var/www/MISP/venv/bin/misp-modules –l 127.0.0.1 –s (code=exited, status=203/EXEC)
Main PID: 6419 (code=exited, status=203/EXEC)

May 16 14:20:37 misp.local systemd[1]: misp-modules.service: main process exited, code=exited, status=203/EXEC
May 16 14:20:37 misp.local systemd[1]: Unit misp-modules.service entered failed state.
May 16 14:20:37 misp.local systemd[1]: misp-modules.service failed.

System Logs

May 16 14:26:38 misp systemd: Started MISP modules.
May 16 14:26:38 misp systemd: Failed at step EXEC spawning /var/www/MISP/venv/bin/misp-modules –l 127.0.0.1 –s: No such file or directory
May 16 14:26:38 misp systemd: misp-modules.service: main process exited, code=exited, status=203/EXEC
May 16 14:26:38 misp systemd: Unit misp-modules.service entered failed state.
May 16 14:26:38 misp systemd: misp-modules.service failed.
May 16 14:26:48 misp systemd: misp-modules.service holdoff time over, scheduling restart.
May 16 14:26:48 misp systemd: Stopped MISP modules.

I am working on CentOS Linux release 7.6.1810 (Core).
I don't know what's wrong with it as I am new to MISP.

[SteveClement]

The misp-modules issue is a 2 parter. I still need to debug it properly.

[duckasylum]

ZeroMQ settings…ZeroMQ script not running. - was in my case a SELinux issue. I made a module with

allow httpd_sys_script_t unconfined_t:dir ( getaddr search );
allow httpd_sys_script_t unconfined_t:file (open read);

Seems to work. I don't know SELinux enough to make a more elegant fix/workaround.

[cudeso]

Things I had to change to get the scheduler worker to work:
Add timezone to /etc/php.ini (was already set in /etc/opt/rh/rh-php72/php.d/timezone.ini)

[liangbingqin]

I followed the instructions as on installation guide but misp-module.service is not working.

[root@misp misp-modules]# systemctl status misp-modules
● misp-modules.service - MISP modules
Loaded: loaded (/etc/systemd/system/misp-modules.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2019-05-16 14:20:37 IST; 4s ago
Process: 6419 ExecStart=/var/www/MISP/venv/bin/misp-modules –l 127.0.0.1 –s (code=exited, status=203/EXEC)
Main PID: 6419 (code=exited, status=203/EXEC)
May 16 14:20:37 misp.local systemd[1]: misp-modules.service: main process exited, code=exited, status=203/EXEC
May 16 14:20:37 misp.local systemd[1]: Unit misp-modules.service entered failed state.
May 16 14:20:37 misp.local systemd[1]: misp-modules.service failed.

System Logs

May 16 14:26:38 misp systemd: Started MISP modules.
May 16 14:26:38 misp systemd: Failed at step EXEC spawning /var/www/MISP/venv/bin/misp-modules –l 127.0.0.1 –s: No such file or directory
May 16 14:26:38 misp systemd: misp-modules.service: main process exited, code=exited, status=203/EXEC
May 16 14:26:38 misp systemd: Unit misp-modules.service entered failed state.
May 16 14:26:38 misp systemd: misp-modules.service failed.
May 16 14:26:48 misp systemd: misp-modules.service holdoff time over, scheduling restart.
May 16 14:26:48 misp systemd: Stopped MISP modules.

I am working on CentOS Linux release 7.6.1810 (Core).
I don't know what's wrong with it as I am new to MISP.

I got the same issue, but when I edited my /etc/systemd/system/misp-modules.service , I removed the double quotes and the -s in ExecStart, and it works for me.

_ExecStart=/var/www/MISP/venv/bin/misp-modules -l 127.0.0.1_

[er587]

I had a problem during this install where "[Exception] Class Redis doesn't exists." error was showing up in the logs and feed jobs were failing. This was due to the permissions

-rw-r-----. 1 root root 19 May 4 19:06 99-gnupg.ini
-rw-r-----. 1 root root 19 May 4 19:05 99-redis.ini
-rw-r-----. 1 root root 20 May 4 19:06 99-ssdeep.ini

The Fix was to apply read permissions.
sudo chmod o+r /etc/opt/rh/rh-php72/php.d/99*

[toddejohnson]

I would suggest against chcon and instead suggest semanage fcontext -a as a relabel of the filesystem will cause these settings to be lost which can be triggered in multiple ways. Also is there a way we could package some of these dependent libraries in a repo to simplify things? RHEL8 and AppStreams should make this easier to accomplish.