INSTALLATION INSTRUCTIONS¶
for Ubuntu 18.04.3-server¶
-1/ Installer and Manual install instructions¶
Make sure you are reading the parsed version of this Document. When in doubt click here.
To install MISP on a fresh Ubuntu 18.04, all you need to do is the following:
# Please check the installer options first to make the best choice for your install wget -O /tmp/INSTALL.sh https://raw.githubusercontent.com/MISP/MISP/2.4/INSTALL/INSTALL.sh bash /tmp/INSTALL.sh # This will install MISP Core wget -O /tmp/INSTALL.sh https://raw.githubusercontent.com/MISP/MISP/2.4/INSTALL/INSTALL.sh bash /tmp/INSTALL.sh -c
Notice
Installer tested working by @SteveClement on 20190513 (works with Ubuntu 18.10/19.04 too)
Notice
This document also serves as a source for the INSTALL-misp.sh script. Which explains why you will see the use of shell functions in various steps. Henceforth the document will also follow a more logical flow. In the sense that all the dependencies are installed first then config files are generated, etc...
Notice
If the next line is [!generic/core.md!]() click here.
Notice
Maintained and tested by the MISP core team.
Enjoy installing MISP. For any issues see here
1/ Minimal Ubuntu install¶
Install a minimal Ubuntu 18.04-server system with the software:¶
- OpenSSH server
- This guide assumes a user name of 'misp' with sudo working
Make sure your system is up2date¶
# <snippet-begin 0_apt-upgrade.sh> aptUpgrade () { debug "Upgrading system" checkAptLock sudo apt-get update # If we run in non-interactive mode, make sure we do not stop all of a sudden if [[ "${PACKER}" == "1" || "${UNATTENDED}" == "1" ]]; then export DEBIAN_FRONTEND=noninteractive export DEBIAN_PRIORITY=critical sudo -E apt-get -qy -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" upgrade sudo -E apt-get -qy autoclean else sudo apt-get upgrade -qy fi } # <snippet-end 0_apt-upgrade.sh>
install etckeeper and sudo (optional)¶
# <snippet-begin 0_sudoKeeper.sh> # check if sudo is installed checkSudoKeeper () { echo "Checking for sudo and installing etckeeper" if [[ ! -f $(which sudo) ]]; then echo "Please enter your root password below to install etckeeper" su -c "apt install etckeeper -y" echo "Please enter your root password below to install sudo" su -c "apt install sudo -y" echo "Please enter your root password below to install sudo" su -c "apt install curl -y" echo "Please enter your root password below to add ${MISP_USER} to sudo group" su -c "/usr/sbin/adduser ${MISP_USER} sudo" echo "We added ${MISP_USER} to group sudo and now we need to log out and in again." exit else sudo apt update sudo apt install etckeeper -y fi } # <snippet-end 0_sudoKeeper.sh>
add the misp user to staff and www-data (mandatory)¶
# <snippet-begin add-user.sh> ## FIXME: This function is a duplicate included in: # <snippet-begin 0_support-functions.sh> # check is /usr/local/src is RW by misp user checkUsrLocalSrc () { echo "" if [[ -e /usr/local/src ]]; then WRITEABLE=$(sudo -H -u $MISP_USER touch /usr/local/src 2> /dev/null ; echo $?) if [[ "$WRITEABLE" == "0" ]]; then echo "Good, /usr/local/src exists and is writeable as $MISP_USER" else # TODO: The below might be shorter, more elegant and more modern #[[ -n $KALI ]] || [[ -n $UNATTENDED ]] && echo "Just do it" sudo chmod 2775 /usr/local/src sudo chown root:staff /usr/local/src fi else echo "/usr/local/src does not exist, creating." mkdir -p /usr/local/src sudo chmod 2775 /usr/local/src # TODO: Better handling /usr/local/src permissions if [[ "$(cat /etc/group |grep staff > /dev/null 2>&1)" == "0" ]]; then sudo chown root:staff /usr/local/src fi fi } # <snippet-end add-user.sh>
Network Interface Name salvage (optional)¶
This will bring back 'ethX' e.g: eth0
# <snippet-end interfaces.sh> GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0" DEFAULT_GRUB=/etc/default/grub echo "--- Using old style name (ethX) for interfaces" #for key in GRUB_CMDLINE_LINUX #do # sudo sed -i "s/^\($key\)=.*/\1=\"$(eval echo \${$key})\"/" $DEFAULT_GRUB #done sed -r 's/^(GRUB_CMDLINE_LINUX=).*/\1\"net\.ifnames=0\ biosdevname=0\"/' /etc/default/grub | sudo tee /etc/default/grub > /dev/null # install ifupdown since ubuntu 18.04 sudo apt-get update sudo apt-get install -y ifupdown # enable eth0 echo "--- Configuring eth0" echo "# The primary network interface auto eth0 iface eth0 inet dhcp" | sudo tee /etc/network/interfaces sudo grub-mkconfig -o /boot/grub/grub.cfg sudo update-grub > /dev/null 2>&1 # <snippet-end interfaces.sh>
Notice
On recent Ubuntu install Netplan is default and you need to change the Network name.
sudo sed -i "s/enp0s3/eth0/" /etc/netplan/50-cloud-init.yaml
sudo sed -i "s/enp0s3/eth0/" /etc/netplan/01-netcfg.yaml
install postfix, there will be some questions.¶
# <snippet-begin postfix.sh> sudo apt-get install postfix dialog -qy # <snippet-end postfix.sh>
Notice
Postfix Configuration: Satellite system
change the relay server later with:
sudo postconf -e 'relayhost = example.com'
sudo postfix reload
MISP configuration variables¶
# <snippet-begin 0_global-vars.sh> # $ eval "$(curl -fsSL https://raw.githubusercontent.com/MISP/MISP/2.4/docs/generic/globalVariables.md | grep -v \`\`\`)" # $ MISPvars MISPvars () { debug "Setting generic ${LBLUE}MISP${NC} variables shared by all flavours" 2> /dev/null # Local non-root MISP user MISP_USER='misp' MISP_PASSWORD="$(openssl rand -hex 32)" # MISP configuration variables PATH_TO_MISP='/var/www/MISP' # The web server user # RHEL/CentOS if [[ -f "/etc/redhat-release" ]]; then WWW_USER='apache' # Debian flavoured elif [[ -f "/etc/debian_version" ]]; then WWW_USER="www-data" # OpenBSD elif [[ "$(uname -s)" == "OpenBSD" ]]; then WWW_USER="www" PATH_TO_MISP="/var/www/htdocs/MISP" # NetBSD elif [[ "$(uname -s)" == "NetBSD" ]]; then WWW_USER="www" PATH_TO_MISP="/usr/pkg/share/httpd/htdocs/MISP" else # I am feeling lucky WWW_USER="www-data" fi if [ -z "$FQDN" ]; then FQDN="misp.local" fi if [ -z "$MISP_BASEURL" ]; then MISP_BASEURL='""' fi MISP_LIVE='1' # Database configuration DBHOST='localhost' DBNAME='misp' DBUSER_ADMIN='root' DBPASSWORD_ADMIN="$(openssl rand -hex 32)" DBUSER_MISP='misp' DBPASSWORD_MISP="$(openssl rand -hex 32)" # OpenSSL configuration OPENSSL_CN=$FQDN OPENSSL_C='LU' OPENSSL_ST='State' OPENSSL_L='Location' OPENSSL_O='Organization' OPENSSL_OU='Organizational Unit' OPENSSL_EMAILADDRESS="info@$FQDN" # GPG configuration GPG_REAL_NAME='Autogenerated Key' # On a REAL install, please do not set a comment, see here for why: https://www.debian-administration.org/users/dkg/weblog/97 GPG_COMMENT='WARNING: MISP AutoGenerated Key consider this Key VOID!' GPG_EMAIL_ADDRESS='admin@admin.test' # 3072 bits used as per suggestions here: https://riseup.net/en/security/message-security/openpgp/best-practices GPG_KEY_LENGTH='3072' GPG_PASSPHRASE="$(openssl rand -hex 32)" # debug alias to make sure people are not confused when blindly copy pasting blobs of code alias debug="echo -e" # checkAptLock alias to make sure people are not confused when blindly copy pasting blobs of code alias checkAptLock="echo 'Function used in Installer to make sure apt is not locked'" # php.ini configuration upload_max_filesize=50M post_max_size=50M max_execution_time=300 memory_limit=2048M CAKE="$PATH_TO_MISP/app/Console/cake" # sudo config to run $LUSER commands if [[ "$(groups ${MISP_USER} |grep -o 'staff')" == "staff" ]]; then SUDO_USER="sudo -H -u ${MISP_USER} -g staff" else SUDO_USER="sudo -H -u ${MISP_USER}" fi SUDO_WWW="sudo -H -u ${WWW_USER} " echo "The following DB Passwords were generated..." echo "Admin (${DBUSER_ADMIN}) DB Password: ${DBPASSWORD_ADMIN}" echo "User (${DBUSER_MISP}) DB Password: ${DBPASSWORD_MISP}" } # <snippet-end 0_global-vars.sh>
2/ Install LAMP & dependencies¶
Once the system is installed you can perform the following steps.
# <snippet-begin 0_installCoreDeps.sh> installCoreDeps () { debug "Installing core dependencies" # Install the dependencies: (some might already be installed) sudo apt-get install curl gcc git gpg-agent make python python3 openssl redis-server sudo vim zip unzip virtualenv libfuzzy-dev sqlite3 moreutils -qy # Install MariaDB (a MySQL fork/alternative) sudo apt-get install mariadb-client mariadb-server -qy # Install Apache2 sudo apt-get install apache2 apache2-doc apache2-utils -qy # install Mitre's STIX and its dependencies by running the following commands: sudo apt-get install python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev python-setuptools -qy sudo apt install expect -qy } # <snippet-end 0_installCoreDeps.sh> # <snippet-begin 0_installDepsPhp72.sh> # Install Php 7.2 dependencies installDepsPhp72 () { debug "Installing PHP 7.2 dependencies" PHP_ETC_BASE=/etc/php/7.2 PHP_INI=${PHP_ETC_BASE}/apache2/php.ini sudo apt update sudo apt install -qy \ libapache2-mod-php \ php php-cli \ php-dev \ php-json php-xml php-mysql php7.2-opcache php-readline php-mbstring \ php-redis php-gnupg \ php-gd for key in upload_max_filesize post_max_size max_execution_time max_input_time memory_limit do sudo sed -i "s/^\($key\).*/\1 = $(eval echo \${$key})/" $PHP_INI done } # <snippet-end 0_installDepsPhp72.sh>
3/ MISP code¶
# <snippet-begin 1_mispCoreInstall.sh> installCore () { debug "Installing ${LBLUE}MISP${NC} core" # Download MISP using git in the /var/www/ directory. sudo mkdir ${PATH_TO_MISP} sudo chown $WWW_USER:$WWW_USER ${PATH_TO_MISP} cd ${PATH_TO_MISP} $SUDO_WWW git clone https://github.com/MISP/MISP.git ${PATH_TO_MISP} $SUDO_WWW git submodule update --init --recursive # Make git ignore filesystem permission differences for submodules $SUDO_WWW git submodule foreach --recursive git config core.filemode false # Make git ignore filesystem permission differences $SUDO_WWW git config core.filemode false # Create a python3 virtualenv $SUDO_WWW virtualenv -p python3 ${PATH_TO_MISP}/venv # make pip happy sudo mkdir /var/www/.cache/ sudo chown $WWW_USER:$WWW_USER /var/www/.cache cd ${PATH_TO_MISP}/app/files/scripts $SUDO_WWW git clone https://github.com/CybOXProject/python-cybox.git $SUDO_WWW git clone https://github.com/STIXProject/python-stix.git $SUDO_WWW git clone https://github.com/MAECProject/python-maec.git # install mixbox to accommodate the new STIX dependencies: $SUDO_WWW git clone https://github.com/CybOXProject/mixbox.git cd ${PATH_TO_MISP}/app/files/scripts/mixbox $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install . cd ${PATH_TO_MISP}/app/files/scripts/python-cybox $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install . cd ${PATH_TO_MISP}/app/files/scripts/python-stix $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install . cd $PATH_TO_MISP/app/files/scripts/python-maec $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install . # install STIX2.0 library to support STIX 2.0 export: cd ${PATH_TO_MISP}/cti-python-stix2 $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install . # install PyMISP cd ${PATH_TO_MISP}/PyMISP $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install . # install pydeep $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install git+https://github.com/kbandla/pydeep.git # install lief $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install https://github.com/lief-project/packages/raw/lief-master-latest/pylief-0.9.0.dev.zip # install zmq needed by mispzmq $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install zmq redis # install python-magic $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install python-magic # install plyara $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install plyara } # <snippet-end 1_mispCoreInstall.sh>
4/ CakePHP¶
# <snippet-begin 1_installCake.sh> installCake () { debug "Installing CakePHP" # Once done, install CakeResque along with its dependencies # if you intend to use the built in background jobs: cd ${PATH_TO_MISP}/app # Make composer cache happy # /!\ composer on Ubuntu when invoked with sudo -u doesn't set $HOME to /var/www but keeps it /home/misp \!/ sudo mkdir /var/www/.composer ; sudo chown $WWW_USER:$WWW_USER /var/www/.composer $SUDO_WWW php composer.phar install # Enable CakeResque with php-redis sudo phpenmod redis sudo phpenmod gnupg # To use the scheduler worker for scheduled tasks, do the following: $SUDO_WWW cp -fa ${PATH_TO_MISP}/INSTALL/setup/config.php ${PATH_TO_MISP}/app/Plugin/CakeResque/Config/config.php # If you have multiple MISP instances on the same system, don't forget to have a different Redis per MISP instance for the CakeResque workers # The default Redis port can be updated in Plugin/CakeResque/Config/config.php } # <snippet-end 1_installCake.sh>
5/ Set the permissions¶
# <snippet-begin 2_permissions.sh> # Main function to fix permissions to something sane permissions () { debug "Setting permissions" sudo chown -R ${WWW_USER}:${WWW_USER} ${PATH_TO_MISP} sudo chmod -R 750 ${PATH_TO_MISP} sudo chmod -R g+ws ${PATH_TO_MISP}/app/tmp sudo chmod -R g+ws ${PATH_TO_MISP}/app/files sudo chmod -R g+ws $PATH_TO_MISP/app/files/scripts/tmp } # <snippet-end 2_permissions.sh>
6/ Create a database and user¶
Set-up DB, User and import empty MISP DB¶
# <snippet-begin 1_prepareDB.sh> prepareDB () { if [[ ! -e /var/lib/mysql/misp/users.ibd ]]; then debug "Setting up database" # FIXME: If user 'misp' exists, and has a different password, the below WILL fail. # Add your credentials if needed, if sudo has NOPASS, comment out the relevant lines if [[ "${PACKER}" == "1" ]]; then pw="Password1234" else pw=${MISP_PASSWORD} fi expect -f - <<-EOF set timeout 10 spawn sudo -k mysql_secure_installation expect "*?assword*" send -- "${pw}\r" expect "Enter current password for root (enter for none):" send -- "\r" expect "Set root password?" send -- "y\r" expect "New password:" send -- "${DBPASSWORD_ADMIN}\r" expect "Re-enter new password:" send -- "${DBPASSWORD_ADMIN}\r" expect "Remove anonymous users?" send -- "y\r" expect "Disallow root login remotely?" send -- "y\r" expect "Remove test database and access to it?" send -- "y\r" expect "Reload privilege tables now?" send -- "y\r" expect eof EOF sudo apt-get purge -y expect ; sudo apt autoremove -qy fi sudo mysql -u ${DBUSER_ADMIN} -p${DBPASSWORD_ADMIN} -e "CREATE DATABASE ${DBNAME};" sudo mysql -u ${DBUSER_ADMIN} -p${DBPASSWORD_ADMIN} -e "CREATE USER '${DBUSER_MISP}'@'localhost' IDENTIFIED BY '${DBPASSWORD_MISP}';" sudo mysql -u ${DBUSER_ADMIN} -p${DBPASSWORD_ADMIN} -e "GRANT USAGE ON *.* to ${DBUSER_MISP}@localhost;" sudo mysql -u ${DBUSER_ADMIN} -p${DBPASSWORD_ADMIN} -e "GRANT ALL PRIVILEGES on ${DBNAME}.* to '${DBUSER_MISP}'@'localhost';" sudo mysql -u ${DBUSER_ADMIN} -p${DBPASSWORD_ADMIN} -e "FLUSH PRIVILEGES;" # Import the empty MISP database from MYSQL.sql ${SUDO_WWW} cat ${PATH_TO_MISP}/INSTALL/MYSQL.sql | mysql -u ${DBUSER_MISP} -p${DBPASSWORD_MISP} ${DBNAME} } # <snippet-end 1_prepareDB.sh>
7/ Apache configuration¶
Now configure your Apache webserver with the DocumentRoot ${PATH_TO_MISP}/app/webroot/
Apache version 2.4 config:¶
Notice
Be aware that the configuration files for apache 2.4 and up have changed. The configuration file has to have the .conf extension in the sites-available directory For more information, visit http://httpd.apache.org/docs/2.4/upgrading.html
# <snippet-begin 1_apacheConfig.sh> apacheConfig () { debug "Generating Apache config" sudo cp ${PATH_TO_MISP}/INSTALL/apache.24.misp.ssl /etc/apache2/sites-available/misp-ssl.conf if [[ ! -z ${MISP_BASEURL} ]] && [[ "$(echo $MISP_BASEURL|cut -f 1 -d :)" == "http" || "$(echo $MISP_BASEURL|cut -f 1 -d :)" == "https" ]]; then echo "Potentially replacing misp.local with $MISP_BASEURL in misp-ssl.conf" fi # If a valid SSL certificate is not already created for the server, # create a self-signed certificate: sudo openssl req -newkey rsa:4096 -days 365 -nodes -x509 \ -subj "/C=${OPENSSL_C}/ST=${OPENSSL_ST}/L=${OPENSSL_L}/O=${OPENSSL_O}/OU=${OPENSSL_OU}/CN=${OPENSSL_CN}/emailAddress=${OPENSSL_EMAILADDRESS}" \ -keyout /etc/ssl/private/misp.local.key -out /etc/ssl/private/misp.local.crt # Enable modules, settings, and default of SSL in Apache sudo a2dismod status sudo a2enmod ssl sudo a2enmod rewrite sudo a2enmod headers sudo a2dissite 000-default sudo a2ensite default-ssl # Apply all changes sudo systemctl restart apache2 # activate new vhost sudo a2dissite default-ssl sudo a2ensite misp-ssl # Restart apache sudo systemctl restart apache2 } # <snippet-end 1_apacheConfig.sh>
Notice
Please find a sample conf file for an SSL enabled conf file in-line below (alternatively use one of the samples provided in /var/www/MISP/INSTALL).
Also remember to verify the SSLCertificateChainFile property in your config file.
This is usually commented out for the self-generated certificate in the sample configurations, such as the one pasted below.
Otherwise, copy the SSLCertificateFile, SSLCertificateKeyFile, and SSLCertificateChainFile to /etc/ssl/private/. (Modify path and config to fit your environment)
============================================= Begin sample working SSL config for MISP
<VirtualHost <IP, FQDN, or *>:80>
ServerName <your.FQDN.here>
Redirect permanent / https://<your.FQDN.here>
LogLevel warn
ErrorLog /var/log/apache2/misp.local_error.log
CustomLog /var/log/apache2/misp.local_access.log combined
ServerSignature Off
</VirtualHost>
<VirtualHost <IP, FQDN, or *>:443>
ServerAdmin admin@<your.FQDN.here>
ServerName <your.FQDN.here>
DocumentRoot /var/www/MISP/app/webroot
<Directory /var/www/MISP/app/webroot>
Options -Indexes
AllowOverride all
Order allow,deny
allow from all
</Directory>
SSLEngine On
SSLCertificateFile /etc/ssl/private/misp.local.crt
SSLCertificateKeyFile /etc/ssl/private/misp.local.key
# SSLCertificateChainFile /etc/ssl/private/misp-chain.crt
LogLevel warn
ErrorLog /var/log/apache2/misp.local_error.log
CustomLog /var/log/apache2/misp.local_access.log combined
ServerSignature Off
</VirtualHost>
============================================= End sample working SSL config for MISP
8/ Log rotation¶
# <snippet-begin 2_logRotation.sh> logRotation () { # MISP saves the stdout and stderr of its workers in ${PATH_TO_MISP}/app/tmp/logs # To rotate these logs install the supplied logrotate script: sudo cp ${PATH_TO_MISP}/INSTALL/misp.logrotate /etc/logrotate.d/misp sudo chmod 0640 /etc/logrotate.d/misp } # <snippet-end 2_logRotation.sh>
9/ MISP configuration¶
# <snippet-begin 2_configMISP.sh> configMISP () { debug "Generating ${LBLUE}MISP${NC} config files" # There are 4 sample configuration files in ${PATH_TO_MISP}/app/Config that need to be copied $SUDO_WWW cp -a ${PATH_TO_MISP}/app/Config/bootstrap.default.php ${PATH_TO_MISP}/app/Config/bootstrap.php $SUDO_WWW cp -a ${PATH_TO_MISP}/app/Config/database.default.php ${PATH_TO_MISP}/app/Config/database.php $SUDO_WWW cp -a ${PATH_TO_MISP}/app/Config/core.default.php ${PATH_TO_MISP}/app/Config/core.php $SUDO_WWW cp -a ${PATH_TO_MISP}/app/Config/config.default.php ${PATH_TO_MISP}/app/Config/config.php echo "<?php class DATABASE_CONFIG { public \$default = array( 'datasource' => 'Database/Mysql', //'datasource' => 'Database/Postgres', 'persistent' => false, 'host' => '$DBHOST', 'login' => '$DBUSER_MISP', 'port' => 3306, // MySQL & MariaDB //'port' => 5432, // PostgreSQL 'password' => '$DBPASSWORD_MISP', 'database' => '$DBNAME', 'prefix' => '', 'encoding' => 'utf8', ); }" | $SUDO_WWW tee $PATH_TO_MISP/app/Config/database.php # Important! Change the salt key in ${PATH_TO_MISP}/app/Config/config.php # The salt key must be a string at least 32 bytes long. # The admin user account will be generated on the first login, make sure that the salt is changed before you create that user # If you forget to do this step, and you are still dealing with a fresh installation, just alter the salt, # delete the user from mysql and log in again using the default admin credentials (admin@admin.test / admin) # and make sure the file permissions are still OK sudo chown -R $WWW_USER:$WWW_USER ${PATH_TO_MISP}/app/Config sudo chmod -R 750 ${PATH_TO_MISP}/app/Config } # <snippet-end 2_configMISP.sh>
# <snippet-begin 2_gnupg.sh> # Generate GnuPG key setupGnuPG () { if [ ! -d $PATH_TO_MISP/.gnupg ]; then # The email address should match the one set in the config.php # set in the configuration menu in the administration menu configuration file echo "%echo Generating a default key Key-Type: default Key-Length: $GPG_KEY_LENGTH Subkey-Type: default Name-Real: $GPG_REAL_NAME Name-Comment: $GPG_COMMENT Name-Email: $GPG_EMAIL_ADDRESS Expire-Date: 0 Passphrase: $GPG_PASSPHRASE # Do a commit here, so that we can later print "done" %commit %echo done" > /tmp/gen-key-script $SUDO_WWW gpg --homedir $PATH_TO_MISP/.gnupg --batch --gen-key /tmp/gen-key-script # Export the public key to the webroot $SUDO_WWW sh -c "gpg --homedir $PATH_TO_MISP/.gnupg --export --armor $GPG_EMAIL_ADDRESS" | $SUDO_WWW tee $PATH_TO_MISP/app/webroot/gpg.asc fi } # <snippet-end 2_gnupg.sh>
Notice
If entropy is not high enough, you can install havegd and then start the service
sudo apt install haveged -qy sudo service haveged start
# <snippet-begin 2_backgroundWorkers.sh> backgroundWorkers () { debug "Setting up background workers" # To make the background workers start on boot sudo chmod +x $PATH_TO_MISP/app/Console/worker/start.sh if [ ! -e /etc/rc.local ] then echo '#!/bin/sh -e' | sudo tee -a /etc/rc.local echo 'exit 0' | sudo tee -a /etc/rc.local sudo chmod u+x /etc/rc.local fi echo "[Unit] Description=MISP background workers After=network.target [Service] Type=forking User=${WWW_USER} Group=${WWW_USER} ExecStart=${PATH_TO_MISP}/app/Console/worker/start.sh Restart=always RestartSec=10 [Install] WantedBy=multi-user.target" | sudo tee /etc/systemd/system/misp-workers.service sudo systemctl daemon-reload sudo systemctl enable --now misp-workers # Add the following lines before the last line (exit 0). Make sure that you replace www-data with your apache user: sudo sed -i -e '$i \echo never > /sys/kernel/mm/transparent_hugepage/enabled\n' /etc/rc.local sudo sed -i -e '$i \echo 1024 > /proc/sys/net/core/somaxconn\n' /etc/rc.local sudo sed -i -e '$i \sysctl vm.overcommit_memory=1\n' /etc/rc.local } # <snippet-end 2_backgroundWorkers.sh>
echo "Admin (root) DB Password: $DBPASSWORD_ADMIN" echo "User (misp) DB Password: $DBPASSWORD_MISP"
Initialize MISP configuration and set some defaults¶
# <snippet-begin 2_core-cake.sh> # Core cake commands to tweak MISP and aleviate some of the configuration pains # The $RUN_PHP is ONLY set on RHEL/CentOS installs and can thus be ignored # This file is NOT an excuse to NOT read the settings and familiarize ourselves with them ;) coreCAKE () { debug "Running core Cake commands to set sane defaults for ${LBLUE}MISP${NC}" # IF you have logged in prior to running this, it will fail but the fail is NON-blocking $SUDO_WWW $RUN_PHP -- $CAKE userInit -q # This makes sure all Database upgrades are done, without logging in. $SUDO_WWW $RUN_PHP -- $CAKE Admin runUpdates # The default install is Python >=3.6 in a virtualenv, setting accordingly $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.python_bin" "${PATH_TO_MISP}/venv/bin/python" # Set default role # TESTME: The following seem defunct, please test. # $SUDO_WWW $RUN_PHP -- $CAKE setDefaultRole 3 # Tune global time outs $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Session.autoRegenerate" 0 $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Session.timeout" 600 $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Session.cookieTimeout" 3600 # Change base url, either with this CLI command or in the UI $SUDO_WWW $RUN_PHP -- $CAKE Baseurl $MISP_BASEURL # example: 'baseurl' => 'https://<your.FQDN.here>', # alternatively, you can leave this field empty if you would like to use relative pathing in MISP # 'baseurl' => '', # The base url of the application (in the format https://www.mymispinstance.com) as visible externally/by other MISPs. # MISP will encode this URL in sharing groups when including itself. If this value is not set, the baseurl is used as a fallback. $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.external_baseurl" $MISP_BASEURL # Enable GnuPG $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "GnuPG.email" "$GPG_EMAIL_ADDRESS" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "GnuPG.homedir" "$PATH_TO_MISP/.gnupg" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "GnuPG.password" "$GPG_PASSPHRASE" # FIXME: what if we have not gpg binary but a gpg2 one? $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "GnuPG.binary" "$(which gpg)" # Enable installer org and tune some configurables $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.host_org_id" 1 $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.email" "info@admin.test" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.disable_emailing" true $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.contact" "info@admin.test" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.disablerestalert" true $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.showCorrelationsOnIndex" true $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.default_event_tag_collection" 0 # Provisional Cortex tunes $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.Cortex_services_enable" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.Cortex_services_url" "http://127.0.0.1" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.Cortex_services_port" 9000 $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.Cortex_timeout" 120 $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.Cortex_authkey" "" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.Cortex_ssl_verify_peer" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.Cortex_ssl_verify_host" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.Cortex_ssl_allow_self_signed" true # Various plugin sightings settings $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.Sightings_policy" 0 $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.Sightings_anonymise" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.Sightings_range" 365 # Plugin CustomAuth tuneable $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.CustomAuth_disable_logout" false # RPZ Plugin settings $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.RPZ_policy" "DROP" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.RPZ_walled_garden" "127.0.0.1" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.RPZ_serial" "\$date00" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.RPZ_refresh" "2h" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.RPZ_retry" "30m" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.RPZ_expiry" "30d" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.RPZ_minimum_ttl" "1h" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.RPZ_ttl" "1w" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.RPZ_ns" "localhost." $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.RPZ_ns_alt" "" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Plugin.RPZ_email" "root.localhost" # Force defaults to make MISP Server Settings less RED $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.language" "eng" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.proposals_block_attributes" false # Redis block $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.redis_host" "127.0.0.1" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.redis_port" 6379 $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.redis_database" 13 $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.redis_password" "" # Force defaults to make MISP Server Settings less YELLOW $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.ssdeep_correlation_threshold" 40 $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.extended_alert_subject" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.default_event_threat_level" 4 $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.newUserText" "Dear new MISP user,\\n\\nWe would hereby like to welcome you to the \$org MISP community.\\n\\n Use the credentials below to log into MISP at \$misp, where you will be prompted to manually change your password to something of your own choice.\\n\\nUsername: \$username\\nPassword: \$password\\n\\nIf you have any questions, don't hesitate to contact us at: \$contact.\\n\\nBest regards,\\nYour \$org MISP support team" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.passwordResetText" "Dear MISP user,\\n\\nA password reset has been triggered for your account. Use the below provided temporary password to log into MISP at \$misp, where you will be prompted to manually change your password to something of your own choice.\\n\\nUsername: \$username\\nYour temporary password: \$password\\n\\nIf you have any questions, don't hesitate to contact us at: \$contact.\\n\\nBest regards,\\nYour \$org MISP support team" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.enableEventBlacklisting" true $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.enableOrgBlacklisting" true $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.log_client_ip" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.log_auth" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.disableUserSelfManagement" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.block_event_alert" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.block_event_alert_tag" "no-alerts=\"true\"" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.block_old_event_alert" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.block_old_event_alert_age" "" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.incoming_tags_disabled_by_default" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.maintenance_message" "Great things are happening! MISP is undergoing maintenance, but will return shortly. You can contact the administration at \$email." $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.footermidleft" "This is an initial install" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.footermidright" "Please configure and harden accordingly" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.welcome_text_top" "Initial Install, please configure" # TODO: Make sure $FLAVOUR is correct $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.welcome_text_bottom" "Welcome to MISP on $FLAVOUR, change this message in MISP Settings" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.attachments_dir" "$PATH_TO_MISP/app/files" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.download_attachments_on_load" true $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.title_text" "MISP" $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.terms_download" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.showorgalternate" false $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "MISP.event_view_filter_fields" "id, uuid, value, comment, type, category, Tag.name" # Force defaults to make MISP Server Settings less GREEN $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Security.password_policy_length" 12 $SUDO_WWW $RUN_PHP -- $CAKE Admin setSetting "Security.password_policy_complexity" '/^((?=.*\d)|(?=.*\W+))(?![\n])(?=.*[A-Z])(?=.*[a-z]).*$|.{16,}/' # Set MISP Live $SUDO_WWW $RUN_PHP -- $CAKE Live $MISP_LIVE } # This updates Galaxies, ObjectTemplates, Warninglists, Noticelists, Templates updateGOWNT () { # AUTH_KEY Place holder in case we need to **curl** somehing in the future # $SUDO_WWW $RUN_MYSQL -- mysql -u $DBUSER_MISP -p$DBPASSWORD_MISP misp -e "SELECT authkey FROM users;" | tail -1 > /tmp/auth.key AUTH_KEY=$(cat /tmp/auth.key) rm /tmp/auth.key debug "Updating Galaxies, ObjectTemplates, Warninglists, Noticelists and Templates" # Update the galaxies… # TODO: Fix updateGalaxies $SUDO_WWW $RUN_PHP -- $CAKE Admin updateGalaxies # Updating the taxonomies… $SUDO_WWW $RUN_PHP -- $CAKE Admin updateTaxonomies # Updating the warning lists… $SUDO_WWW $RUN_PHP -- $CAKE Admin updateWarningLists # Updating the notice lists… $SUDO_WWW $RUN_PHP -- $CAKE Admin updateNoticeLists # Updating the object templates… $SUDO_WWW $RUN_PHP -- $CAKE Admin updateObjectTemplates "1337" } # <snippet-end 2_core-cake.sh>
Install misp-modules (optional)¶
# <snippet-begin 3_misp-modules.sh> # Main MISP Modules install function mispmodules () { cd /usr/local/src/ ## TODO: checkUsrLocalSrc in main doc debug "Cloning misp-modules" $SUDO_USER git clone https://github.com/MISP/misp-modules.git cd misp-modules # some misp-modules dependencies sudo apt install libpq5 libjpeg-dev tesseract-ocr libpoppler-cpp-dev imagemagick libopencv-dev zbar-tools libzbar0 libzbar-dev libfuzzy-dev -y # If you build an egg, the user you build it as need write permissions in the CWD sudo chgrp $WWW_USER . sudo chmod g+w . $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install -I -r REQUIREMENTS sudo chgrp staff . $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install -I . ## sudo gem install asciidoctor-pdf --pre # Start misp-modules as a service sudo cp etc/systemd/system/misp-modules.service /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now misp-modules # Sleep 9 seconds to give misp-modules a chance to spawn sleep 9 # Enable Enrichment, set better timeouts $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_services_enable" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_hover_enable" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_timeout" 300 $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_hover_timeout" 150 # TODO:"Investigate why the next one fails" #$SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_asn_history_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_cve_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_dns_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_btc_steroids_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_ipasn_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_yara_syntax_validator_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_yara_query_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_pdf_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_docx_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_xlsx_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_pptx_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_ods_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_odt_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_services_url" "http://127.0.0.1" $SUDO_WWW $CAKE Admin setSetting "Plugin.Enrichment_services_port" 6666 # Enable Import modules, set better timeout $SUDO_WWW $CAKE Admin setSetting "Plugin.Import_services_enable" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Import_services_url" "http://127.0.0.1" $SUDO_WWW $CAKE Admin setSetting "Plugin.Import_services_port" 6666 $SUDO_WWW $CAKE Admin setSetting "Plugin.Import_timeout" 300 $SUDO_WWW $CAKE Admin setSetting "Plugin.Import_ocr_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Import_mispjson_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Import_openiocimport_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Import_threatanalyzer_import_enabled" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Import_csvimport_enabled" true # Enable Export modules, set better timeout $SUDO_WWW $CAKE Admin setSetting "Plugin.Export_services_enable" true $SUDO_WWW $CAKE Admin setSetting "Plugin.Export_services_url" "http://127.0.0.1" $SUDO_WWW $CAKE Admin setSetting "Plugin.Export_services_port" 6666 $SUDO_WWW $CAKE Admin setSetting "Plugin.Export_timeout" 300 $SUDO_WWW $CAKE Admin setSetting "Plugin.Export_pdfexport_enabled" true } # <snippet-end 3_misp-modules.sh>
Warning
If you have installed the recommended Python 3 virtualenv to the recommended place of ${PATH_TO_MISP}/venv set the following MISP configurable
sudo -H -u www-data $CAKE Admin setSetting "MISP.python_bin" "${PATH_TO_MISP}/venv/bin/python"
sudo -u apache $RUN_PHP "$CAKE Admin setSetting "MISP.python_bin" "${PATH_TO_MISP}/venv/bin/python""
Warning
Make sure that the STIX libraries and GnuPG work as intended, if not, refer to the relevant sections in the install guide you are currently reading.
Notice
Now log in using the webinterface: http://misp/users/login
The default user/pass = admin@admin.test/admin
Using the server settings tool in the admin interface (Administration -> Server Settings), set MISP up to your preference.
It is especially vital that no critical issues remain!
Don't forget to change the email, password and authentication key after installation.
Once done, have a look at the diagnostics.
Notice
If any of the directories that MISP uses to store files is not writeable to the apache user, change the permissions
you can do this by running the following commands:
chmod -R 750 ${PATH_TO_MISP}/<directory path with an indicated issue> # /!\ Depending on your OS replace www-data with apache or www or whatever user is the web server user. chown -R www-data:www-data ${PATH_TO_MISP}/<directory path with an indicated issue>
Notice
If anything goes wrong, make sure that you check MISP's logs for errors:
# ${PATH_TO_MISP}/app/tmp/logs/error.log
# ${PATH_TO_MISP}/app/tmp/logs/resque-worker-error.log
# ${PATH_TO_MISP}/app/tmp/logs/resque-scheduler-error.log
# ${PATH_TO_MISP}/app/tmp/logs/resque-2018-10-25.log //where the actual date is the current date
Recommended actions¶
-
By default CakePHP exposes its name and version in email headers. Apply a patch to remove this behavior.
-
You should really harden your OS
- You should really harden the configuration of Apache
- You should really harden the configuration of MySQL
- Keep your software up2date (MISP, CakePHP and everything else)
- Log and audit
Optional features¶
MISP has a new pub/sub feature, using ZeroMQ. To enable it, simply run the following command¶
$SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install pyzmq
MISP has a feature for publishing events to Kafka. To enable it, simply run the following commands¶
# <snippet-begin 4_kafka.sh> installKafka () { sudo apt-get install librdkafka-dev php-dev -y sudo pecl channel-update pecl.php.net sudo pecl install rdkafka echo "extension=rdkafka.so" | sudo tee ${PHP_ETC_BASE}/mods-available/rdkafka.ini sudo phpenmod rdkafka sudo service apache2 restart } # <snippet-end 4_kafka.sh>
MISP Dashboard¶
# <snippet-begin 4_misp-dashboard.sh> # Main MISP Dashboard install function mispDashboard () { debug "Install misp-dashboard" # Install pyzmq to main MISP venv debug "Installing PyZMQ" $SUDO_WWW ${PATH_TO_MISP}/venv/bin/pip install pyzmq cd /var/www sudo mkdir misp-dashboard sudo chown $WWW_USER:$WWW_USER misp-dashboard $SUDO_WWW git clone https://github.com/MISP/misp-dashboard.git cd misp-dashboard sudo -H /var/www/misp-dashboard/install_dependencies.sh sudo sed -i "s/^host\ =\ localhost/host\ =\ 0.0.0.0/g" /var/www/misp-dashboard/config/config.cfg sudo sed -i '/Listen 80/a Listen 0.0.0.0:8001' /etc/apache2/ports.conf sudo apt install libapache2-mod-wsgi-py3 net-tools -y echo "<VirtualHost *:8001> ServerAdmin admin@misp.local ServerName misp.local DocumentRoot /var/www/misp-dashboard WSGIDaemonProcess misp-dashboard \ user=misp group=misp \ python-home=/var/www/misp-dashboard/DASHENV \ processes=1 \ threads=15 \ maximum-requests=5000 \ listen-backlog=100 \ queue-timeout=45 \ socket-timeout=60 \ connect-timeout=15 \ request-timeout=60 \ inactivity-timeout=0 \ deadlock-timeout=60 \ graceful-timeout=15 \ eviction-timeout=0 \ shutdown-timeout=5 \ send-buffer-size=0 \ receive-buffer-size=0 \ header-buffer-size=0 \ response-buffer-size=0 \ server-metrics=Off WSGIScriptAlias / /var/www/misp-dashboard/misp-dashboard.wsgi <Directory /var/www/misp-dashboard> WSGIProcessGroup misp-dashboard WSGIApplicationGroup %{GLOBAL} Require all granted </Directory> LogLevel info ErrorLog /var/log/apache2/misp-dashboard.local_error.log CustomLog /var/log/apache2/misp-dashboard.local_access.log combined ServerSignature Off </VirtualHost>" | sudo tee /etc/apache2/sites-available/misp-dashboard.conf # Enable misp-dashboard in apache and reload sudo a2ensite misp-dashboard sudo systemctl restart apache2 # Needs to be started after apache2 is reloaded so the port status check works $SUDO_WWW bash /var/www/misp-dashboard/start_all.sh # Add misp-dashboard to rc.local to start on boot. sudo sed -i -e '$i \sudo -u www-data bash /var/www/misp-dashboard/start_all.sh > /tmp/misp-dashboard_rc.local.log\n' /etc/rc.local } # <snippet-end 4_misp-dashboard.sh> # <snippet-begin 4_misp-dashboard-cake.sh> dashboardCAKE () { # Enable ZeroMQ for misp-dashboard $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_enable" true $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_event_notifications_enable" true $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_object_notifications_enable" true $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_object_reference_notifications_enable" true $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_attribute_notifications_enable" true $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_sighting_notifications_enable" true $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_user_notifications_enable" true $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_organisation_notifications_enable" true $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_port" 50000 $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_redis_host" "localhost" $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_redis_port" 6379 $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_redis_database" 1 $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_redis_namespace" "mispq" $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_include_attachments" false $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_tag_notifications_enable" false $SUDO_WWW $CAKE Admin setSetting "Plugin.ZeroMQ_audit_notifications_enable" false } # <snippet-end 4_misp-dashboard-cake.sh>
Install viper framework (with a virtualenv)¶
# <snippet-begin 6_viper.sh> # viper-web is broken ATM # Main Viper install function viper () { export PATH=$PATH:/home/misp/.local/bin debug "Installing Viper dependencies" cd /usr/local/src/ sudo apt-get install \ libssl-dev swig python3-ssdeep p7zip-full unrar-free sqlite python3-pyclamd exiftool radare2 \ python3-magic python3-sqlalchemy python3-prettytable libffi-dev libfreetype6-dev libpng-dev -qy if [[ -f "/etc/debian_version" ]]; then if [[ "$(cat /etc/debian_version)" == "9.9" ]]; then sudo apt-get install libpython3.5-dev -qy fi fi echo "Cloning Viper" $SUDO_USER git clone https://github.com/viper-framework/viper.git $SUDO_USER git clone https://github.com/viper-framework/viper-web.git sudo chown -R $MISP_USER:$MISP_USER viper sudo chown -R $MISP_USER:$MISP_USER viper-web cd viper echo "Creating virtualenv" $SUDO_USER virtualenv -p python3 venv echo "Submodule update" # TODO: Check for current user install permissions $SUDO_USER git submodule update --init --recursive echo "pip install deps" $SUDO_USER ./venv/bin/pip install pefile olefile jbxapi Crypto pypdns pypssl r2pipe pdftools virustotal-api SQLAlchemy PrettyTable python-magic scrapy https://github.com/lief-project/packages/raw/lief-master-latest/pylief-0.9.0.dev.zip $SUDO_USER ./venv/bin/pip install . echo 'update-modules' |/usr/local/src/viper/venv/bin/viper cd /usr/local/src/viper-web $SUDO_USER sed -i '1 s/^.*$/\#!\/usr\/local\/src\/viper\/venv\/bin\/python/' viper-web $SUDO_USER /usr/local/src/viper/venv/bin/pip install -r requirements.txt echo "Launching viper-web" $SUDO_USER /usr/local/src/viper-web/viper-web -p 8888 -H 0.0.0.0 & echo 'PATH="/home/misp/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/src/viper:/var/www/MISP/app/Console"' |sudo tee /etc/environment echo ". /etc/environment" >> /home/${MISP_USER}/.profile # TODO: Perms, MISP_USER_HOME, nasty hack cuz Kali on R00t if [ -f /home/${MISP_USER}/.viper/viper.conf ]; then VIPER_HOME="/home/${MISP_USER}/.viper" else VIPER_HOME="${HOME}/.viper" fi echo "Setting misp_url/misp_key" $SUDO_USER sed -i "s/^misp_url\ =/misp_url\ =\ http:\/\/localhost/g" ${VIPER_HOME}/viper.conf $SUDO_USER sed -i "s/^misp_key\ =/misp_key\ =\ $AUTH_KEY/g" ${VIPER_HOME}/viper.conf # Reset admin password to: admin/Password1234 echo "Fixing admin.db with default password" VIPER_COUNT=0 while [ "$(sudo sqlite3 ${VIPER_HOME}/admin.db 'UPDATE auth_user SET password="pbkdf2_sha256$100000$iXgEJh8hz7Cf$vfdDAwLX8tko1t0M1TLTtGlxERkNnltUnMhbv56wK/U="'; echo $?)" -ne "0" ]; do # FIXME This might lead to a race condition, the while loop is sub-par sudo chown $MISP_USER:$MISP_USER ${VIPER_HOME}/admin.db echo "Updating viper-web admin password, giving process time to start-up, sleeping 5, 4, 3,…" sleep 6 VIPER_COUNT=$[$VIPER_COUNT+1] if [[ "$VIPER_COUNT" > '10' ]]; then echo "Something is wrong with updating viper. Continuing without db update." break fi done # Add viper-web to rc.local to be started on boot sudo sed -i -e '$i \sudo -u misp /usr/local/src/viper/viper-web -p 8888 -H 0.0.0.0 > /tmp/viper-web_rc.local.log &\n' /etc/rc.local } # <snippet-end 6_viper.sh>
Experimental ssdeep correlations¶
installing ssdeep¶
# <snippet-begin 6_ssdeep.sh> ssdeep () { debug "Install ssdeep 2.14.1" cd /usr/local/src $SUDO_USER wget https://github.com/ssdeep-project/ssdeep/releases/download/release-2.14.1/ssdeep-2.14.1.tar.gz $SUDO_USER tar zxvf ssdeep-2.14.1.tar.gz cd ssdeep-2.14.1 $SUDO_USER ./configure --datadir=/usr --prefix=/usr --localstatedir=/var --sysconfdir=/etc $SUDO_USER make sudo make install #installing ssdeep_php sudo pecl channel-update pecl.php.net sudo pecl install ssdeep # You should add "extension=ssdeep.so" to mods-available - Check /etc/php for your current version echo "extension=ssdeep.so" | sudo tee ${PHP_ETC_BASE}/mods-available/ssdeep.ini sudo phpenmod ssdeep sudo service apache2 restart } # <snippet-end 6_ssdeep.sh>
Install mail to misp¶
# <snippet-begin 5_mail_to_misp.sh> # Main mail2misp install function mail2misp () { debug "Installing Mail2${LBLUE}MISP${NC}" cd /usr/local/src/ sudo apt-get install cmake libcaca-dev liblua5.3-dev -y $SUDO_USER git clone https://github.com/MISP/mail_to_misp.git $SUDO_USER git clone git://github.com/stricaud/faup.git faup $SUDO_USER git clone git://github.com/stricaud/gtcaca.git gtcaca sudo chown -R ${MISP_USER}:${MISP_USER} faup mail_to_misp gtcaca cd gtcaca $SUDO_USER mkdir -p build cd build $SUDO_USER cmake .. && $SUDO_USER make sudo make install cd ../../faup $SUDO_USER mkdir -p build cd build $SUDO_USER cmake .. && $SUDO_USER make sudo make install sudo ldconfig cd ../../mail_to_misp $SUDO_USER virtualenv -p python3 venv $SUDO_USER ./venv/bin/pip install https://github.com/lief-project/packages/raw/lief-master-latest/pylief-0.9.0.dev.zip $SUDO_USER ./venv/bin/pip install -r requirements.txt $SUDO_USER cp mail_to_misp_config.py-example mail_to_misp_config.py ##$SUDO cp mail_to_misp_config.py-example mail_to_misp_config.py $SUDO_USER sed -i "s/^misp_url\ =\ 'YOUR_MISP_URL'/misp_url\ =\ 'https:\/\/localhost'/g" /usr/local/src/mail_to_misp/mail_to_misp_config.py $SUDO_USER sed -i "s/^misp_key\ =\ 'YOUR_KEY_HERE'/misp_key\ =\ '${AUTH_KEY}'/g" /usr/local/src/mail_to_misp/mail_to_misp_config.py } # <snippet-end 5_mail_to_misp.sh>
Hardening a base system¶
Intro¶
MISP is a web-based information sharing platform, by design it is kept rather simple and hardening can be done by following the common best practices.
Bare in mind that neither the MISP documentation efforts or the core MISP project can give you the ultimate guide on how to harden your system. This is not the purpose of the MISP Project but the purpose and care of those individuals and organizations deploying MISP Instances.
Nevertheless here is a very rough food for thoughts bulletpoint list for you to consider, and a list of some hardening ressources below.
- Are we using SSL by default? (Especially when syncing over the internet and exposing the API)
- How to we access the machine remotely? Via ssh? What is the path to get there? Does a bastion host make sense?
- Is the machine shared with other user accounts? Do I need to care about useri-land security due to this sharing?
- Is the instance deployed in the "cloud"? Is it a VPS? AWS? docker? ansible? kubernetes? whateverCloudContainterMagicIsFancibleNow?
- Do we need to encrypt the partitions where some data is stored?
- Are we redundant in case one MISP instance might fail?
- Is the database server and any other servers running on the machine bound to localhost? Do we need to expose because our setup is more complex?
- Do we have enough storage? What about MISP and size estimation anyways?
- Do we care about BIOS updates?
- Do we care about physical access to the servers? (Disabling USB ports etc...)
- Is any fancy management engine à la IME in use?
Apache¶
To make Apache less verbose in terms of sending banners, the belo might help.
diff --git a/apache2/conf-available/security.conf b/apache2/conf-available/security.conf index f9f69d4..2e8fd78 100644 --- a/apache2/conf-available/security.conf +++ b/apache2/conf-available/security.conf @@ -22,7 +22,7 @@ # Set to one of: Full | OS | Minimal | Minor | Major | Prod # where Full conveys the most information, and Prod the least. #ServerTokens Minimal -ServerTokens OS +ServerTokens Prod #ServerTokens Full # @@ -33,7 +33,7 @@ ServerTokens OS # Set to "EMail" to also include a mailto: link to the ServerAdmin. # Set to one of: On | Off | EMail #ServerSignature Off -ServerSignature On +ServerSignature Off # # Allow TRACE method
Resources¶
IT Security Guidelines for TLS by NCSC.nl
Weak Diffie-Hellman and the Logjam Attack
INSTALL.sh¶
Notice
The following section is an administrative section that is used by the "INSTALL.sh" script. Please ignore.
# <snippet-begin 0_support-functions.sh> # Leave empty for NO debug messages, if run with set -x or bash -x it will enable DEBUG by default DEBUG= case "$-" in *x*) NO_PROGRESS=1; DEBUG=1 ;; *) NO_PROGRESS=0 ;; esac ## Function Section ## ## Usage of this script usage () { if [ "$0" == "bash" ]; then WEB_INSTALL=1 SCRIPT_NAME="Web Installer Command" else SCRIPT_NAME=$0 fi exec &> /dev/tty space echo -e "Please specify what type of ${LBLUE}MISP${NC} setup you want to install." space echo -e "${SCRIPT_NAME} -c | Install ONLY ${LBLUE}MISP${NC} Core" # core echo -e " -M | ${LBLUE}MISP${NC} modules" # modules echo -e " -D | ${LBLUE}MISP${NC} dashboard" # dashboard echo -e " -V | Viper" # viper echo -e " -m | Mail 2 ${LBLUE}MISP${NC}" # mail2 echo -e " -S | Experimental ssdeep correlations" # ssdeep echo -e " -A | Install ${YELLOW}all${NC} of the above" # all space echo -e " -C | Only do ${YELLOW}pre-install checks and exit${NC}" # pre space echo -e " -u | Do an unattended Install, no questions asked" # UNATTENDED echo -e "${HIDDEN} -U | Attempt and upgrade of selected item${NC}" # UPGRADE echo -e "${HIDDEN} -N | Nuke this MISP Instance${NC}" # NUKE echo -e "${HIDDEN} -f | Force test install on current Ubuntu LTS schim, add -B for 18.04 -> 18.10, or -BB 18.10 -> 19.10)${NC}" # FORCE echo -e "Options can be combined: ${SCRIPT_NAME} -c -V -D # Will install Core+Viper+Dashboard" space echo -e "Recommended is either a barebone MISP install (ideal for syncing from other instances) or" echo -e "MISP + modules - ${SCRIPT_NAME} -c -M" space } # Check if element is contained in array containsElement () { local e match="$1" shift for e; do [[ "$e" == "$match" ]] && return 0; done return 1 } checkOpt () { # checkOpt feature containsElement $1 "${options[@]}" } setOpt () { options=() for o in $@; do case "$o" in ("-c") echo "core"; CORE=1 ;; ("-V") echo "viper"; VIPER=1 ;; ("-M") echo "modules"; MODULES=1 ;; ("-D") echo "dashboard"; DASHBOARD=1 ;; ("-m") echo "mail2"; MAIL2=1 ;; ("-S") echo "ssdeep"; SSDEEP=1 ;; ("-A") echo "all"; ALL=1 ;; ("-C") echo "pre"; PRE=1 ;; ("-U") echo "upgrade"; UPGRADE=1 ;; ("-N") echo "nuke"; NUKE=1 ;; ("-u") echo "unattended"; UNATTENDED=1 ;; ("-f") echo "force"; FORCE=1 ;; (*) echo "$o is not a valid argument"; exit 1 ;; esac done } # check if command_exists command_exists () { command -v "$@" > /dev/null 2>&1 } # TODO: fix os detection mess # Try to detect what we are running on checkCoreOS () { # lsb_release can exist on any platform. RedHat package: redhat-lsb LSB_RELEASE=$(which lsb_release > /dev/null ; echo $?) APT=$(which apt > /dev/null 2>&1; echo -n $?) APT_GET=$(which apt-get > /dev/null 2>&1; echo $?) # debian specific # /etc/debian_version ## os-release #generic # /etc/os-release # Redhat checks if [[ -f "/etc/redhat-release" ]]; then echo "This is some redhat flavour" REDHAT=1 RHfla=$(cat /etc/redhat-release | cut -f 1 -d\ | tr '[:upper:]' '[:lower:]') fi } # Extract debian flavour checkFlavour () { FLAVOUR="" # Every system that we officially support has /etc/os-release if [ -r /etc/os-release ]; then FLAVOUR="$(. /etc/os-release && echo "$ID"| tr '[:upper:]' '[:lower:]')" fi case "$FLAVOUR" in ubuntu) if command_exists lsb_release; then dist_version="$(lsb_release --codename | cut -f2)" fi if [ -z "$dist_version" ] && [ -r /etc/lsb-release ]; then dist_version="$(. /etc/lsb-release && echo "$DISTRIB_CODENAME")" fi ;; debian|raspbian) dist_version="$(sed 's/\/.*//' /etc/debian_version | sed 's/\..*//')" case "$dist_version" in 10) dist_version="buster" ;; 9) dist_version="stretch" ;; esac ;; centos) if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then dist_version="$(. /etc/os-release && echo "$VERSION_ID")" fi echo "$FLAVOUR not supported at the moment" exit 1 ;; rhel|ol|sles) if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then dist_version="$(. /etc/os-release && echo "$VERSION_ID")" fi echo "$FLAVOUR not supported at the moment" exit 1 ;; *) if command_exists lsb_release; then dist_version="$(lsb_release --release | cut -f2)" fi if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then dist_version="$(. /etc/os-release && echo "$VERSION_ID")" fi ;; esac # FIXME: The below want to be refactored if [ "$FLAVOUR" == "ubuntu" ]; then RELEASE=$(lsb_release -s -r) debug "We detected the following Linux flavour: ${YELLOW}$(tr '[:lower:]' '[:upper:]' <<< ${FLAVOUR:0:1})${FLAVOUR:1} ${RELEASE}${NC}" else debug "We detected the following Linux flavour: ${YELLOW}$(tr '[:lower:]' '[:upper:]' <<< ${FLAVOUR:0:1})${FLAVOUR:1}${NC}" fi } # Check if this is a forked Linux distro check_forked () { # Check for lsb_release command existence, it usually exists in forked distros if command_exists lsb_release; then # Check if the `-u` option is supported set +e lsb_release -a -u > /dev/null 2>&1 lsb_release_exit_code=$? set -e # Check if the command has exited successfully, it means we're in a forked distro if [ "$lsb_release_exit_code" = "0" ]; then # Print info about current distro cat <<-EOF You're using '$FLAVOUR' version '$dist_version'. EOF # Get the upstream release info FLAVOUR=$(lsb_release -a -u 2>&1 | tr '[:upper:]' '[:lower:]' | grep -E 'id' | cut -d ':' -f 2 | tr -d '[:space:]') dist_version=$(lsb_release -a -u 2>&1 | tr '[:upper:]' '[:lower:]' | grep -E 'codename' | cut -d ':' -f 2 | tr -d '[:space:]') # Print info about upstream distro cat <<-EOF Upstream release is '$FLAVOUR' version '$dist_version'. EOF else if [ -r /etc/debian_version ] && [ "$FLAVOUR" != "ubuntu" ] && [ "$FLAVOUR" != "raspbian" ]; then # We're Debian and don't even know it! FLAVOUR=debian dist_version="$(sed 's/\/.*//' /etc/debian_version | sed 's/\..*//')" case "$dist_version" in 10) dist_version="buster" ;; 9) dist_version="stretch" ;; 8|'Kali Linux 2') dist_version="jessie" ;; esac fi fi fi } checkInstaller () { # TODO: Implement $FLAVOUR checks and install depending on the platform we are on if [[ $(which shasum > /dev/null 2>&1 ; echo $?) != 0 ]]; then sudo apt install libdigest-sha-perl -qyy fi # SHAsums to be computed, not the -- notatiation is for ease of use with rhash SHA_SUMS="--sha1 --sha256 --sha384 --sha512" for sum in $(echo ${SHA_SUMS} |sed 's/--sha//g'); do /usr/bin/wget --no-cache -q -O /tmp/INSTALL.sh.sha${sum} https://raw.githubusercontent.com/MISP/MISP/2.4/INSTALL/INSTALL.sh.sha${sum} INSTsum=$(shasum -a ${sum} ${0} | cut -f1 -d\ ) chsum=$(cat /tmp/INSTALL.sh.sha${sum} | cut -f1 -d\ ) if [[ "${chsum}" == "${INSTsum}" ]]; then echo "sha${sum} matches" else echo "sha${sum}: ${chsum} does not match the installer sum of: ${INSTsum}" echo "Delete installer, re-download and please run again." exit 1 fi done } # Extract manufacturer checkManufacturer () { if [ -z $(which dmidecode) ]; then checkAptLock sudo apt install dmidecode -qy fi MANUFACTURER=$(sudo dmidecode -s system-manufacturer) echo $MANUFACTURER } # Dynamic horizontal spacer if needed, for autonomeous an no progress bar install, we are static. space () { if [[ "$NO_PROGRESS" == "1" ]] || [[ "$PACKER" == "1" ]]; then echo "--------------------------------------------------------------------------------" return fi # Check terminal width num=`tput cols` for i in `seq 1 $num`; do echo -n "-" done echo "" } # Spinner so the user knows something is happening spin() { if [[ "$NO_PROGRESS" == "1" ]]; then return fi spinner="/|\\-/|\\-" while : do for i in `seq 0 7` do echo -n "${spinner:$i:1}" echo -en "\010" sleep 0.$i done done } # Progress bar progress () { progress=$[$progress+$1] if [[ "$NO_PROGRESS" == "1" ]] || [[ "$PACKER" == "1" ]]; then echo "progress=${progress}" > /tmp/INSTALL.stat return fi bar="#" # Prevent progress of overflowing if [[ $progress -ge 100 ]]; then echo -ne "##################################################################################################### (100%)\r" return fi # Display progress for p in $(seq 1 $progress); do bar+="#" echo -ne "$bar ($p%)\r" done echo -ne '\n' echo "progress=${progress}" > /tmp/INSTALL.stat } # Check locale checkLocale () { debug "Checking Locale" # If locale is missing, generate and install a common UTF-8 if [[ ! -f /etc/default/locale || $(wc -l /etc/default/locale| cut -f 1 -d\ ) == "1" ]]; then checkAptLock sudo DEBIAN_FRONTEND=noninteractive apt install locales -qy sudo sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/g' /etc/locale.gen sudo locale-gen en_US.UTF-8 sudo update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 fi } # Simple function to check command exit code checkFail () { if [[ $2 -ne 0 ]]; then echo "iAmError: $1" echo "The last command exited with error code: $2" exit $2 fi } ask_o () { ANSWER="" if [ -z "${1}" ]; then echo "This function needs at least 1 parameter." exit 1 fi [ -z "${2}" ] && OPT1="y" || OPT1="${2}" [ -z "${3}" ] && OPT2="n" || OPT2="${3}" while true; do case "${ANSWER}" in "${OPT1}" | "${OPT2}") break ;; esac echo -e -n "${1} (${OPT1}/${OPT2}) " read ANSWER ANSWER=$(echo "${ANSWER}" | tr '[:upper:]' '[:lower:]') done } clean () { rm /tmp/INSTALL.stat rm /tmp/INSTALL.sh.* } # Check if misp user is present and if run as root checkID () { debug "Checking if run as root and $MISP_USER is present" if [[ $EUID == 0 ]]; then echo "This script cannot be run as a root" clean > /dev/null 2>&1 exit 1 elif [[ $(id $MISP_USER >/dev/null; echo $?) -ne 0 ]]; then if [[ "$UNATTENDED" != "1" ]]; then echo "There is NO user called '$MISP_USER' create a user '$MISP_USER' (y) or continue as $USER (n)? (y/n) " read ANSWER ANSWER=$(echo $ANSWER |tr '[:upper:]' '[:lower:]') else ANSWER="y" fi if [[ $ANSWER == "y" ]]; then sudo useradd -s /bin/bash -m -G adm,cdrom,sudo,dip,plugdev,www-data,staff $MISP_USER echo $MISP_USER:$MISP_PASSWORD | sudo chpasswd echo "User $MISP_USER added, password is: $MISP_PASSWORD" elif [[ $ANSWER == "n" ]]; then echo "Using $USER as install user, hope that is what you want." echo -e "${RED}Adding $USER to groups $WWW_USER and staff${NC}" MISP_USER=$USER sudo adduser $MISP_USER staff sudo adduser $MISP_USER $WWW_USER else echo "yes or no was asked, try again." sudo adduser $MISP_USER staff sudo adduser $MISP_USER $WWW_USER exit 1 fi else echo "User ${MISP_USER} exists, skipping creation" echo -e "${RED}Adding $MISP_USER to groups $WWW_USER and staff${NC}" sudo adduser $MISP_USER staff sudo adduser $MISP_USER $WWW_USER fi # FIXME: the below SUDO_USER check is a duplicate from global variables, try to have just one check # sudo config to run $LUSER commands if [[ "$(groups ${MISP_USER} |grep -o 'staff')" == "staff" ]]; then SUDO_USER="sudo -H -u ${MISP_USER} -g staff" else SUDO_USER="sudo -H -u ${MISP_USER}" fi } # pre-install check to make sure what we will be installing on, is ready and not a half installed system preInstall () { # preInstall needs to be able to be called before ANY action. Install/Upgrade/AddTool # Pre install wants to be the place too where the following is checked and set via ENV_VAR: # Check if composer is installed and functioning # Check if misp db is installed (API call would confirm that the DB indeed works) # Check apache config (Maybe try to talk to the server via api, this would confirm quite a lot) # Check if workers are running/installed, maybe kick them if they are not # /var/www/MISP/app/Config/[bootstrap,databases,core,config].php exists # /var/www/MISP perms are correct (for $SUDO_WWW useage) # # Check if $PATH_TO_MISP exists and is writable by $WWW_USER [[ -d "$PATH_TO_MISP" ]] && PATH_TO_MISP_EXISTS=1 && echo "$PATH_TO_MISP exists" # .git exists and git is working for $WWW_USER [[ -d "$PATH_TO_MISP/.git" ]] && PATH_TO_GIT_EXISTS=1 && echo "$PATH_TO_MISP/.git exists" && cd $PATH_TO_MISP && $SUDO_WWW git status # .gnupg exists and working correctly [[ -d "$PATH_TO_MISP/.gnupg" ]] && PATH_TO_GNUPG_EXISTS=1 && echo "$PATH_TO_MISP/.gnupg exists" # Extract username, password and dbname ##cat database.php |grep -v // |grep -e database -e login -e password |tr -d \' |tr -d \ |tr -d , |tr -d \> DBPASSWORD_MISP=$(cat database.php |grep -v // |grep -e password |tr -d \' |tr -d \ |tr -d , |tr -d \> |cut -f 2 -d=) DBUSER_MISP=$(cat database.php |grep -v // |grep -e login |tr -d \' |tr -d \ |tr -d , |tr -d \> |cut -f 2 -d=) DBNAME=$(cat database.php |grep -v // |grep -e database |tr -d \' |tr -d \ |tr -d , |tr -d \> |cut -f 2 -d=) AUTH_KEY=$(mysql --disable-column-names -B -u $DBUSER_MISP -p"$DBPASSWORD_MISP" $DBNAME -e 'SELECT authkey FROM users WHERE role_id=1 LIMIT 1') # Check if db exists [[ -d "/var/lib/mysql/$DBNAME" ]] && MISP_DB_DIR_EXISTS=1 && echo "/var/lib/mysql/$DBNAME exists" echo -e "${RED}Place-holder, not implemented yet.${NC}" exit } # Upgrade function upgrade () { headerJSON="application/json" Acc="Accept:" Autho="Authorization:" CT="Content-Type:" MISP_BASEURL="https://127.0.0.1" cd $PATH_TO_MISP/app ; $SUDO_WWW php composer.phar update $SUDO_WWW php composer.phar self-update for URN in $(echo "galaxies warninglists noticelists objectTemplates taxonomies"); do curl --header "$Autho $AUTH_KEY" --header "$Acc $headerJSON" --header "$CT $headerJSON" -k -X POST $MISP_BASEURL/$URN/update done echo -e "${RED}Place-holder, not implemented yet.${NC}" exit } # check is /usr/local/src is RW by misp user checkUsrLocalSrc () { echo "" if [[ -e /usr/local/src ]]; then WRITEABLE=$(sudo -H -u $MISP_USER touch /usr/local/src 2> /dev/null ; echo $?) if [[ "$WRITEABLE" == "0" ]]; then echo "Good, /usr/local/src exists and is writeable as $MISP_USER" else # TODO: The below might be shorter, more elegant and more modern #[[ -n $KALI ]] || [[ -n $UNATTENDED ]] && echo "Just do it" sudo chmod 2775 /usr/local/src sudo chown root:staff /usr/local/src fi else echo "/usr/local/src does not exist, creating." mkdir -p /usr/local/src sudo chmod 2775 /usr/local/src # TODO: Better handling /usr/local/src permissions if [[ "$(cat /etc/group |grep staff > /dev/null 2>&1)" == "0" ]]; then sudo chown root:staff /usr/local/src fi fi } kaliSpaceSaver () { # Future function in case Kali overlay on LiveCD is full echo "${RED}Not implement${NC}" } # Because Kali is l33t we make sure we run as root kaliOnRootR0ckz () { if [[ $EUID -ne 0 ]]; then echo "This script must be run as root" exit 1 elif [[ $(id $MISP_USER >/dev/null; echo $?) -ne 0 ]]; then useradd -s /bin/bash -m -G adm,cdrom,sudo,dip,plugdev,www-data,staff $MISP_USER echo $MISP_USER:$MISP_PASSWORD | chpasswd else # TODO: Make sure we consider this further down the road echo "User ${MISP_USER} exists, skipping creation" fi } setBaseURL () { debug "Setting Base URL" CONN=$(ip -br -o -4 a |grep UP |head -1 |tr -d "UP") IFACE=`echo $CONN |awk {'print $1'}` IP=`echo $CONN |awk {'print $2'}| cut -f1 -d/` # TODO: Consider "QEMU" if [[ "$(checkManufacturer)" != "innotek GmbH" ]] && [[ "$(checkManufacturer)" != "VMware, Inc." ]]; then debug "We guess that this is a physical machine and cannot possibly guess what the MISP_BASEURL might be." if [[ "$UNATTENDED" != "1" ]]; then echo "You can now enter your own MISP_BASEURL, if you wish to NOT do that, the MISP_BASEURL will be empty, which will work, but ideally you configure it afterwards." echo "Do you want to change it now? (y/n) " read ANSWER ANSWER=$(echo $ANSWER |tr '[:upper:]' '[:lower:]') if [[ "$ANSWER" == "y" ]]; then if [[ ! -z $IP ]]; then echo "It seems you have an interface called $IFACE UP with the following IP: $IP - FYI" echo "Thus your Base URL could be: https://$IP" fi echo "Please enter the Base URL, e.g: 'https://example.org'" echo "" echo -n "Enter Base URL: " read MISP_BASEURL else MISP_BASEURL='""' fi else MISP_BASEURL="https://misp.local" # Webserver configuration FQDN='misp.local' fi elif [[ $KALI == "1" ]]; then MISP_BASEURL="https://misp.local" # Webserver configuration FQDN='misp.local' else MISP_BASEURL='https://localhost:8443' IP=$(ip addr show | awk '$1 == "inet" {gsub(/\/.*$/, "", $2); print $2}' |grep -v "127.0.0.1" |tail -1) sudo iptables -t nat -A OUTPUT -p tcp --dport 8443 -j DNAT --to ${IP}:443 # Webserver configuration FQDN='localhost.localdomain' fi } # Test and install software RNG installRNG () { sudo modprobe tpm-rng 2> /dev/null if [ "$?" -eq "0" ]; then echo tpm-rng | sudo tee -a /etc/modules fi checkAptLock sudo apt install -qy rng-tools # This might fail on TPM grounds, enable the security chip in your BIOS sudo service rng-tools start if [ "$?" -eq "1" ]; then sudo apt purge -qy rng-tools sudo apt install -qy haveged sudo /etc/init.d/haveged start fi } # Kali upgrade kaliUpgrade () { debug "Running various Kali upgrade tasks" sudo apt update checkAptLock sudo DEBIAN_FRONTEND=noninteractive apt install --only-upgrade bash libc6 -y sudo DEBIAN_FRONTEND=noninteractive apt autoremove -y } # Disables sleep disableSleep () { debug "Disabling sleep etc if run from a Laptop as the install might take some time…" > /dev/tty gsettings set org.gnome.settings-daemon.plugins.power sleep-inactive-ac-timeout 0 2> /dev/null gsettings set org.gnome.settings-daemon.plugins.power sleep-inactive-battery-timeout 0 2> /dev/null gsettings set org.gnome.settings-daemon.plugins.power sleep-inactive-battery-type nothing 2> /dev/null gsettings set org.gnome.desktop.screensaver lock-enabled false 2> /dev/null gsettings set org.gnome.desktop.screensaver idle-activation-enabled false 2> /dev/null setterm -blank 0 -powersave off -powerdown 0 xset s 0 0 2> /dev/null xset dpms 0 0 2> /dev/null xset dpms force off xset s off 2> /dev/null service sleepd stop kill $(lsof | grep 'sleepd' | awk '{print $2}') checkAptLock } # Remove alias if present if [[ $(type -t checkAptLock) == "alias" ]]; then unalias checkAptLock; fi # Simple function to make sure APT is not locked checkAptLock () { SLEEP=3 while [ "$DONE" != "0" ]; do sudo apt-get check 2> /dev/null > /dev/null && DONE=0 echo -e "${LBLUE}apt${NC} is maybe ${RED}locked${NC}, waiting ${RED}$SLEEP${NC} seconds." > /dev/tty sleep $SLEEP SLEEP=$[$SLEEP+3] done unset DONE } # <snippet-begin 0_installDepsPhp70.sh> # Install Php 7.0 dependencies installDepsPhp70 () { debug "Installing PHP 7.0 dependencies" PHP_ETC_BASE=/etc/php/7.0 PHP_INI=${PHP_ETC_BASE}/apache2/php.ini sudo apt update sudo apt install -qy \ libapache2-mod-php \ php php-cli \ php-dev \ php-json php-xml php-mysql php-opcache php-readline php-mbstring \ php-redis php-gnupg \ php-gd for key in upload_max_filesize post_max_size max_execution_time max_input_time memory_limit do sudo sed -i "s/^\($key\).*/\1 = $(eval echo \${$key})/" $PHP_INI done } # <snippet-end 0_installDepsPhp70.sh> # <snippet-begin 0_installDepsPhp73.sh> # Install Php 7.3 deps installDepsPhp73 () { debug "Installing PHP 7.3 dependencies" PHP_ETC_BASE=/etc/php/7.3 PHP_INI=${PHP_ETC_BASE}/apache2/php.ini sudo apt update checkAptLock sudo apt install -qy \ libapache2-mod-php7.3 \ php7.3 php7.3-cli \ php7.3-dev \ php7.3-json php7.3-xml php7.3-mysql php7.3-opcache php7.3-readline php7.3-mbstring \ php-redis php-gnupg \ php-gd } # <snippet-end 0_installDepsPhp73.sh> # Installing core dependencies installDeps () { debug "Installing core dependencies" checkAptLock sudo apt update sudo apt install -qy etckeeper # Skip dist-upgrade for now, pulls in 500+ updated packages #sudo apt -y dist-upgrade gitMail=$(git config --global --get user.email ; echo $?) if [ "$?" -eq "1" ]; then git config --global user.email "root@kali.lan" fi gitUser=$(git config --global --get user.name ; echo $?) if [ "$?" -eq "1" ]; then git config --global user.name "Root User" fi [[ -n $KALI ]] || [[ -n $UNATTENDED ]] && sudo DEBIAN_FRONTEND=noninteractive apt install -qy postfix || sudo apt install -qy postfix sudo apt install -qy \ curl gcc git gnupg-agent make openssl redis-server neovim unzip zip libyara-dev python3-yara python3-redis python3-zmq sqlite3 \ mariadb-client \ mariadb-server \ apache2 apache2-doc apache2-utils \ python3-dev python3-pip libpq5 libjpeg-dev libfuzzy-dev ruby asciidoctor \ libxml2-dev libxslt1-dev zlib1g-dev python3-setuptools expect installRNG } # On Kali, the redis start-up script is broken. This tries to fix it. fixRedis () { # As of 20190124 redis-server init.d scripts are broken and need to be replaced sudo mv /etc/init.d/redis-server /etc/init.d/redis-server_`date +%Y%m%d` echo '#! /bin/sh ### BEGIN INIT INFO # Provides: redis-server # Required-Start: $syslog # Required-Stop: $syslog # Should-Start: $local_fs # Should-Stop: $local_fs # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: redis-server - Persistent key-value db # Description: redis-server - Persistent key-value db ### END INIT INFO PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin DAEMON=/usr/bin/redis-server DAEMON_ARGS=/etc/redis/redis.conf NAME=redis-server DESC=redis-server PIDFILE=/var/run/redis.pid test -x $DAEMON || exit 0 test -x $DAEMONBOOTSTRAP || exit 0 set -e case "$1" in start) echo -n "Starting $DESC: " touch $PIDFILE chown redis:redis $PIDFILE if start-stop-daemon --start --quiet --umask 007 --pidfile $PIDFILE --chuid redis:redis --exec $DAEMON -- $DAEMON_ARGS then echo "$NAME." else echo "failed" fi ;; stop) echo -n "Stopping $DESC: " if start-stop-daemon --stop --retry 10 --quiet --oknodo --pidfile $PIDFILE --exec $DAEMON then echo "$NAME." else echo "failed" fi rm -f $PIDFILE ;; restart|force-reload) ${0} stop ${0} start ;; *) echo "Usage: /etc/init.d/$NAME {start|stop|restart|force-reload}" >&2 exit 1 ;; esac exit 0' | sudo tee /etc/init.d/redis-server sudo chmod 755 /etc/init.d/redis-server sudo /etc/init.d/redis-server start } # generate MISP apache conf genApacheConf () { echo "<VirtualHost _default_:80> ServerAdmin admin@localhost.lu ServerName misp.local Redirect permanent / https://misp.local LogLevel warn ErrorLog /var/log/apache2/misp.local_error.log CustomLog /var/log/apache2/misp.local_access.log combined ServerSignature Off </VirtualHost> <VirtualHost _default_:443> ServerAdmin admin@localhost.lu ServerName misp.local DocumentRoot $PATH_TO_MISP/app/webroot <Directory $PATH_TO_MISP/app/webroot> Options -Indexes AllowOverride all Require all granted Order allow,deny allow from all </Directory> SSLEngine On SSLCertificateFile /etc/ssl/private/misp.local.crt SSLCertificateKeyFile /etc/ssl/private/misp.local.key # SSLCertificateChainFile /etc/ssl/private/misp-chain.crt LogLevel warn ErrorLog /var/log/apache2/misp.local_error.log CustomLog /var/log/apache2/misp.local_access.log combined ServerSignature Off Header set X-Content-Type-Options nosniff Header set X-Frame-Options DENY </VirtualHost>" | tee /etc/apache2/sites-available/misp-ssl.conf } # Add git pull update mechanism to rc.local - TODO: Make this better gitPullAllRCLOCAL () { sed -i -e '$i \git_dirs="/usr/local/src/misp-modules/ /var/www/misp-dashboard /usr/local/src/faup /usr/local/src/mail_to_misp /usr/local/src/misp-modules /usr/local/src/viper /var/www/misp-dashboard"\n' /etc/rc.local sed -i -e '$i \for d in $git_dirs; do\n' /etc/rc.local sed -i -e '$i \ echo "Updating ${d}"\n' /etc/rc.local sed -i -e '$i \ cd $d && sudo git pull &\n' /etc/rc.local sed -i -e '$i \done\n' /etc/rc.local } # Composer on php 7.0 does not need any special treatment the provided phar works well alias composer70='composer72' # Composer on php 7.2 does not need any special treatment the provided phar works well composer72 () { cd $PATH_TO_MISP/app mkdir /var/www/.composer ; chown $WWW_USER:$WWW_USER /var/www/.composer $SUDO_WWW php composer.phar install } # Composer on php 7.3 needs a recent version of composer.phar composer73 () { cd $PATH_TO_MISP/app mkdir /var/www/.composer ; chown $WWW_USER:$WWW_USER /var/www/.composer # Update composer.phar # If hash changes, check here: https://getcomposer.org/download/ and replace with the correct one # Current Sum for: v1.8.3 SHA384_SUM='48e3236262b34d30969dca3c37281b3b4bbe3221bda826ac6a9a62d6444cdb0dcd0615698a5cbe587c3f0fe57a54d8f5' $SUDO_WWW php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" $SUDO_WWW php -r "if (hash_file('SHA384', 'composer-setup.php') === '$SHA384_SUM') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); exit(137); } echo PHP_EOL;" checkFail "composer.phar checksum failed, please investigate manually. " $? $SUDO_WWW php composer-setup.php $SUDO_WWW php -r "unlink('composer-setup.php');" $SUDO_WWW php composer.phar install } # Enable various core services enableServices () { update-rc.d mysql enable update-rc.d apache2 enable update-rc.d redis-server enable } # Generate rc.local genRCLOCAL () { if [ ! -e /etc/rc.local ]; then echo '#!/bin/sh -e' | tee -a /etc/rc.local echo 'exit 0' | tee -a /etc/rc.local chmod u+x /etc/rc.local fi sed -i -e '$i \echo never > /sys/kernel/mm/transparent_hugepage/enabled\n' /etc/rc.local sed -i -e '$i \echo 1024 > /proc/sys/net/core/somaxconn\n' /etc/rc.local sed -i -e '$i \sysctl vm.overcommit_memory=1\n' /etc/rc.local sed -i -e '$i \[ -f /etc/init.d/firstBoot ] && bash /etc/init.d/firstBoot\n' /etc/rc.local } # Run PyMISP tests runTests () { echo "url = '${MISP_BASEURL}' key = '${AUTH_KEY}'" |sudo tee ${PATH_TO_MISP}/PyMISP/tests/keys.py sudo chown -R $WWW_USER:$WWW_USER $PATH_TO_MISP/PyMISP/ sudo -H -u $WWW_USER sh -c "cd $PATH_TO_MISP/PyMISP && git submodule foreach git pull origin master" sudo -H -u $WWW_USER ${PATH_TO_MISP}/venv/bin/pip install -e $PATH_TO_MISP/PyMISP/.[fileobjects,neo,openioc,virustotal,pdfexport] sudo -H -u $WWW_USER git clone https://github.com/viper-framework/viper-test-files.git $PATH_TO_MISP/PyMISP/tests/viper-test-files sudo -H -u $WWW_USER sh -c "cd $PATH_TO_MISP/PyMISP && ${PATH_TO_MISP}/venv/bin/python tests/testlive_comprehensive.py" } # Nuke the install, meaning remove all MISP data but no packages, this makes testing the installer faster nuke () { echo -e "${RED}YOU ARE ABOUT TO DELETE ALL MISP DATA! Sleeping 10, 9, 8...${NC}" sleep 10 sudo rm -rvf /usr/local/src/{misp-modules,viper,mail_to_misp,LIEF,faup} sudo rm -rvf /var/www/MISP sudo mysqladmin drop misp sudo mysql -e "DROP USER misp@localhost" } # Final function to let the user know what happened theEnd () { space echo "Admin (root) DB Password: $DBPASSWORD_ADMIN" |$SUDO_USER tee /home/${MISP_USER}/mysql.txt echo "User (misp) DB Password: $DBPASSWORD_MISP" |$SUDO_USER tee -a /home/${MISP_USER}/mysql.txt echo "Authkey: $AUTH_KEY" |$SUDO_USER tee -a /home/${MISP_USER}/MISP-authkey.txt clear space echo -e "${LBLUE}MISP${NC} Installed, access here: ${MISP_BASEURL}" echo echo "User: admin@admin.test" echo "Password: admin" space [[ -n $KALI ]] || [[ -n $DASHBOARD ]] || [[ -n $ALL ]] && echo -e "${LBLUE}MISP${NC} Dashboard, access here: ${MISP_BASEURL}:8001" [[ -n $KALI ]] || [[ -n $DASHBOARD ]] || [[ -n $ALL ]] && space [[ -n $KALI ]] || [[ -n $VIPER ]] || [[ -n $ALL ]] && echo -e "viper-web installed, access here: ${MISP_BASEURL}:8888" [[ -n $KALI ]] || [[ -n $VIPER ]] || [[ -n $ALL ]] && echo -e "viper-cli configured with your ${LBLUE}MISP${NC} ${RED}Site Admin Auth Key${NC}" [[ -n $KALI ]] || [[ -n $VIPER ]] || [[ -n $ALL ]] && echo [[ -n $KALI ]] || [[ -n $VIPER ]] || [[ -n $ALL ]] && echo "User: admin" [[ -n $KALI ]] || [[ -n $VIPER ]] || [[ -n $ALL ]] && echo "Password: Password1234" [[ -n $KALI ]] || [[ -n $VIPER ]] || [[ -n $ALL ]] && space echo -e "The following files were created and need either ${RED}protection or removal${NC} (${YELLOW}shred${NC} on the CLI)" echo "/home/${MISP_USER}/mysql.txt" echo -e "${RED}Contents:${NC}" cat /home/${MISP_USER}/mysql.txt echo "/home/${MISP_USER}/MISP-authkey.txt" echo -e "${RED}Contents:${NC}" cat /home/${MISP_USER}/MISP-authkey.txt space echo -e "The ${RED}LOCAL${NC} system credentials:" echo "User: ${MISP_USER}" echo "Password: ${MISP_PASSWORD} # Or the password you used of your custom user" space echo "GnuPG Passphrase is: ${GPG_PASSPHRASE}" space echo "To enable outgoing mails via postfix set a permissive SMTP server for the domains you want to contact:" echo echo "sudo postconf -e 'relayhost = example.com'" echo "sudo postfix reload" space echo -e "Enjoy using ${LBLUE}MISP${NC}. For any issues see here: https://github.com/MISP/MISP/issues" space if [[ "$UNATTENDED" == "1" ]]; then echo -e "${RED}Unattended install!${NC}" echo -e "This means we guessed the Base URL, it might be wrong, please double check." space fi if [[ "$PACKER" == "1" ]]; then echo -e "${RED}This was an Automated Packer install!${NC}" echo -e "This means we forced an unattended install." space fi if [[ "$USER" != "$MISP_USER" && "$UNATTENDED" != "1" ]]; then sudo su - ${MISP_USER} fi } ## End Function Section Nothing allowed in .md after this line ## # <snippet-end 0_support-functions.sh>