MISP - Elastic Stack - Docker Lab

By Luciano Righetti April 5, 2024

MISP - Elastic Stack - Docker

This lab explains how to connect MISP to the Elastic Stack in order to leverage IOCs from MISP and trigger alerts based on user defined rules.

Elastic-MISP Overview Diagram

MISP is used to gather IOCs from different sources such as open source Threat Intelligence feeds.

Filebeat pulls IOCs from MISP and pushed them to the Elasticsearch instance via the Threat Intel module 1.

To simplify this guide Elastic Agent 2 in installed standalone mode is installed on the hosts we want to monitor, but Fleet 3 could be used instead.

Kibana 4 is used for exploring the IOCs, creating rules and visualizing the alerts.

Everything in this lab is run on Docker 5.

Installation

  1. Clone the lab repository.
    $ git clone https://github.com/righel/elastic-misp-docker-lab.git & cd elastic-misp-docker-lab

MISP

  1. Create the .env file:

    $ cp template.env .env

  2. Start the MISP containers.

    $ docker compose up -d

  3. When MISP containers finish starting, create a sync user for Elastic on MISP.

    Using MISP CLI:

    $ docker-compose exec misp-core app/Console/cake User create elastic@admin.test 5 1
$ docker-compose exec misp-core app/Console/cake User change_authkey elastic2@admin.test
Old authentication keys disabled and new key created: 06sDmKQK3E6MSJwsOhYT3N4NzfTpe53ruV0Bydf0

    Using MISP UI:

    Default MISP credentials

    User: admin@admin.test

    Password: admin

Elastic Stack

The yaml configuration files for Elasticsearch, Kibana and Filebeat are located in elastic/config/ directory. You can review and change these settings before deploying Elastic Stack.

For adjusting the Filebeat MISP Threat Intel module, check elastic/config/filebeat.yml:

filebeat.modules:
  - module: threatintel
    misp:
      enabled: true
      var.input: httpjson
      var.url: "https://${MISP_HOST}/events/restSearch"
      var.api_token: "${MISP_ELASTIC_API_KEY}"
      var.first_interval: 24h
      var.interval: 10m
      var.ssl.verification_mode: none
      var.filters:
        type: ["md5", "sha256", "sha512", "url", "uri", "ip-src", "ip-dst", "hostname", "domain"]
        tags: ['workflow:state="complete"']

For more details refer to the official docs:

  1. Modify some environment variables before firing up Elastic Stack.

    • KIBANA_ENCRYPTION_KEY: Has to be 32 chars string to set up xpack.encryptedSavedObjects.encryptionKey on Kibana.
    • MISP_ELASTIC_API_KEY: The MISP API key generated in step 4 for the sync user elastic@admin.test.

  2. Start the Elastic Stack containers.

    $ docker-compose -f docker-compose.elastic.yml -d

Now you can use MISP and the Elastic Stack.

Creating Kibana detection rules

  1. Go to Kibana > Security > Rules and click on Detection rules (SIEM)

    Kibana Security Rules page

  2. Click on Kibana Add Elastic Rules and search for Threat Intel, and select the rules you are interested on.

    Kibana Install Threat Intel rules

  3. Click on Kibana Install rules

  4. Go back to Kibana > Security > Rules and click on Detection rules (SIEM), click on the Disabled rules filter to show the recently installed rules (by default they are disabled), and enable them.

    Kibana Enabled rules

Now Elastic will generate alerts if it detects any hash, url or domain matching with MISP IOCs.

Demo

Install Elastic Agent (standalone)

  1. Go to Kibana > Management > Fleet and switch to the Agent policies tab and click on the Create agent policy button.

Kibana Fleet Create Agent Policy

  1. Click on the policy to access it and click on Add integration and search for Network Packet Capture, click on it and click on Add Network Packet Capture

    Add Network Packet Capture

  2. Configure the Network Packet Capture integration if needed. Network Packet Capture configuration

    Click on Save and continue.

    If prompted to add a Elastic Agent, click on Add Elastic Agent later.

    Kibana Elastic Agent Policy

  3. Click on Actions > Add agent, switch to the Run standalone tab.

    Configure standanlone Elastic Agent

    Click on Download Policy.

  4. Create an API key for the Elastic Agent communication to Elasticsearch 6. Navigate to Kibana > Stack Management > API keys and click Create API key.

    Kibana Create Elastic Agent API key

    1. Enable the Restrict privileges toggle and copy the following configuration.

      {
  "standalone_agent": {
    "cluster": [
      "monitor"
    ],
    "indices": [
      {
        "names": [
          "logs-*-*", "metrics-*-*", "traces-*-*", "synthetics-*-*" 
        ],
        "privileges": [
          "auto_configure", "create_doc"
        ]
      }
    ]
  }
}

    2. Click on Create API key.

    3. Select Beats and copy the API key.

      Elastic Agent API key

  5. Open the elastic-agent.yml configuration downloaded on step 5 and modify the outputs section it to use API authentication and change the Elasticsearch host from localhost to elasticsearch.

    outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://elasticsearch:9200'
    api_key: 'RZAxqY4BZZJcsxX19zAw:d9AabeoqRkKwm27K_KEgKA'
    preset: balanced

  6. Install Elastic Agent in the host we want to monitor. Start an Ubuntu container and plug it to the MISP-Elastic Stack lab network.

    $ docker run -it --network=elastic-misp-docker-lab_default --name=my_monitored_host ubuntu

  7. Follow the steps to install the standalone Elastic Agent 7.

    $ curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.13.1-linux-x86_64.tar.gz
root@be44a9a86e24:/# apt update
...
root@be44a9a86e24:/# apt install curl -y
...
root@be44a9a86e24:/# cd /tmp
root@be44a9a86e24:/# curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.13.1-linux-x86_64.tar.gz
root@be44a9a86e24:/# tar xzvf elastic-agent-8.13.1-linux-x86_64.tar.gz
...

# from a different terminal, copy the elastic-agent.yml config file into the container
docker cp elastic-agent.yml my_monitored_host:/tmp/elastic-agent-8.13.1-linux-x86_64/elastic-agent.yml

# back on the docker terminal, install the agent
root@be44a9a86e24:/# cd elastic-agent-8.13.1-linux-x86_64/
root@be44a9a86e24:/# ./elastic-agent install
    Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y
    Do you want to enroll this Agent into Fleet? [Y/n]:n
    [=   ] Service Started  [32s] Elastic Agent successfully installed, starting enrollment.
    [=   ] Done  [32s]                       
    Elastic Agent has been successfully installed.

    Done. Now the docker container is being monitored by the Elastic Agent and the metrics are being pushed to Elasticsearch.

Testing

  1. Go to MISP, create a new Event an add a ip-dst type attribute. 2. Add MISP Attribute

  2. Add the workflow:state=“complete” tag to the event to it is picked up by Elasticsearch.

    MISP Add Tag

  3. Publish the Event.

Depending on the vars.interval set on the Threat Intel module in the filebeat.yml configuration file, it may take some time for the IOC to get into Elasticsearch.

  1. From your monitored docker host, generate traffic to one of the MISP IOCs.

    root@be44a9a86e24:/# curl -I https://circl.lu
...

  2. Navigate to Kibana > Security > Alerts, you should now see some alerts triggered by the Threat Intel IP Address Indicator Match. Kibana Security Alerts

By default Elastic runs these detection rules every 4 hours, you can configure the internal reduce

Done! Now you will get alerts on Kibana when an event in one of your monitored hosts matches a MISP IOC.

  1. https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-threatintel.html ↩︎

  2. https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html ↩︎

  3. https://www.elastic.co/guide/en/fleet/current/fleet-overview.html ↩︎

  4. https://www.elastic.co/kibana ↩︎

  5. https://github.com/righel/elastic-misp-docker-lab ↩︎

  6. https://www.elastic.co/guide/en/fleet/current/grant-access-to-elasticsearch.html ↩︎

  7. https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html ↩︎

Search

Tags