misp-grafana

A real-time Grafana dashboard using MISP ZeroMQ message queue and InfluxDB .

Grafana Dashboard

Infrastructure

Installation

Using containers

Using docker is the easiest way to do it and comes pre-configured with the dashboard and InfluxDB datasource.

$ cd docker
$ docker-compose up -d

NOTE: For production usage change the default credentials.

Pushing MISP metrics

After InfluxDB and Grafana are up and running, adjust .env file to your environment, and run the ZMQ subscriber script:

$ cd src/
$ pip install -r requirements.txt
$ python3 src/push_zmq_to_influxdb.py
[INFO] [2022-03-31 17:32:51,602] - Subscribed to ZMQ
[INFO] [2022-03-31 17:32:56,945] - Received message from topic: misp_json_self
[INFO] [2022-03-31 17:32:56,945] - ZMQ status pushed to InfluxDB
...

NOTE: This script must be running all times to feed InfluxDB with your MISP activity.

Pushing MISP logs

Telegraf agent is used to parse MISP logs and push them to InfluxDB, to install it follow this guide:

Run the agent:

$ telegraf --config telegraf/telegraf.conf

NOTE: For the HTTP response time panels you need to extend the default Apache combined log format with the %D option, your Apache log configuration as follow:

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %D" combined

MISP

Go to your ZeroMQ plugin settings in MISP and set the following values:

  'ZeroMQ_enable' => true,
  'ZeroMQ_host' => '127.0.0.1',
  'ZeroMQ_port' => 50000,
  'ZeroMQ_redis_host' => 'localhost',
  'ZeroMQ_redis_port' => 6379,
  'ZeroMQ_redis_database' => '1',
  'ZeroMQ_redis_namespace' => 'mispq',
  'ZeroMQ_event_notifications_enable' => true,
  'ZeroMQ_object_notifications_enable' => true,
  'ZeroMQ_object_reference_notifications_enable' => true,
  'ZeroMQ_attribute_notifications_enable' => true,
  'ZeroMQ_sighting_notifications_enable' => true,
  'ZeroMQ_user_notifications_enable' => true,
  'ZeroMQ_organisation_notifications_enable' => true,
  'ZeroMQ_tag_notifications_enable' => true,

Monitoring multiple MISP instances

The included sample Grafana dashboard supports showing metrics from different MISP instances, for this its required that the data points coming from each instance have an associated instance tag.

Telegraf

Each instance should have running it’s own Telegraf agent, for each instance set an unique identifier global_tags telegraf.conf as follows:

Internal MISP instance

[global_tags]
  instance = "internal"

External MISP instance

[global_tags]
  instance = "external"

ZeroMQ

For each MISP instance there must be one push_zmq_to_influxdb.py script running, each connected to the corresponding ZeroMQ publisher.

Internal MISP instance

$ python3 src/push_zmq_to_influxdb.py -id=internal --url=tcp://misp.internal:50000
[INFO] [2022-04-04 14:18:24,638] - Subscribed to ZMQ
...

External MISP instance

$ python3 src/push_zmq_to_influxdb.py -id=external --url=tcp://misp.external:50000
[INFO] [2022-04-04 14:18:24,638] - Subscribed to ZMQ
...

InfluxDB v1 compatibility

If you want to add a panel using a InfluxQL query language instead of Flux, you can do so by creating a database and retention policy mapping (DBRP) for InfluxDB v1 compatibility.

cd docker/
$ docker-compose exec influxdb bash
$ influx bucket list --name=misp
ID			Name	Retention	Shard group duration	Organization ID		Schema Type
2123809cf4de9c68	misp	infinite	168h0m0s		b28ccb862d147bdd	implicit
$ influx v1 dbrp create \
  --db misp \
  --rp misp-rp \
  --bucket-id 2123809cf4de9c68 \
  -o org \
  -t tokentokentoken
ID			Database	Bucket ID		Retention Policy	Default	Organization ID
0924213ebf9ba000	misp		2123809cf4de9c68	misp-rp			true	b28ccb862d147bdd

$ influx v1 auth create \
	--read-bucket 2123809cf4de9c68 \
	--write-bucket 2123809cf4de9c68 \
	--username grafana \
  	-o org \
  	-t tokentokentoken
? Please type your password ******** (grafana1)
? Please type your password again ******** (grafana1)
ID			Description	Username	v2 User Name	v2 User ID		Permissions
092421c139dba000			grafana		admin		0923ff89a4587000	[read:orgs/b28ccb862d147bdd/buckets/2123809cf4de9c68 write:orgs/b28ccb862d147bdd/buckets/2123809cf4de9c68]

Create a new datasource in Grafana with the following parameters:

NOTE: For production usage change the sample credentials.

Guide: https://docs.influxdata.com/influxdb/v2.0/tools/grafana/?t=InfluxQL