Bulk delete MISP events¶

Introduction¶

  • UUID: 7d3fd7fe-fe4b-48a3-8e7e-8bd4e4bf2f0d
  • Started from issue 29
  • State: Published
  • Purpose: A playbook to assist MISP users in doing bulk deletes of MISP events.
    • Deletes are done for events created by organisations, for events before or after specific dates, published or unpublished events or for events with specific tags.
    • A summary of the actions is printed and published on Mattermost.
  • Tags: [ "delete", "clean", "curation"]
  • External resources: Mattermost
  • Target audience: CTI

Playbook¶

  • Bulk delete MISP events
    • Introduction
  • Preparation
    • PR:1 Initialise environment
    • PR:2 Load helper functions
  • Identify events
    • SR:1 Execution flow
    • SR:2 Search options
    • SR:3 Remove from blocklist?
  • Delete events
    • DL:1 Delete events created by an organisation
    • DL:2 Delete events created or changed after or before a specific date
    • DL:3 Delete published or not published events
    • DL:4 Delete events with (or without) specific tags
  • Closure
    • EN:1 Create the summary of the playbook
    • EN:2 Send a summary to Mattermost
    • EN:3 End of the playbook
  • External references
  • Technical details

Preparation¶

PR:1 Initialise environment¶

This section initialises the playbook environment and loads the required Python libraries.

The credentials for MISP (API key) and other services are loaded from the file keys.py in the directory vault. A PyMISP object is created to interact with MISP and the active MISP server is displayed. By printing out the server name you know that it's possible to connect to MISP. In case of a problem PyMISP will indicate the error with PyMISPError: Unable to connect to MISP.

The contents of the keys.py file should contain at least :

misp_url="<MISP URL>"                  # The URL to our MISP server
misp_key="<MISP API KEY>"              # The MISP API key
misp_verifycert=<True or False>        # Ignore certificate errors
mattermost_playbook_user="<MATTERMOST USER>"
mattermost_hook="<MATTERMOST WEBHOOK>"
In [ ]:
# Initialise Python environment
import urllib3
import sys
import json
from pyfaup.faup import Faup
from prettytable import PrettyTable, MARKDOWN
from IPython.display import Image, display, display_markdown, HTML
from datetime import date
import requests
import uuid
from uuid import uuid4
from pymisp import *
from pymisp.tools import GenericObjectGenerator
import re
import time
from datetime import datetime

# Load the credentials
sys.path.insert(0, "../vault/")
from keys import *
if misp_verifycert is False:
    import urllib3
    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
print("The \033[92mPython libraries\033[90m are loaded and the \033[92mcredentials\033[90m are read from the keys file.")

# Create the PyMISP object
misp = PyMISP(misp_url, misp_key, misp_verifycert)
print("I will use the MISP server \033[92m{}\033[90m for this playbook.\n\n".format(misp_url))

PR:2 Load helper functions¶

The next cell contains helper functions that are used in this playbook.

Instead of distributing helper functions as separate Python files this playbook includes all the required code as one code cell. This makes portability of playbooks between instances easier. The downside is that functions defined in this playbook need to be defined again in other playbooks, which is not optimal for code re-use. For this iteration of playbooks it is chosen to include the code in the playbook (more portability), but you can easily create one "helper" file that contains all the helper code and then import that file in each playbook (for example by adding to the previous cell from helpers import *). Note that the graphical workflow image is included as an external image. A missing image would not influence the further progress of the playbook.

In [ ]:
def table_print_before_delete(event_list, return_str=False):
    '''
    Table to print with the events from event_list that are about to be deleted
    '''    
    delete_events_table = PrettyTable()
    delete_events_table.field_names = ["List ID", "Event ID", "Org", "Info", "Date", "Published"]
    delete_events_table.align["ID"] = "l"
    delete_events_table.align["Org"] = "l"
    delete_events_table.align["Info"] = "l"    
    delete_events_table._max_width = {"Org": 30, "Info": 30}
    position = 0
    for event in event_list:
        delete_events_table.add_row([position, event.id, event.orgc.name, event.info, event.date, event.published])
        position += 1
    if not return_str:
        print(delete_events_table.get_string(sortby="List ID"))
    else:
        return delete_events_table



def delete_events(event_list):
    '''
    Do the delete of the events in event_list
    '''    
    print("Starting with delete ...")
    for event in event_list:
        print(" Delete \033[91m{}\033[90m (ID: {}, UUID: {})".format(event.info, event.id, event.uuid))
        result = misp.delete_event(event.uuid)
        if not "errors" in result:
            if remove_from_event_block_list:
                result = misp.delete_event_blocklist(event.uuid)
    print("Delete finished")

Identify events¶

SR:1 Execution flow¶

The next sections use a similar approach:

  1. First search for the events;
  2. List and review the results;
  3. Do the actual delete.

SR:2 Search options¶

First search (or identify) the events that you'd like to delete. The following sections provide different ways to search for these events. You can

  • Delete events created by an organisation
  • Delete events before or after a specific date
  • Delete published or not-published events
  • Delete events with or without specific tags

SR:3 Remove from blocklist?¶

When an event is deleted in MISP, it's unique identifier (the UUID) is put on the MISP event blocklist. Blocklisting an event prevents the event from being added again on the instance, for example if you synchronise with external communities or import events from external sources. It's also a feature often overlooked when setting up synchronised MISP servers. During the configuration phase it can confuse administrators that some events are added, and others are skipped for no apparent reason. As a rule of thumb, if you want to delete the event, and never receive an update again, then put it on the blocklist. If you are configuring and testing the synchronisation process then remove the event from the blocklist after deleting it.

In this playbook you can control this behaviour with remove_from_event_block_list.

In [ ]:
# After deleting the event, also remove it from the blocklist
remove_from_event_block_list = True

Delete events¶

DL:1 Delete events created by an organisation¶

This section allows you to delete events that are created by a specific organisation.

Search¶

You can search for the organisation based on their name, organisation ID or UUID. The playbook attempts to guess the type of input you provided in the variable org.

In [ ]:
# Provide an organisation name, ID or UUID
org = ""


# Code block to search for events
org_id = False
if isinstance(org, int):
    result = misp.get_organisation(org, pythonify=True)
    if not "errors" in result:
        org_id = result.id
elif isinstance(org, str):
    try:
        uuid_obj = uuid.UUID(org, version=4)
        result = misp.get_organisation(str(org), pythonify=True)
        if not "errors" in result:
            org_id = result.id
    except ValueError:  
        organisations = misp.organisations(scope="all", pythonify=True)
        for org_search in organisations:
            if org_search.name.lower().strip() == org.lower().strip():
                org_id = org_search.id
                break

if org_id:
    org_delete_events = misp.get_organisation(org_id, pythonify=True)
    print("Found organsation: \033[92m{}\033[90m (ID: {}, UUID: {})".format(org_delete_events.name, org_delete_events.id, org_delete_events.uuid))
    print("Searching for events ...")
    event_list = misp.search("events", org=org_delete_events.id, pythonify=True)
    if not "errors" in event_list and len(event_list) > 0:
        print(" \033[92m{}\033[90m events found.".format(len(event_list)))
    else:
        print(" No \033[91mevents\033[90m found for this organisation.")
else:
    print("Unable to find organisation with identifier \033[91m{}\033[90m".format(org))

List and review¶

Print the list of events stored in event_list.

Remove events from the list with pop. For example event_list.pop(2) removes the event with List ID two (we start counting at zero, so it's actually the third element) from the event list. Do not confuse List ID with the Event ID.

You can execute these cells multiple times to review and edit the results, before doing the actual delete.

In [ ]:
# Print out the list of events before deleting them
table_print_before_delete(event_list)
In [ ]:
# Example on how to remove one element from the list
#print(event_list.pop(1))
print("There are \033[92m{}\033[90m events found.".format(len(event_list)))

Delete¶

The next cell deletes the events from your MISP instance.

In [ ]:
delete_events(event_list)

DL:2 Delete events created or changed after or before a specific date¶

This section allows you to delete events created, changed or published before or after a specific date.

Search¶

There are multiple ways to identify events based on their date. If you are not familiar with the different dates and timestamps that are part of MISP events then it's best to check out the Using timestamps in MISP playbook before continuing.

  • You can search for recently published events with the publish_timestamp.
    • Use publish_timestamp="10d" (publish timestamp of 10d or less) for a specific date or publish_timestamp=["3d","10d"] (publish timestamp between 3 and 10 days ago) for a time range.
  • Search for recently changed events with timestamp.
    • The syntax is similar as with published events. Both timestamp="3d" and timestamp=["3d","10d"] are valid.
  • Use the event date field with date_from and date_to. This field expects a YYYY-MM-DD format.
In [ ]:
print("Searching for events ...")
event_list = misp.search("events", date_from="2020-12-13", date_to="2020-12-14", pythonify=True)
#event_list = misp.search("events", timestamp="3d", pythonify=True)
#event_list = misp.search("events", publish_timestamp=["3d","10d"], pythonify=True)

# Search for the events
if not "errors" in event_list and len(event_list) > 0:
    print(" \033[92m{}\033[90m events found.".format(len(event_list)))
else:
    print(" No \033[91mevents\033[90m found for this organisation.")

List and review¶

Print the list of events stored in event_list.

Remove events from the list with pop. For example event_list.pop(2) removes the event with List ID two (we start counting at zero, so it's actually the third element) from the event list. Do not confuse List ID with the Event ID.

You can execute these cells multiple times to review and edit the results, before doing the actual delete.

In [ ]:
# Print out the list of events before deleting them
table_print_before_delete(event_list)
In [ ]:
# Example on how to remove one element from the list
#print(event_list.pop(0))
print("There are \033[92m{}\033[90m events found.".format(len(event_list)))

Delete¶

The next cell deletes the events from your MISP instance.

In [ ]:
delete_events(event_list)

DL:3 Delete published or not published events¶

Search¶

You can combine the previous activities by limiting the search to published or not published events. If you ommit the published option from the search query (as is the case for the previous cells), then the playbook ignores the published state of an event.

  • published=0 only events that are not published
  • published=1 only events that are published
  • Do not add the published option in the search if you want to have both published and not published events.

List and review¶

In this section there's no list or review activity. Scroll back to the previous cells and add the published option to the search query. You can also use the published state in the next section as well.

As an example, the query event_list = misp.search("events", date_from="2023-07-04", date_to="2023-07-06", published=1, pythonify=True) returns the published events between 4 Jul and 6 Jul.

DL:4 Delete events with (or without) specific tags¶

Tags in MISP allow you to specify contextual information. You can also use them to identify events you'd like to delete.

Search¶

In the next example we search for events that are not tlp:red (TLP), have the workflow state incomplete and require a review for false positive. The search uses the build_complex_query feature of PyMISP to combine logical ORs, ANds and NOTs.

In [ ]:
or_parameters = [ ]
and_parameters = [ "workflow:state=\"incomplete\"", "workflow:todo=\"review-for-false-positive\"" ]
not_parameters = [ "tlp:red"]

tags = misp.build_complex_query(or_parameters=or_parameters, and_parameters=and_parameters, not_parameters=not_parameters)

print("Searching for events ...")
event_list = misp.search("events", tags=tags, pythonify=True)
# Search for the events
if not "errors" in event_list and len(event_list) > 0:
    print(" \033[92m{}\033[90m events found.".format(len(event_list)))
else:
    print(" No \033[91mevents\033[90m found for this organisation.")

List and review¶

Print the list of events stored in event_list.

Remove events from the list with pop. For example event_list.pop(2) removes the event with List ID two (we start counting at zero, so it's actually the third element) from the event list. Do not confuse List ID with the Event ID.

You can execute these cells multiple times to review and edit the results, before doing the actual delete.

In [ ]:
# Print out the list of events before deleting them
table_print_before_delete(event_list)
In [ ]:
# Example on how to remove one element from the list
#print(event_list.pop(1))
print("There are \033[92m{}\033[90m events found.".format(len(event_list)))

Delete¶

The next cell deletes the events from your MISP instance.

In [ ]:
delete_events(event_list)

Closure¶

In this closure or end step we create a summary of the actions that were performed by the playbook. The summary is printed and can also be send to a chat channel.

EN:1 Create the summary of the playbook¶

The next section creates a summary and stores the output in the variable summary in Markdown format. It also stores an intro text in the variable intro. These variables can later be used when sending information to Mattermost or TheHive.

In [ ]:
summary = "# MISP Playbook summary\nBulk delete events \n\n"

current_date = datetime.now()
formatted_date = current_date.strftime("%Y-%m-%d")
summary += "## Action\n\n"
summary += "- Date: **{}**\n".format(formatted_date)
summary += "- Events affected: **{}**\n".format(len(event_list))
if remove_from_event_block_list:
    summary += "- Remove events from **blocklist** after delete.\n"
else:
    summary += "- Keep events in the **blocklist** after delete.\n"
summary += "\n\n"
summary += "## Events\n\n"
summary_table = table_print_before_delete(event_list, True)
summary_table.set_style(MARKDOWN)
summary += summary_table.get_string(sortby="List ID")
summary += "\n\n"

print("The \033[92msummary\033[90m of the playbook is available.\n")

EN:2 Send a summary to Mattermost¶

Now you can send the summary to Mattermost. You can send the summary in two ways by selecting one of the options for the variable send_to_mattermost_option in the next cell.

  • The default option where the entire summary is in the chat, or
  • a short intro and the summary in a card

For this playbook we rely on a webhook in Mattermost. You can add a webhook by choosing the gear icon in Mattermost, then choose Integrations and then Incoming Webhooks. Set a channel for the webhook and lock the webhook to this channel with "Lock to this channel".

In [ ]:
send_to_mattermost_option = "via a chat message"
#send_to_mattermost_option = "via a chat message with card"
In [ ]:
message = False
if send_to_mattermost_option == "via a chat message":
    message = {"username": mattermost_playbook_user, "text": summary}
elif send_to_mattermost_option == "via a chat message with card":
    message = {"username": mattermost_playbook_user, "text": intro, "props": {"card": summary}}

if message:
    r = requests.post(mattermost_hook, data=json.dumps(message))
    r.raise_for_status()
if message and r.status_code == 200:
    print("Summary is \033[92msent to Mattermost.\n")
else:
    print("\033[91mFailed to sent summary\033[90m to Mattermost.\n")

EN:3 End of the playbook¶

In [ ]:
print("\033[92m End of the playbook")

External references ¶

  • The MISP Project
  • Mattermost

Technical details¶

Documentation¶

This playbook requires these Python libraries to exist in the environment where the playbook is executed. You can install them with pip install <library>.

pyfaup
chardet
PrettyTable
ipywidgets

Colour codes¶

The output from Python displays some text in different colours. These are the colour codes

Red = '\033[91m'
Green = '\033[92m'
Blue = '\033[94m'
Cyan = '\033[96m'
White = '\033[97m'
Yellow = '\033[93m'
Magenta = '\033[95m'
Grey = '\033[90m'
Black = '\033[90m'
Default = '\033[99m'
In [ ]: