API Reference

As discussed previously, the BroAPT system has two different implementation architectures. They are similar in overall concepts and processing, but may various in underlying internal source codes. We’ll try to break down into details of each implementation for you to develop new extensions, hooks, scripts for the BroAPT system in humans way.

System Runtime

The whole BroAPT folder in the Docker container (of bundled implementation) at runtime would be like:

# project root
/broapt/
│   # entrypoint wrapper script
├── init.sh
│   # Python source codes
├── python
│   │   # setup PYTHONPATH
│   ├── __init__.py
│   │   # entry point
│   ├── __main__.py
│   │   # config parser
│   ├── cfgparser.py
│   │   # Bro script composer
│   ├── compose.py
│   │   # global constants
│   ├── const.py
│   │   # Bro log parser
│   ├── logparser.py
│   │   # BroAPT-Core logic
│   ├── process.py
│   │   # multiprocessing support
│   ├── remote.py
│   │   # BroAPT-App logic
│   ├── scan.py
│   │   # Python hooks
│   ├── sites
│   │   │   # register hooks
│   │   ├── __init__.py
│   │   └── ...
│   │   # utility functions
│   └── utils.py
│   # Bro source scripts
└── scripts
    │   # load FileExtraction module
    ├── __load__.bro
    │   # configurations
    ├── config.bro
    │   # MIME-extension mappings
    ├── file-extensions.bro
    │   # protocol hooks
    ├── hooks/
    │   │   # extract DTLS
    │   ├── extract-dtls.bro
    │   │   # extract FTP_DATA
    │   ├── extract-ftp.bro
    │   │   # extract HTTP
    │   ├── extract-http.bro
    │   │   # extract IRC_DATA
    │   ├── extract-irc.bro
    │   │   # extract SMTP
    │   └── extract-smtp.bro
    │   # core logic
    ├── main.bro
    │   # MIME hooks
    │── plugins/
    │   │   # extract all files
    │   ├── extract-all-files.bro
    │   │   # extract by BRO_MIME
    │   ├── extract-white-list.bro
    │   │   # generated scripts by BRO_MIME
    │   └── ...
    │   # site functions by user
    └── sites/
        │   # load site functions
        ├── __load__.bro
        └── ...

where /broapt/python/sites is the path for custom Python hooks and /broapt/scripts/sites/ is the path for custom Bro scripts.

And most importantly, the very entrypoint for the whole BroAPT system is as following:

#!/usr/bin/env bash

set -aex

# change curdir
cd /broapt

# load environs
if [ -f .env ] ; then
    source .env
fi

# compose Bro scripts
/usr/bin/python3.6 python/compose.py

# run scripts
/usr/bin/python3.6 python $@

# sleep
sleep infinity
  1. The script will first change the current working directory to the root path /broapt/.

  2. If there is a .env dotenv file for environment variables configuration, it will be loaded and saved into current runtime scope (set -a).

  3. Generate Bro scripts based on environment variables.

  4. Start the main application, i.e. BroAPT-Core and BroAPT-App frameworks.

Developer Notes

Since the BroAPT system was not intended for packaging and distribution, we didn’t provide a setup.py to wrap everything as a broapt module. However, in a quite hacky way, we injected the sys.path import path, so that we can directly import the files as if they’re at top levels.

As you can see in the /broapt/python/sites/__init__.py, i.e. the module entry of Python hooks is as following:

# -*- coding: utf-8 -*-
# pylint: disable=all

###############################################################################
# site customisation
import os
import sys

sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
###############################################################################

from extracted_files import generate_log as info_log
from http_parser import generate as http_log, close as http_log_exit

# log analysis hook list
HOOK = [
   http_log,
   info_log,
]

# exit hooks
EXIT = [
   http_log_exit,
]

where extracted_files refers to /broapt/python/sites/extracted_files.py and http_parser refers to /broapt/python/sites/http_parser.py.

You may have noticed the lines in site customisation modified the sys.path import path so that we don’t need to worry about importing stuff from the BroAPT Python source codes.

If you wish to use auxiliary functions and module constants from the main application, then you can still import them as if from the top level:

# path to logs from module constants
from const import LOGS_PATH
# Bro log parsing utilities
from logparser import parse
# auxiliary functions for BroAPT
from utils import is_nan, print_file