====================
BroAPT-App Framework
====================
The BroAPT-App framework is the analysis framework for the BroAPT
system. For more information about the framework, please refer to previous
documentation at :doc:`broapt-app`.
Python Modules
==============
.. toctree::
:maxdepth: 4
api.app.python.__init__
api.app.python.__main__
api.app.python.cfgparser
api.app.python.const
api.app.python.process
api.app.python.remote
api.app.python.utils
API Configurations
==================
:File location:
* Bundled implementation: ``source/include/api/``
* Cluster implementation: ``cluster/app/include/api/``
As discussed in previous documentation, we provided a YAML configuration file
``api.yml`` for registering MIME type specific detection methods.
For example, following is the requirements of an API for analysing PDF files
(MIME type: ``application/pdf``):
- Root: ``/api/``
- Target:
- MIME type: ``application/pdf``
- file name: ``/dump/application/pdf/test.pdf``
- API:
- working directory: ``./pdf_analysis``
- environment: ``ENV_FOO=1``, ``ENV_BAR=this is an environment variable``
The configuration section should then be:
.. code:: yaml
application:
... # other APIs
pdf:
remote: false
workdir: pdf_analysis
environ:
ENV_FOO: 1
ENV_BAR: this is an environment variable
install:
- apt-get update
- apt-get install -y python python-pip
- python -m pip install -r requirements.txt
- rm -rf /var/lib/apt/lists/*
- apt-get remove -y --auto-remove python-pip
- apt-get clean
scripts:
- ${PYTHON27} detect.py [...] # refer to /usr/bin/python
- ... # and some random command
report: ${PYTHON27} report.py # generate final report
.. important::
``report`` section is **MANDATORY**.
If ``remote`` is ``true``, then the BroAPT-APP framework will run the
corresponding API in the host machine through the BroAPT-Daemon server.
The BroAPT-App framework will work as following:
1. set the following environment variables:
* per target file
- ``BROAPT_PATH="/dump/application/pdf/test.pdf"``
- ``BROAPT_MIME="application/pdf"``
* per API configuration
- ``ENV_FOO=1``
- ``ENV_BAR="this is an environment variable"``
2. change the current working directory to
``/api/application/pdf/pdf_analysis``
3. if run for the first time, run the following commands:
- ``apt-get update``
- ``apt-get install -y python python-pip``
- ``python -m pip install -r requirements.txt``
- ``rm -rf /var/lib/apt/lists/*``
- ``apt-get remove -y --auto-remove python-pip``
- ``apt-get clean``
4. run the following mid-stage commands:
- ``/usr/bin/python detect.py [...]``
- ...
5. generate final report:
``/usr/bin/python report.py``
.. note::
The registered MIME types support *shell*-like patterns.
If the API of a specific MIME type is not provided, it will then fallback
to the API configuration registered under the special ``example`` MIME type.
.. raw:: html
Content of api.yml
(bundled implementation)
.. literalinclude:: ../../../source/include/api/api.yml
:language: yaml
.. raw:: html
.. raw:: html
Content of api.yml
(cluster implementation)
.. literalinclude:: ../../../cluster/app/include/api/api.yml
:language: yaml
.. raw:: html
.. caution::
For bundled implementation, the runtime of *local* APIs are in the CentOS 7
Docker container.
For cluster implementation, the runtime of *local* APIs are in the Ubuntu 16.04
Docker container.
Wrapper Scripts
===============
For the Docker container, we have created some Shell/Bash wrapper scripts to
make the life a little bit better.
----------------------
Bundled Implementation
----------------------
:File location: ``source/client/init.sh``
As the BroAPT-App framework is already integrated into the source codes, there's
no need to another wrapper script to start the BroAPT-App framework. It shall be
run directly after the BroAPT-Core framework.
.. literalinclude:: ../../../source/client/init.sh
:language: shell
----------------------
Cluster Implementation
----------------------
:File location: ``cluster/app/source/init.sh``
.. literalinclude:: ../../../cluster/app/source/init.sh
:language: shell