malshare_db (Version 0.1)

MalShare to ClamAV converter

Usage: malshare_db.py [--help|--version|--cgi|--fcgi[-server]|--wsgi(ref|aio)|--offline]

Command line arguments:
  --help   Show this message and exit.
  --version Show script version and installed capabilities.

  --cgi   Run this script as CGI script.
  --fcgi   Run this script as FastCGI script (for e.g. FastCGI spawn).
  --fcgi-server Run this script as FastCGI server on 127.0.0.1:9000 (see WSGI_HOST and WSGI_PORT).
  --wsgiref Start this script on the WSGI reference server on 127.0.0.1:8000 (see WSGI_HOST and WSGI_PORT).
  --wsgiaio Start this script on the aiohttp WSGI adapter on 127.0.0.1:8000 (see WSGI_HOST and WSGI_PORT).

  --offline (Default) Update "MalShare.hdb" in the current directory.


For --fcgi and --fcgi-server one of the packages flup (Python 2 and Python 3 in dev version), flup6 (Python >= 3), gevent-fastcgi (Python 2) or flipflop (also Python >= 3) is required. The package flipflop does not support --fcgi-server.

The --wsgiref option also validates the script if it is conforming to the WSGI standart. Do not use --wsgiref in production, use a proper WSGI server. While it has a decent speed, it can exhaust resources quite fast and might have security vulnerabilities. USE --wsgiref FOR TESTING ONLY!

Using the --wsgiaio flag requires the packages aiohttp and aiohttp_wsgi to be installed.


Environment variables:
  DEBUG      Enables debug output to stderr

  WSGI_HOST=host Defines the hostname for servers (HTTP or FCGI).
    Default: 127.0.0.1
  WSGI_PORT=port Defines the port for servers (HTTP or FCGI).
    Default: 8000 for HTTP; 9000 for FastCGI

  WSGI_PATH_STRIP=path  Strips path from the beginning of a request path. This is a hack as some webservers do not allow stripping the beginning of a path. Implementation detail: Additional endpoints starting with path are added.

  WSGI_FCGI_LIB=lib  Sets the FCGI library to use (default: choose automatically). Possible values: flup, flup6, flipflop, gevent-fastcgi. flup and flup6 are equivalent - both packages share the same namespace.

Optimizations:
  - Install requests-cache. This will enable caching the MalShare-current.* files. The cache is on a per-process-basis for security reasons, so the cache must be initialized for each process in multiprocess deployments. There will be no benefits if the application is only executed once for each request (e.g. CGI) as the cache is not shared. If security problems are solved within requests-cache a shared cache might be readded.

  - Use an external WSGI server for deployment. uWSGI and gunicorn both seem to be a good choice. This script offers the common entrypoints for WSGI servers 'app' and 'application'. If you use cherrypy malshare_db can be simply integrated in your application.
For aiohttp deployments 'aioapp' is defined. See below for examples.

Examples:
  Deployment without external server:
    PATH_STRIP=/malshare WSGI_PORT=1234 malshare_db.py --wsgiref
    - Start this script on the WSGIref server on http://127.0.0.1:1234
    - Valid requests:
      http://127.0.0.1:1234/MalShare.hdb
      http://127.0.0.1:1234/MalShare-current.hdb
      http://127.0.0.1:1234/MalShare-current.hsb
      http://127.0.0.1:1234/malshare/MalShare.hdb
      ...

  Deployment with externel server (e.g. uWSGI, Gunicorn):
    uwsgi --http-socket 127.0.0.1:1234 malshare_db.py
    - Start this script on uWSGI on http://127.0.0.1:1234

    gunicorn -k aiohttp.GunicornWebWorker -b 127.0.0.1:1234 malshare_db:aioapp
    - Start this script on Gunicorn with aiohttp on http://127.0.0.1:1234

Resources:
- malshare.com (https://malshare.com)
- gunicorn 'Green unicorn' (http://gunicorn.org)
- uWSGI (https://uwsgi-docs.readthedocs.io)
- CherryPy - Host a foreign WSGI application in CherryPy (http://docs.cherrypy.org/en/latest/advanced.html#host-a-foreign-wsgi-application-in-cherrypy)
- WSGI standart PEP-3333 (https://www.python.org/dev/peps/pep-3333/)

Imported modules

__future__

aiohttp

aiohttp.web

aiohttp_wsgi

aiohttp_wsgi.wsgi

asyncio

datetime

itertools

logging

multiprocessing

multiprocessing.context

os

portalocker

requests

requests_cache

requests_cache.core

sys

traceback

wsgiref

wsgiref.handlers

wsgiref.simple_server

wsgiref.validate

Functions

app,application(environ, start_response, strip='')

WSGI application object for distributing MalShare's database.

Endpoints:
  /MalShare.hdb
  /MalShare-current.hdb
  /MalShare-current.hsb

environ (dict): WSGI environment

start_response (callable(status, headers)):
  Starts the WSGI response.

strip (string): Strips this at the start of every requests' path.
  Defaults os.environ["WSGI_PATH_STRIP"] or "".

Returns:
  Iterable returning bytes.

generator function daterange(start, stop)

Creates a generator which returns dates from start to stop.

fastcgi(wsgi_application, spawn=False, address=('127.0.0.1', 9000), use=None)

Creates an FastCGI server for wsgi_application.

hash_to_db(lines, tag, suffix, template='{sig}:*:MalShare.{tag}-{n}{suffix}:73')

Converts an iterable of strings into a simple ClamAV database.

The created signatures are in the following form by default:
  {sig}:*:MalShare.{tag}-{n}{suffix}:73

lines (list of strings): List of the signatures
tag (string): How to "name" all signatures.
suffix (string): Second "name" for the signatures.
template (string): formatable string for the conversion

malshare_by_date(pub_date=None, suffix='')

Download a single day's MalShare hashes and convert them.

pub_data (date object): The date.
suffix (string): A suffix to add to the signatures name.
  Usefull to indicate a different algorithm.

malshare_by_dates(start=None, stop=None, suffix='', silent=False)

Downloads MalShare database from start to stop.

start (date or None): The start date. None equals today - 1 day.
stop (date or None): The stop date. None equals today - 15 days,
suffix (string): A suffix to add to the signatures name.    Usefull to indicate a different algorithm.
silent (boolean): Silence any errors (skip days with errors).

malshare_current(suffix='', forward=None)

Download today's MalShare signatures

malshare_update(filename, suffix='')

Update a ClamAV database with MalShare signatures.

validated_app(*args, **kw)

'application' wrapped with wsgiref.validate.validator