MalwareMultiScan

Self-hosted VirusTotal wannabe API for scanning URLs and files by multiple antivirus solutions.

Check out the demo UI with ClamAV, Dummy and Windows Defender scan backends. The demo is running on a cheap Vultr node, so it might get slow or unavailable occasionally.

Introduction

I faced a need to scan user-uploaded files in one of my work projects in an automated mode to ensure they don't contain any malware. Using VirusTotal was not an option because of a) legal restrictions and data residency limitations b) scanning by hash-sums would not be sufficient because the majority of files are generated / modified by users.

After googling, I stumbled upon a fantastic maliceio/malice project. Unfortunately, it looks abandoned, and most plugins do not work for the moment. In addition to that, I had an intention to use the .NET stack to align with the internal infrastructure.

In the end, it's nothing but the set of Docker containers running the agent. That agent downloads the remote file to the temp folder, launches vendor command-line scanning utility with proper arguments and parses the output with a regular expression to extract a detected malware name.

Installation & Usage

IMPORTANT: MalwareMultiScan is not intended as a publicly-facing API / UI. It has no authorization, authentication, rate-limiting and logging (intentionally). Therefore, it should be used only as internal / private API or behind the restrictive API gateway.

See components chapter below and the (docker-compose.yaml)[docker-compose.yaml] file.

Configuration

Configuration of API and Scanners is performed by passing the environment variables. Descriptions and default values are provided below.

MalwareMultiScan.Api

ConnectionStrings__Mongo=mongodb://localhost:27017 - MongoDB connection string.
DatabaseName=MalwareMultiScan - MongoDB collection name.
ConnectionStrings__RabbitMQ=host=localhost - RabbitMQ connection string. See EasyNetQ Wiki for details.
ResultsSubscriptionId=mms.results - RabbitMQ subscription id for scan results. Should match with values defined for scanners.
MaxFileSize=52428800 - Maximum size of a file that can be handled for the file scanning. The size of the URL content is not verified.
BackendsConfiguration=backends.yaml - Path to the backends.yaml file.

MalwareMultiScan.Scanner

ConnectionStrings__RabbitMQ=host=localhost - RabbitMQ connection string. See EasyNetQ Wiki for details.
ResultsSubscriptionId=mms.results - RabbitMQ subscription id for scan results. Should match with values defined for scanners.
BackendType=Dummy - A type of a scanner backend used by the running instance. Should correspond to the BackendType enum.
MaxScanningTime=60 - Scan time limit. Is used not just for actual scanning, but also for getting the file.

MalwareMultiScan.Ui

API_URL=http://localhost:5000 - Absolute URL incl. port number for the running instance of MalwareMultiScan.Api.

API Endpoints:

POST /api/queue/url with url parameter passed via the form data. Returns 201 Accepted response with a ScanResult or 400 Bad Request error.
POST /api/queue/file with file parameter passed via the form data. Returns 201 Accepted response with a ScanResult or 400 Bad Request error.
GET /api/results/{result-id} where {result-id} corresponds to the id value of a ScanResult. Returns 200 OK response with a ScanResult or 404 Not Found error.

Supported Scan Engines

Name	Dockerfile	Enabled	Comments
ClamAV	Clamav.Dockerfile	✅
Comodo	Comodo.Dockerfile	⬜
DrWeb	DrWeb.Dockerfile	⬜	Pass licence key to the DRWEB_KEY build arg.
Dummy	Dockerfile	✅	Scan backend made for testing. Returns Malware.Dummy.Result threat for every scan after 5 seconds.
Kaspersky Endpoint Security	KES.Dockerfile	⬜	Pass licence key to the KES_KEY build arg. KES 11 does not work in Docker.
McAfee	McAfee.Dockerfile	⬜
Sophos	Sophos.Dockerfile	⬜
Windows Defender	WindowsDefender.Dockerfile	✅

More scan backends can be added in the future. Some of popular ones do not have command line scanning utility, Linux version, or don't start in Docker container. Feel free to raise an issue if you know any in addition to the list above.

Components

Prerequisites

MongoDB of version 3.x. Used for storing scan results and files in GridFS. The communication is happening though the official C#/.NET driver.
RabbitMQ of version 3.x. Used for IPC and scan tasks queueing. The communication is happening through the EasyNetQ library.
Docker and docker-compose running under Windows (in Linux containers mode), Linux or OSX. Docker Compose is needed only for test / local deployments.
Optional: DockerSwarm / Kubernetes cluster for scaling up the scanning capacities. At the moment, simultaneous scan by a single node is not supported, yet load-balancing is possible by the built-in orchestrators round-robin routing.

Parts

MalwareMultiScan.Api. Simple ASP.NET Core WebApi for queueing files & urls for the scan and returning the result. Also acts as a receiver of a scan results from the scanning backend nodes. See Dockerfile. Configuration of available backends is performed via the backends.yaml location passed via the BackendsConfiguration environment variable.
MalwareMultiScan.Backends. Shared components between API and Worker. Includes Dockerfiles and implementation classes for third-party vendor scan backends.
MalwareMultiScan.Scanner. .NET Core Worker service that subscribes to messages corresponding to the backend ID, fires up scanning command-line utility and parses the output. See Dockerfile. The image of MalwareMultiScan.Scanner acts as a basic image for the rest of scan backends. Check Dockerfiles from the table above for details.
MalwareMultiScan.Ui. Nuxt.js TypeScript SPA for demoing the API capabilities. See Dockerfile.

7.9 KiB Raw Blame History