diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..28aff3f --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2020 Volodymyr Smirnov (volodymyr@smirnov.im) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..ebe130f --- /dev/null +++ b/README.md @@ -0,0 +1,99 @@ +# MalwareMultiScan + +Self-hosted [VirusTotal](https://www.virustotal.com/) wannabe API for scanning URLs and files by multiple antivirus solutions. + +![MalwareMultiScan UI](.github/img/malware-multi-scan-ui.gif) + +[Check out the demo UI](https://malware-multi-scan.lab.smirnov.im) with ClamAV, Dummy and Windows Defender scan backends. The demo is running on a cheap Vultr node, so it might get slow or unavailable occasionally. + +## Introduction + +I faced a need to scan user-uploaded files in one of my work projects in an automated mode to ensure they don't contain any malware. Using VirusTotal was not an option because of a) legal restrictions and data residency limitations b) scanning by hash-sums would not be sufficient because the majority of files are generated / modified by users. + +After googling, I stumbled upon a fantastic [maliceio/malice](https://github.com/maliceio/malice) project. Unfortunately, it [looks abandoned](https://github.com/maliceio/malice/issues/100#issuecomment-529246119), and most plugins do not work for the moment. In addition to that, I had an intention to use the .NET stack to align with the internal infrastructure. + +In the end, it's nothing but the set of Docker containers running the agent. That agent downloads the remote file to the temp folder, launches vendor command-line scanning utility with proper arguments and parses the output with a regular expression to extract a detected malware name. + +## Installation & Usage + +**IMPORTANT**: MalwareMultiScan is not intended as a publicly-facing API / UI. It has no authorization, authentication, rate-limiting and logging (intentionally). Therefore, it should be used only as internal / private API or behind the restrictive API gateway. + +See [components](#components) chapter below and the (docker-compose.yaml)[docker-compose.yaml] file. + +### Configuration + +Configuration of API and Scanners is performed by passing the environment variables. Descriptions and default values are provided below. + +#### MalwareMultiScan.Api + +* `ConnectionStrings__Mongo=mongodb://localhost:27017` - MongoDB connection string. + +* `DatabaseName=MalwareMultiScan` - MongoDB collection name. + +* `ConnectionStrings__RabbitMQ=host=localhost` - RabbitMQ connection string. See [EasyNetQ Wiki](https://github.com/EasyNetQ/EasyNetQ/wiki/Connecting-to-RabbitMQ) for details. + +* `ResultsSubscriptionId=mms.results` - RabbitMQ subscription id for scan results. Should match with values defined for scanners. + +* `MaxFileSize=52428800` - Maximum size of a file that can be handled for the file scanning. The size of the URL content is not verified. + +* `BackendsConfiguration=backends.yaml` - Path to the [backends.yaml](MalwareMultiScan.Api/backends.yaml) file. + +#### MalwareMultiScan.Scanner + +* `ConnectionStrings__RabbitMQ=host=localhost` - RabbitMQ connection string. See [EasyNetQ Wiki](https://github.com/EasyNetQ/EasyNetQ/wiki/Connecting-to-RabbitMQ) for details. + +* `ResultsSubscriptionId=mms.results` - RabbitMQ subscription id for scan results. Should match with values defined for scanners. + +* `BackendType=Dummy` - A type of a scanner backend used by the running instance. Should correspond to the [BackendType](MalwareMultiScan.Backends/Enums/BackendType.cs) enum. + +* `MaxScanningTime=60` - Scan time limit. Is used not just for actual scanning, but also for getting the file. + +#### MalwareMultiScan.Ui + +* `API_URL=http://localhost:5000` - Absolute URL incl. port number for the running instance of MalwareMultiScan.Api. + +### API Endpoints: + +* POST `/api/queue/url` with `url` parameter passed via the form data. Returns `201 Accepted` response with a [ScanResult](MalwareMultiScan.Api/Data/Models/ScanResult.cs) or `400 Bad Request` error. + + * POST `/api/queue/file` with `file` parameter passed via the form data. Returns `201 Accepted` response with a [ScanResult](MalwareMultiScan.Api/Data/Models/ScanResult.cs) or `400 Bad Request` error. + +* GET `/api/results/{result-id}` where `{result-id}` corresponds to the id value of a [ScanResult](MalwareMultiScan.Api/Data/Models/ScanResult.cs). Returns `200 OK` response with a [ScanResult](MalwareMultiScan.Api/Data/Models/ScanResult.cs) or `404 Not Found` error. + +## Supported Scan Engines + +| Name | Dockerfile | Enabled | Comments | +|------|------------|---------|----------| +| [ClamAV](https://www.clamav.net/) | [Clamav.Dockerfile](MalwareMultiScan.Backends/Dockerfiles/Clamav.Dockerfile) | :white_check_mark: | | +| [Comodo](https://www.comodo.com/home/internet-security/antivirus-for-linux.php) | [Comodo.Dockerfile](MalwareMultiScan.Backends/Dockerfiles/Comodo.Dockerfile) | :white_large_square: | | +| [DrWeb](https://download.drweb.com/linux/?lng=en) | [DrWeb.Dockerfile](MalwareMultiScan.Backends/Dockerfiles/DrWeb.Dockerfile) | :white_large_square: | Pass licence key to the DRWEB_KEY build arg. | +| Dummy | [Dockerfile](MalwareMultiScan.Scanner/Dockerfile) | :white_check_mark: | Scan backend made for testing. Returns Malware.Dummy.Result threat for every scan after 5 seconds. | +| [Kaspersky Endpoint Security](https://support.kaspersky.com/kes10linux) | [KES.Dockerfile](MalwareMultiScan.Backends/Dockerfiles/KES.Dockerfile) | :white_large_square: | Pass licence key to the KES_KEY build arg. KES 11 does not work in Docker. | +| [McAfee](https://www.mcafee.com/enterprise/en-us/products/virusscan-enterprise-for-linux.html) | [McAfee.Dockerfile](MalwareMultiScan.Backends/Dockerfiles/McAfee.Dockerfile) | :white_large_square: | | +| [Sophos](https://www.sophos.com/en-us/support/documentation/sophos-anti-virus-for-linux.aspx) | [Sophos.Dockerfile](MalwareMultiScan.Backends/Dockerfiles/Sophos.Dockerfile) | :white_large_square: | | +| [Windows Defender](https://github.com/taviso/loadlibrary#windows-defender) | [WindowsDefender.Dockerfile](MalwareMultiScan.Backends/Dockerfiles/WindowsDefender.Dockerfile) | :white_check_mark: | | + +More scan backends can be added in the future. Some of popular ones do not have command line scanning utility, Linux version, or don't start in Docker container. Feel free to raise an issue if you know any in addition to the list above. + +## Components + +### Prerequisites + +* **MongoDB** of version 3.x. Used for storing scan results and files in GridFS. The communication is happening though the [official C#/.NET driver](https://docs.mongodb.com/drivers/csharp). + +* **RabbitMQ** of version 3.x. Used for IPC and scan tasks queueing. The communication is happening through the [EasyNetQ](https://github.com/EasyNetQ/EasyNetQ) library. + +* **Docker** and **docker-compose** running under Windows (in Linux containers mode), Linux or OSX. Docker Compose is needed only for test / local deployments. + +* **Optional**: DockerSwarm / Kubernetes cluster for scaling up the scanning capacities. At the moment, simultaneous scan by a single node is not supported, yet load-balancing is possible by the built-in orchestrators round-robin routing. + +### Parts + +* [MalwareMultiScan.Api](MalwareMultiScan.Api). Simple ASP.NET Core WebApi for queueing files & urls for the scan and returning the result. Also acts as a receiver of a scan results from the scanning backend nodes. See [Dockerfile](MalwareMultiScan.Api/Dockerfile). Configuration of available backends is performed via the [backends.yaml](MalwareMultiScan.Api/backends.yaml) location passed via the `BackendsConfiguration` environment variable. + +* [MalwareMultiScan.Backends](MalwareMultiScan.Backends). Shared components between API and Worker. Includes Dockerfiles and implementation classes for third-party vendor scan backends. + +* [MalwareMultiScan.Scanner](MalwareMultiScan.Scanner). .NET Core Worker service that subscribes to messages corresponding to the backend ID, fires up scanning command-line utility and parses the output. See [Dockerfile](MalwareMultiScan.Scanner/Dockerfile). The image of MalwareMultiScan.Scanner acts as a basic image for the rest of scan backends. Check Dockerfiles from the table above for details. + +* [MalwareMultiScan.Ui](MalwareMultiScan.Ui). Nuxt.js TypeScript SPA for demoing the API capabilities. See [Dockerfile](MalwareMultiScan.Ui/Dockerfile). + \ No newline at end of file