Merge pull request #259 from rackerlabs/scan-diff-email-alerts

Scan diff email alerts

README.md

@@ -10,12 +10,12 @@
 ## Overview
 
 Scantron is a distributed nmap and masscan scanner comprised of two components.  The first is a console node that
-consists of a web front end used for scheduling scans and storing nmap scan targets and results.  The second component
-is an engine that pulls scan jobs from the console and conducts the actual nmap scanning.  A majority of the
-application's logic is purposely placed on the console to make the engine(s) as "dumb" as possible.  All nmap target
-files and nmap results reside on the console and are shared through a network file share (NFS) leveraging SSH tunnels.
-The engines call back to the console periodically using a REST API to check for scan tasks and provide scan status
-updates.
+consists of a web front end used for scheduling scans and storing scan targets and results.  The second component is an
+engine that pulls scan jobs from the console and conducts the actual scanning.  A majority of the application's logic is
+purposely placed on the console to make the engine(s) as "dumb" as possible.  All scan target files and scan results
+reside on the console and are shared through a network file share (NFS) leveraging SSH tunnels.  The engines call back
+to the console periodically using a REST API to check for scan tasks and provide scan status updates.  There is also an
+option to generate nmap scan diffs emailed to you using the [pyndiff](https://github.com/rackerlabs/pyndiff) library.
 
 Checkout the Python [Scantron API client](https://github.com/rackerlabs/scantron/tree/master/scantron_api_client) for
 interacting with the Scantron API and driving automated workflows.
@@ -47,8 +47,9 @@ concepts, there are some great overviews and tutorials out there:
 
 Scantron is not engineered to be quickly deployed to a server to scan for a few minutes, then torn down and destroyed.  
 It's better suited for having a set of static scanners (e.g., "internal-scanner", "external-scanner") with a relatively
-static set of assets to scan.  A [Scantron API client](https://github.com/rackerlabs/scantron/tree/master/scantron_api_client)
-is also available for creating, retrieving, updating, or deleting sites, scan commands, scans, etc.
+static set of assets to scan.
+A [Scantron API client](https://github.com/rackerlabs/scantron/tree/master/scantron_api_client) is also available for
+creating, retrieving, updating, or deleting sites, scan commands, scans, etc.
 
 ## Architecture Diagram
 
@@ -111,7 +112,7 @@ to
 
 Edit the hosts in this file:
 
-* `ansible-playbooks/hosts`
+* `ansible-playbooks/hosts.ini`
 
 ### Console Installation
 
@@ -269,7 +270,7 @@ su - autossh -s /bin/bash -c 'autossh -M 0 -f -N -o "StrictHostKeyChecking no" -
 
 ### Engine's engine_config.json
 
-engine_config.json is a configuration file used by engines to provide basic settings and bootstrap communication with
+`engine_config.json` is a configuration file used by engines to provide basic settings and bootstrap communication with
 the console.  Each engine can have a different configuration file.  
 
 ```none
@@ -318,8 +319,8 @@ This repo also contains a stand-alone binary `engine/engine` that can be used fo
 allows for a quicker deployment if managing the Python environment is difficult or cumbersome.  The basic requirements
 are:
 
-* nmap and masscan must exist on the system
-* the `engine_config.json` file exists
+* `nmap` and `masscan` must exist on the system
+* the `engine_config.json` file exists and the `scan_engine` and `api_token` values have been updated
 * An SSH tunnel to/from the console still exists to read target files and write scan results
 
 #### Creating the standalone binary
@@ -343,7 +344,7 @@ rm -rf __pycache__ build dist engine.spec .venv
 
 ### Engine Execution
 
-Update all the engines' engine_config.json files with their respective `api_token` for the engine by logging in as
+Update all the engines' `engine_config.json` files with their respective `api_token` for the engine by logging in as
 `admin` and browsing to `https://<HOST>/scantron-admin/authtoken/token` to see the corresponding API token for each
 user / engine.
 
@@ -380,7 +381,10 @@ screen -S engine1  # Create a screen session and name it engine1, if using scree
 
 cd engine
 source .venv/bin/activate
+# Option 1: Python virtual environment
 python engine.py -c engine_config.json
+# Option 2: Stand alone binary
+./engine -c engine_config.json
 
 CTRL + a + d  # Break out of screen session, if using screen.
 screen -ls  # View screen job, if using screen.
@@ -407,8 +411,8 @@ crontab -l -u root
 
 ### Test Engine API
 
-If you need to test the API without running the engine, ensure there is a 'pending' scan set to start earlier than the
-current date and time.  The server only returns scan jobs that have a 'pending' status and start datetime earlier than
+If you need to test the API without running the engine, ensure there is a "pending" scan set to start earlier than the
+current date and time.  The server only returns scan jobs that have a "pending" status and start datetime earlier than
 the current datetime.
 
 ```bash
@@ -532,11 +536,10 @@ Source: <https://security.stackexchange.com/questions/78618/is-there-a-nmap-comm
 
 ## nmap_port_range_carver
 
-A standalone [script](https://github.com/rackerlabs/scantron/tree/master/nmap_port_range_carver)
- to carve out a range of the top TCP/UDP ports according to the nmap-services file.
+A standalone [script](https://github.com/rackerlabs/scantron/tree/master/nmap_port_range_carver) to carve out a range
+of the top TCP/UDP ports according to the nmap-services file.
 
-This is useful
-when:
+This is useful when:
 
 1. You want to scan a subset of the ports specified in `--top-ports`, say the 10th through 20th top TCP ports, but not
 the 1st or 9th ports.
@@ -573,6 +576,7 @@ the 1st or 9th ports.
     ![create_site](./img/create_site.png)
 
 4. Create scan
+
     * Select start time
     * Add start date
     * Add recurrence rules (if applicable)
@@ -601,9 +605,6 @@ the 1st or 9th ports.
     `/home/scantron/console/scan_results/complete` - Completed scan files from engines are stored here before being
     processed by `nmap_to_csv.py`
 
-    The `scantron` user executes a cron job (`nmap_to_csv.sh` which calls `nmap_to_csv.py`) every 5 minutes that will
-    process the `.xml` scan results found in the `complete` directory and move them to the `processed` directory.
-
     `/home/scantron/console/scan_results/processed` - nmap scan files already processed by `nmap_to_csv.py` reside here.
 
     `/home/scantron/console/for_bigdata_analytics` - .csv files for big data analytics ingestion if applicable

ansible-playbooks/ansible.cfg

@@ -2,4 +2,4 @@
 display_args_to_stdout = True
 stdout_callback = debug
 host_key_checking = False
-inventory = ./hosts
+inventory = ./hosts.ini

ansible-playbooks/roles/console/vars/main.yml

@@ -43,7 +43,7 @@ venv_python: "{{ venv_dir }}/bin/python3.6"
 django_project_name: django_scantron
 
 # uwsgi
-uwsgi_version: 2.0.18
+uwsgi_version: 2.0.19.1
 
 # postfix
 postfix_hostname: scantron

console/django_scantron/__init__.py

@@ -1 +1 @@
-__version__ = "1.42"
+__version__ = "1.43"

console/django_scantron/admin.py

@@ -59,6 +59,8 @@ class SiteAdmin(admin.ModelAdmin):
         "scan_engine_pool",
         "email_scan_alerts",
         "email_alert_addresses",
+        "email_scan_diff",
+        "email_scan_diff_addresses",
     )
 
 

console/django_scantron/api/serializers.py

@@ -120,6 +120,24 @@ def validate(self, attrs):
                 email_alert_addresses
             )
 
+        # Email nmap_scan diff and email addresses.
+        if ("email_scan_diff" in attrs) and ("email_scan_diff_addresses" in attrs):
+
+            email_scan_diff = attrs["email_scan_diff"]
+            email_scan_diff_addresses = attrs["email_scan_diff_addresses"]
+
+            if email_scan_diff and not email_scan_diff_addresses:
+                raise serializers.ValidationError(f"Provide an email address if enabling 'Email nmap scan diff'")
+
+        # Check for valid email addresseses string.
+        if "email_scan_diff_addresses" in attrs:
+            """Checks that email addresses are valid and returns a cleaned up string of them to save to the database."""
+
+            email_scan_diff_addresses = attrs["email_scan_diff_addresses"]
+            attrs["email_scan_diff_addresses"] = email_validation_utils.validate_string_of_email_addresses(
+                email_scan_diff_addresses
+            )
+
         return attrs
 
     class Meta:
@@ -135,6 +153,8 @@ class Meta:
             "scan_engine_pool",
             "email_scan_alerts",
             "email_alert_addresses",
+            "email_scan_diff",
+            "email_scan_diff_addresses",
         )
 
 

console/django_scantron/models.py

@@ -57,7 +57,7 @@ class EnginePool(models.Model):
 
     id = models.AutoField(primary_key=True, verbose_name="Engine Pool ID")
     engine_pool_name = models.CharField(unique=True, max_length=255, verbose_name="Engine Pool Name")
-    scan_engines = models.ManyToManyField(Engine, verbose_name="Scan engines in pool",)
+    scan_engines = models.ManyToManyField(Engine, verbose_name="Scan engines in pool")
 
     def __str__(self):
         return str(self.engine_pool_name)
@@ -185,6 +185,12 @@ class Site(models.Model):
     email_alert_addresses = models.CharField(
         unique=False, blank=True, max_length=4096, verbose_name="Email alert addresses, comma separated"
     )
+    email_scan_diff = models.BooleanField(
+        verbose_name="Email nmap scan diff after each scan? (Only applies to nmap scans)"
+    )
+    email_scan_diff_addresses = models.CharField(
+        unique=False, blank=True, max_length=4096, verbose_name="Email nmap scan diff addresses, comma separated"
+    )
 
     def clean(self):
         """Checks for any invalid IPs, IP subnets, or FQDNs in the targets and excluded_targets fields."""
@@ -225,7 +231,7 @@ def clean(self):
 
         # Email scan alerts and email addresses.
         if self.email_scan_alerts and not self.email_alert_addresses:
-            raise ValidationError(f"Provide an email address if enabling 'Email scan alerts'")
+            raise ValidationError("Provide an email address if enabling 'Email scan alerts'")
 
         # Check for valid email addresseses string.
         if self.email_alert_addresses:
@@ -234,6 +240,17 @@ def clean(self):
                 self.email_alert_addresses
             )
 
+        # Email nmap_scan diff and email addresses.
+        if self.email_scan_diff and not self.email_scan_diff_addresses:
+            raise ValidationError("Provide an email address if enabling 'Email nmap scan diff'")
+
+        # Check for valid email addresseses string.
+        if self.email_scan_diff_addresses:
+            """Checks that email addresses are valid and returns a cleaned up string of them to save to the database."""
+            self.email_scan_diff_addresses = email_validation_utils.validate_string_of_email_addresses(
+                self.email_scan_diff_addresses
+            )
+
     def __str__(self):
         return str(self.site_name)
 
@@ -379,8 +396,8 @@ def clean(self):
                 f"to: {valid_scan_states}"
             )
 
-        # If a scan is paused and needs to be cancelled, don't set the state to "cancel", because the engine will try and
-        # cancel a running process that doesn't exist and error out.  Just bypass the "cancel" state and set it to
+        # If a scan is paused and needs to be cancelled, don't set the state to "cancel", because the engine will try
+        # and cancel a running process that doesn't exist and error out.  Just bypass the "cancel" state and set it to
         # "cancelled".  This logic is not needed on the client / API side in the ScheduledScanViewSet class in
         # console/django_scantron/api/views.py.
         if current_scan_status == "paused" and self.scan_status == "cancel":

console/requirements/base.txt

@@ -2,19 +2,19 @@
 argon2-cffi==20.1.0
 
 # Django
-Django==2.2.18
+Django==2.2.20
 
 # Forms
-django-crispy-forms==1.11.1
+django-crispy-forms==1.11.2
 
 # Django debug toolbar
-django-debug-toolbar==3.2
+django-debug-toolbar==3.2.1
 
 # Django environ
 django-environ==0.4.5
 
 # Django Extensions
-django-extensions==3.1.1
+django-extensions==3.1.2
 
 # Django filters for API
 django-filter==2.4.0
@@ -26,7 +26,7 @@ django-recurrence==1.10.3
 django_saml2_auth==2.2.1
 
 # Django REST Framework
-djangorestframework==3.12.2
+djangorestframework==3.12.4
 
 # Yet another Swagger generator
 drf-yasg==1.20.0
@@ -37,8 +37,8 @@ fqdn==1.5.1
 # ipython
 ipython
 
-# packaging required for drf-yasg, adding this to be safe
-packaging
+# Generate human-readable ndiff output when comparing 2 nmap XML scan files.
+pyndiff==1.0.2
 
 # Python-PostgreSQL Database Adapter
 psycopg2-binary==2.8.6
@@ -53,7 +53,7 @@ python-libnmap==0.7.2
 redis==3.5.3
 
 # python-rq - Redis Queue
-rq==1.7.0
+rq==1.8.0
 
 # XML to JSON
 xmljson==0.2.1