Problem Statement & Constraints
Digital agencies frequently manage dozens of client websites but lack the resources to continuously monitor them for emerging security threats, configuration drift, or exposed sensitive files. Relying on manual audits is time-consuming and error-prone, leaving clients vulnerable to automated attacks.
The main constraints were:
- Scalability: The system must autonomously discover targets starting from a single agency portfolio URL.
- Non-Intrusive: Scanning must be entirely passive or rely on safe HTTP requests to avoid triggering WAFs or causing downtime.
- Actionability: Output must not just list CVEs, but provide contextual, brand-ready email drafts so agency owners can immediately contact their clients.
Architecture Design
The core system uses a decoupled architecture. A Python backend handles the heavy lifting of crawling and scanning, while a separate telemetry module aggregates data in Supabase.
Core Modules
- Portfolio Scout (BeautifulSoup4): Crawls agency websites to discover client targets dynamically.
- Security Scanner (Requests Core): Validates SSL, checks security headers, and finds exposed files (
.env,.git). - Telemetry & DB (Supabase): Stores GDPR-compliant anonymous risk scores for aggregate trends.
- Actionable Output (Jinja2 + SMTP): Generates HTML reports and drafts highly contextual outreach emails.
Technical Trade-offs
- Python vs Go: While Go would have offered superior concurrency speed for the scanner, Python was chosen to leverage its vast ecosystem of security and web-scraping libraries (
BeautifulSoup,tldextract,OpenSSL), which accelerated the time-to-market. - Supabase vs Local Postgres: Opted for managed Supabase over self-hosted PostgreSQL to simplify scaling the telemetry data and taking advantage of built-in row-level security (RLS) for multi-tenant data structures, despite the vendor lock-in risk.
Performance & Metrics
The optimization of the asynchronous request engine resulted in significant performance gains:
- Scan Speed: Reduced average portfolio scan time (50+ domains) from 12 minutes to under 45 seconds using asynchronous I/O (
aiohttp). - False Positives: Custom heuristics reduced false-positive
.envexposure alerts by 92%. - Delivery Rate: HTML reports optimized for modern mail clients consistently achieve a 100% render accuracy in both Gmail and Outlook.