Website Intelligence Platform & Bounded Analysis Pipeline
Website intelligence and analysis platform that accepts a URL, safely fetches and analyzes the site, calculates SEO and performance signals, persists scan history, supports comparisons and monitoring, and produces structured insights.
Core Challenge: Websites expose data through diverse layers: metadata, headings, content structure, media, scripts, and crawl rules. The challenge was turning an unparsed URL into structured, persistent, and actionable website intelligence without unbounded execution or server crashes.
Websites expose information through many different layers: metadata, headings, content structure, media, scripts, response characteristics, and crawl rules. The challenge was to turn a raw URL into structured, persistent, and actionable website intelligence.
Scope Boundary: WebScope is engineered as a lightweight, serverless-friendly website intelligence platform for fast heuristic analysis, continuous monitoring, and automated comparisons. It does not claim to be a heavyweight headless browser replacement for Google Lighthouse, Google Search Console, or full commercial enterprise crawler suites.
Authoritative system flow and architecture topology spec.
HORIZONTAL SCROLL ON MOBILE ↔Rather than issuing unbounded scraping calls, WebScope structures website analysis into a deterministic 8-stage pipeline wrapped with concurrency control, timeout safeguards, and decoupled AI fallbacks.
Queries /robots.txt before touching the page, immediately rejecting disallowed paths.
Acquires 1 of 4 slots via acquireSlot() and fetches the HTML with a 5s AbortController timeout.
Cheerio builds a lightweight in-memory AST to extract titles, descriptions, headings, body text, and images.
Evaluates title length, meta presence, single H1 hygiene, heading hierarchy, content length, and image alt tags.
Measures TTFB server response time, total HTML payload size, script tag count, and asset weight distribution.
Pre-computes a deterministic qualitative output first; attempts Gemini AI generation with a strict 6-second timeout.
Saves normalized snapshots into PostgreSQL via Prisma (ScanHistory, ScrapedData, RequestLog).
Feeds interactive UI dashboards, automated side-by-side comparisons, scheduled batch monitors, and PDF/CSV exports.
Authoritative system flow and architecture topology spec.
HORIZONTAL SCROLL ON MOBILE ↔Queries /robots.txt before touching the page, immediately rejecting disallowed paths.
Acquires 1 of 4 slots via acquireSlot() and fetches the HTML with a 5s AbortController timeout.
Cheerio builds a lightweight in-memory AST to extract titles, descriptions, headings, body text, and images.
Evaluates title length, meta presence, single H1 hygiene, heading hierarchy, content length, and image alt tags.
Measures TTFB server response time, total HTML payload size, script tag count, and asset weight distribution.
Pre-computes a deterministic qualitative output first; attempts Gemini AI generation with a strict 6-second timeout.
Saves normalized snapshots into PostgreSQL via Prisma (ScanHistory, ScrapedData, RequestLog).
Feeds interactive UI dashboards, automated side-by-side comparisons, scheduled batch monitors, and PDF/CSV exports.
A unified Next.js 14 full-stack architecture coupling bounded asynchronous worker queues, in-memory HTML DOM extraction, relational snapshot persistence, and automated downstream comparison and monitoring services.
MAX_CONCURRENT_ANALYSIS = 4. Incoming analysis jobs acquire an explicit slot or wait in a FIFO queue, preventing thread starvation and target server flooding.
Axios and Cheerio execute in pure Node.js serverless runtimes with sub-100ms startup times, avoiding the 150MB+ memory footprint and cold-start latency of headless browsers.
Normalized PostgreSQL models link scans to authenticated users, request logs, automated monitoring rules, and side-by-side comparison verdicts with foreign key cascades.
Authoritative system flow and architecture topology spec.
HORIZONTAL SCROLL ON MOBILE ↔MAX_CONCURRENT_ANALYSIS = 4. Incoming analysis jobs acquire an explicit slot or wait in a FIFO queue, preventing thread starvation and target server flooding.
Axios and Cheerio execute in pure Node.js serverless runtimes with sub-100ms startup times, avoiding the 150MB+ memory footprint and cold-start latency of headless browsers.
Normalized PostgreSQL models link scans to authenticated users, request logs, automated monitoring rules, and side-by-side comparison verdicts with foreign key cascades.
From user submission to persistent dashboard reporting, data transitions through a series of strict validation, extraction, scoring, and persistence steps.
User Submission: User enters target URL via web form or triggers scheduled monitoring batch.
API Validation: Next.js API route validates format, protocol, and sanitization rules with Zod.
Robots Check: Parser verifies target site robots.txt crawl permissions before opening socket connections.
HTTP Fetch: Axios retrieves target HTML within a 5-second AbortController timeout window.
DOM Extraction: Cheerio builds in-memory AST and parses title, meta, headings, text, media, and scripts.
Analysis Engines: SEO and performance analyzers calculate weighted scores and identify optimization gaps.
AI Interpretation: Gemini AI produces qualitative archetypes with pre-computed fallback guarantees.
Database & UI: Prisma persists the full snapshot to PostgreSQL, immediately updating user dashboards and monitors.
Authoritative system flow and architecture topology spec.
HORIZONTAL SCROLL ON MOBILE ↔User enters target URL via web form or triggers scheduled monitoring batch.
Next.js API route validates format, protocol, and sanitization rules with Zod.
Parser verifies target site robots.txt crawl permissions before opening socket connections.
Axios retrieves target HTML within a 5-second AbortController timeout window.
Cheerio builds in-memory AST and parses title, meta, headings, text, media, and scripts.
SEO and performance analyzers calculate weighted scores and identify optimization gaps.
Gemini AI produces qualitative archetypes with pre-computed fallback guarantees.
Prisma persists the full snapshot to PostgreSQL, immediately updating user dashboards and monitors.
Operating an automated website intelligence crawler requires handling slow hosts, external rate limits, third-party AI timeouts, and multi-tenant persistence.
External websites are inherently unreliable. Remote targets can hang indefinitely, enter infinite redirect loops, drop connections abruptly, or explicitly disallow automated crawling via robots.txt.
Multiple users and automated monitoring batches trigger URL scans simultaneously. Unbounded concurrent operations risk socket exhaustion, Vercel serverless function throttling, and remote IP bans.
Third-party LLM APIs suffer rate limits, unpredictable latency, and outages. Qualitative AI synthesis cannot become a hard blocking dependency that halts the underlying deterministic SEO and performance analysis.
Analysis snapshots must remain relational and queryable across user accounts, longitudinal monitoring alerts, side-by-side domain comparisons, and export formats without data inconsistency.
Rather than relying on unconstrained scripts or optimistic client assumptions, WebScope enforces clear operational boundaries across concurrency, timeouts, validation, and data resilience.
Examining how key operational constraints were addressed within the WebScope codebase.
A remote website may be slow, enter infinite redirects, or be unreachable.
Bound requests with 5-second AbortController timeout protection and classify failures into meaningful error categories (TIMEOUT, NETWORK, DISALLOWED_BY_ROBOTS, UNKNOWN).
The analysis pipeline fails predictably instead of hanging indefinitely or exhausting server sockets.
Multiple URL analyses can run simultaneously across interactive users and batch cron jobs.
Implement an in-memory bounded analysis queue with a strict maximum of four active parallel analyses (MAX_CONCURRENT_ANALYSIS = 4).
Concurrent analysis work is explicitly controlled, preventing serverless function throttling and target server bans.
Not every target website permits automated crawling or indexation.
Fetch and parse target robots.txt prior to scraping; immediately abort and return a classified error if path or agent is restricted.
Disallowed targets are rejected cleanly before scraping execution begins, respecting target webmaster rules.
The AI insight layer can encounter rate limits, model outages, or slow generation times.
Generate a deterministic fallback insight locally first and attempt AI generation with a strict 6-second timeout, falling back on error.
The core SEO, performance, and comparison pipeline remains 100% functional without hard dependency on external AI services.
Engineering choices grounded in operational simplicity, serverless runtime constraints, and decoupled resilience.
Lightweight HTTP fetching and fast, low-overhead server-side HTML parsing.
Client-rendered SPA content that only materializes after heavy JavaScript browser execution is not equivalent to a full browser-rendered crawl.
Lightweight HTTP fetching and fast, low-overhead server-side HTML parsing.Client-rendered SPA content that only materializes after heavy JavaScript browser execution is not equivalent to a full browser-rendered crawl.
Relational data naturally models users, multi-page scan histories, request logs, threshold monitors, and comparison sets.
Requires declarative schema migrations and strict relational index design across foreign keys.
Relational data naturally models users, multi-page scan histories, request logs, threshold monitors, and comparison sets.Requires declarative schema migrations and strict relational index design across foreign keys.
URL analysis is network-bound, external, and unpredictable.
Excess parallel requests wait in a brief FIFO queue during high burst periods.
URL analysis is network-bound, external, and unpredictable.Excess parallel requests wait in a brief FIFO queue during high burst periods.
Generative AI enriches the qualitative interpretation of raw numbers with persona archetypes.
Requires maintaining a dual-path code pipeline that derives local heuristics before calling LLM APIs.
Generative AI enriches the qualitative interpretation of raw numbers with persona archetypes.Requires maintaining a dual-path code pipeline that derives local heuristics before calling LLM APIs.
Unified full-stack architecture combining React Server Components and route handlers.
Couples frontend dashboard deployments with API gateway lifecycles.
Unified full-stack architecture combining React Server Components and route handlers.Couples frontend dashboard deployments with API gateway lifecycles.
Robust system design accounts for every branch in the execution graph. WebScope defines explicit transition paths for successful extractions, robots rejections, network dropouts, and AI timeouts.
Authoritative system flow and architecture topology spec.
HORIZONTAL SCROLL ON MOBILE ↔Authoritative system flow and architecture topology spec.
HORIZONTAL SCROLL ON MOBILE ↔Rather than storing raw unindexed blobs, WebScope models scans, audits, monitors, and comparisons as structured relational entities in PostgreSQL.
Tenant identity & authentication owner
Immutable website scan record with scoring snapshot
Parsed DOM content & extracted technical attributes
Granular structural inspection & accessibility breakdown
Continuous domain health tracking configuration
Threshold breach event & resolution state
Side-by-side performance & SEO competitive analysis
HTTP audit log for monitoring queue seed and rate checks
Authoritative system flow and architecture topology spec.
HORIZONTAL SCROLL ON MOBILE ↔Tenant identity & authentication owner
Immutable website scan record with scoring snapshot
Parsed DOM content & extracted technical attributes
Granular structural inspection & accessibility breakdown
Continuous domain health tracking configuration
Threshold breach event & resolution state
Side-by-side performance & SEO competitive analysis
HTTP audit log for monitoring queue seed and rate checks
These metrics represent verified architectural constraints enforced directly in source code, rather than synthetic marketing benchmarks.
Explicit bounded queue slots in analyzeWebsite.ts preventing worker exhaustion.
Hard AbortController cancellation ceiling on all remote Axios HTTP requests.
Maximum time permitted for LLM generation before falling back to local heuristics.
Strict limit on HTTP redirect follow-throughs to eliminate infinite crawl loops.
Upper bound on unique URLs processed in a single automated monitoring sweep.
Chunk size for parallel analysis in monitorService.ts to manage memory load.
Engineering depth is proven through architectural discipline, failure awareness, and data modeling.
Not just a frontend dashboard. The product connects UI, API route handlers, HTTP scraping, analytical scoring engines, relational persistence, and automated background-style monitoring sweeps into one cohesive platform.
Dual timeout barriers, robots.txt restriction checks, deterministic AI fallbacks, and typed error classification are designed directly into the core execution path, ensuring resilience when external targets fail.
Scans do not evaporate in client memory. Every analysis becomes structured relational data in PostgreSQL that can be revisited, exported to PDF/CSV, compared against competitors, or tracked for performance regressions.
A clear progression of value: Raw HTML payload → structured metrics extraction → weighted scoring → qualitative persona interpretation → historical trend insights.
Not just a frontend dashboard. The product connects UI, API route handlers, HTTP scraping, analytical scoring engines, relational persistence, and automated background-style monitoring sweeps into one cohesive platform.
Dual timeout barriers, robots.txt restriction checks, deterministic AI fallbacks, and typed error classification are designed directly into the core execution path, ensuring resilience when external targets fail.
Scans do not evaporate in client memory. Every analysis becomes structured relational data in PostgreSQL that can be revisited, exported to PDF/CSV, compared against competitors, or tracked for performance regressions.
A clear progression of value: Raw HTML payload → structured metrics extraction → weighted scoring → qualitative persona interpretation → historical trend insights.
From unparsed remote HTTP streams to persistent operational intelligence.
Authoritative system flow and architecture topology spec.
HORIZONTAL SCROLL ON MOBILE ↔