How a simple collection of web scrapers evolved into a scalable, schema-driven workflow engine.
When we started this project, our goal was straightforward: Can we build an application that helps people find the best deals on products and services?
Every day, consumers spend hours jumping between dozens of e-commerce sites, marketplaces, and social media channels comparing prices. We wanted to aggregate this fragmented data into a unified platform where users could instantly compare offers and make informed purchasing decisions.
We weren't trying to build a complex, production-ready distributed platform on day one. We just wanted to validate the core hypothesis.
Phase 1 — Collecting the Initial Data
My first responsibility was simple: acquire data at scale.
I wrote a suite of Python scripts using BeautifulSoup and Selenium to extract product information across target websites and APIs. The initial setup was modest, but it performed remarkably well — soon, hundreds of product listings were flowing into our environment daily.
The initial data ingestion challenge was solved, but a harder problem immediately took its place.
Phase 2 — Normalization and LLM Integration
Raw extracted data is inherently messy. Different vendors describe identical products in wildly different ways:
- Store A:
iPhone 16 Pro Max 256GB Black Titanium - Store B:
Apple iPhone Pro Max 16 (256 GB)
To compare prices accurately, we needed to parse, clean, and map unstructured text into a canonical schema.
Instead of writing hundreds of brittle regular expressions and custom parsers, we leveraged Large Language Models (LLMs) to perform structured extraction:
Raw HTML / Unstructured Text -> LLM Processing -> Normalized JSON Schema
Because we were still in the validation phase, we deliberately avoided setting up a traditional database infrastructure. Instead, we used Google Sheets as our primary datastore.
This provided immediate, pragmatic advantages:
- Zero DBA overhead: No migrations, connection pools, or schema management.
- Non-technical access: Operations and business teammates could inspect live data directly.
- Human-in-the-loop validation: Teammates could manually review, flag, or correct LLM extraction errors inline.
For an early-stage proof of concept, Google Sheets was precisely the right tool.
Phase 3 — Orchestration via Flask and n8n
As our library of Python scripts grew, triggering them manually became a massive bottleneck. We needed an orchestration layer.
Rather than rewriting our scripts into formal worker tasks, I wrapped each script in a lightweight Flask API. We then adopted n8n as our visual workflow automation platform to trigger these endpoints over HTTP.
┌─────────────────┐ ┌───────────┐ ┌─────┐ ┌───────────────┐
│ Python Scripts │ ◄───► │ Flask API │ ◄───► │ n8n │ ◄───► │ Google Sheets │
└─────────────────┘ └───────────┘ └─────┘ └───────────────┘
From an operational standpoint, this setup was fantastic. Non-technical team members could build and modify data pipelines using n8n's visual node interface without writing code.
However, as the system expanded, as a developer my experience degraded rapidly.
Phase 4 — The Cracks in the Foundation
As our data sources multiplied, our lightweight setup hit structural limits:
- Zero Code Reuse: Every new web source required building a script from scratch with repeated boilerplate.
- Brittle, Untyped Configuration: Workflow logic and endpoint configurations were split between n8n nodes and Google Sheets formulas. Everything was untyped plain text—a single typo in a URL string or key name would silently break an entire execution pipeline.
- No Relational Integrity: Google Sheets is a spreadsheet, not a relational database. Enforcing foreign key constraints, unique indexes, or nested entity relationships became near impossible.
- Concurrency Bottlenecks: Multiple concurrent scrapers attempting to write to the same Google Sheet triggered continuous race conditions and API rate-limiting errors.
We were slowly attempting to re-engineer core database features inside a spreadsheet. It was time to replace the weakest link.
Phase 5 — Building a Domain-Specific Workflow Builder
Rather than tearing down the entire stack, we replaced the most error-prone component first: the workflow configuration layer.
I built a custom, type-safe Workflow Builder tailored specifically to our extraction workflows. Instead of manually entering API URLs and parameters inside n8n nodes, users configured pipelines via a visual UI backed by strong schema validation.
┌──────────────────┐ ┌────────────────────────┐ ┌─────┐ ┌─────────────┐
│ Workflow Builder │ ───► │ Typed JSON Defintion │ ───► │ n8n │ ───► │ Python APIs │
└──────────────────┘ └────────────────────────┘ └─────┘ └─────────────┘
n8n was demoted to a pure execution engine. It no longer held business logic or configuration; it simply parsed structured workflow definitions generated by our builder and executed the corresponding HTTP tasks.
Phase 6 — Database Migration & Dual-Storage Sync
Eventually, Google Sheets reached its hard concurrency and storage limits. We migrated our primary source of truth to PostgreSQL, instantly resolving our transaction, indexing, and data integrity issues.
However, our operational team relied heavily on Google Sheets for daily auditing and reporting. Completely cutting off access would have severely disrupted business operations.
To solve this, I designed a decoupled synchronization service:
- PostgreSQL served as the authoritative transactional database for all incoming pipeline data.
- A background sync service continuously listened for updates, transformed the relational records, and published curated views back into Google Sheets.
Developers gained an ACID-compliant relational database, while operations retained their familiar interface without touching raw SQL.
Phase 7 — The Metadata & Schema-Driven Dashboard
With PostgreSQL in place, managing manual UI forms and API endpoints for every new database table created a new engineering bottleneck. Adding a new entity required updating the ORM model, writing CRUD API handlers, building frontend forms, and configuring validation logic.
To eliminate this boilerplate, I built a schema-driven management dashboard that automatically generated CRUD forms and API endpoints based on database models annotated with rich metadata.:
┌─────────────────────────┐
│ SQLAlchemy Model │
│ + UI Metadata Decorator│
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Generic CRUD API │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Dynamic UI Dashboard │
│ (Auto-generated forms) │
└─────────────────────────┘
By annotating our database models with rich UI metadata (display labels, input fields, validation rules, lookup dropdowns), the frontend automatically rendered tailored management screens on the fly.
Adding a new database table now required writing zero frontend code and zero new API endpoints.
Phase 8 — Replacing n8n with an In-House Orchestrator
While n8n served us well initially, observing workflow execution at scale remained challenging. Native support for deep execution tracing, granular retries, custom failure alerts, and complex cron scheduling required heavy workarounds.
Instead of building hacks around n8n, I replaced it entirely with a lightweight, custom orchestrator.
This wasn't an attempt to build a generic replacement for n8n. By strictly scoping the engine to our specific execution model, the custom orchestrator remained surprisingly compact while offering seamless integration with our database, monitoring tools, and schema dashboard.
Phase 9 — Service Isolation & Horizontal Scalability
To prepare the platform for high-throughput scaling, we restructured our monolithic single-deployment setup into a single-repository, multi-service architecture.
Not all components scale equally. The administrative UI, CRUD APIs, and database require minimal resources, whereas concurrent web-scraping jobs demand significant CPU and I/O.
┌───────────────────┐
│ Scheduler │
└─────────┬─────────┘
│
▼
┌───────────────────────┐
│ Script Runner API │
└───────────┬───────────┘
│
┌─────────────────────┼─────────────────────┐
▼ ▼ ▼
┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐
│ Worker Instance 1 │ │ Worker Instance 2 │ │ Worker Instance N │
└───────────────────┘ └───────────────────┘ └───────────────────┘
The system was split into four core services, each with its own Dockerfile and independent deployment pipeline:
- API Service: Handles authentication, schema management, the workflow builder, and CRUD operations.
- Scheduler: Manages workflow state machines, execution queues, and job dispatching.
- Script Runner: Executes scraping and data-processing tasks behind a unified execution interface.
- Health Monitor: Tracks execution failure rates, system health metrics, and alerting channels.
Future Outlook
Because the Script Runner is isolated behind an API abstraction, scaling horizontally is trivial. Today, it processes tasks via local subprocesses; tomorrow, it can sit behind a load balancer managing a fleet of remote worker nodes across multiple cloud instances — without changing a single line of orchestration logic.
Lessons Learned
Looking back, the single most valuable lesson was knowing when to accept tech debt — and when to pay it down.
- Google Sheets was not a mistake: It gave us instant business validation without wasting weeks on database architecture.
- n8n was not a mistake: It allowed us to rapidly stitch together prototype pipelines before we understood our exact abstraction requirements.
- Flask was not a mistake: It let us reuse our existing Python script ecosystem with zero friction.
Every architectural migration was executed only after the existing solution reached its natural scaling threshold. By focusing continuously on the smallest change that solves today’s bottleneck while preserving tomorrow’s options, we transformed a fragile collection of Python scripts into a resilient, distributed data processing platform.