Data Infrastructure · active · 2026
A storage-free ingestion pipeline that turns a retailer's own product APIs into clean, continuously-refreshed beauty-product data for competitive-intelligence dashboards.
Competitive intelligence for beauty retail needs clean, structured, current product data — prices, discounts, ratings, review volume, ingredients, stock — across a full catalogue of thousands of SKUs. Scraping retail sites the usual way returns JavaScript-rendered HTML where price, rating and stock never appear in the markup, so the data is incomplete and every field needs heavy cleanup. The conventional answer is a large ELT stack: raw capture, an AI/transform cleaning layer, normalization and promotion — expensive to run and heavy to operate.
DIP Engine is a storage-free two-step engine. A discovery pass maps the retailer’s full catalogue via its own category-crawl API into a lean 8-column map; then for each product it calls the site’s private JSON endpoints (detail-data, details-info, review-summary) in memory and transforms the response straight into one clean, typed row plus an append-only daily snapshot. There is no raw storage and no LLM in the path — parsing is deterministic (name/size/set splitting, a K-beauty ingredient dictionary, loop-until-stable HTML-entity decoding). A downstream `distribute` step projects that single clean store into the normalized dimension and fact tables a dashboard reads. Recovery is re-running from the map, not restoring archives.
The first ten phases were a full ELT stack: raw HTML capture, a hybrid transform/normalization engine, an AI review layer (Mistral), canonical promotion, and facts/enrichment/media/report layers feeding a normalized eight-table core. Once we confirmed the source exposes its own JSON APIs that already return clean, structured fields, that entire middle chain was dead weight. We retired the AI, promotion and enrichment packages, their workers, and the core-eight plus its staging tables in one migration — recoverable via the `pre-retire-middle-chain` git tag.
Ingestion originally wrote each fetched envelope to object storage (DigitalOcean Spaces, MinIO locally) and a `raw_documents` table before a separate clean-load read it back. The round-trip produced a byte-identical clean row and only added an object-storage dependency and MinIO operational pain. We moved to fetching and cleaning in memory in a single pass; the discovery map already lets any product be re-fetched, so nothing was lost.
We evaluated three capture methods against the live site. A plain HTML GET returned 422 KB but only name and image in the markup — price, rating and stock are filled by JavaScript after load. Playwright rendered correctly but cost roughly 5–8 s per product with Chromium overhead and scrape-from-DOM fragility. The site’s own JSON APIs returned complete data at about 2 s per product over plain HTTP with no session, so we chose direct API and kept the browser path documented as a fallback.
The consuming dashboard’s schema expected sales, units, revenue and market-share tables. The retailer’s API exposes none of that. Rather than seed plausible numbers, we filled those columns with an explicit review-volume proxy (review count as the demand weight), left every revenue column null, and documented the substitution. Where real signals exist — out-of-stock rate from live stock, promo rate from discount — those columns carry actual data.
A storage-free two-step engine — map the catalogue, then fetch each product’s own JSON APIs in memory and clean straight to a typed row — with a downstream projection into the tables a dashboard reads.
TypeScript, Node.js 22, Fastify 5, Drizzle ORM, Zod, Pino, Observability console (vanilla JS), PostgreSQL, DigitalOcean Managed Postgres, drizzle-kit migrations, pnpm workspaces, Docker Compose, DigitalOcean droplet, systemd, cron