Data Infrastructure · active · 2026 · Client: AsiaCosmePulse
A backend engine that turns scattered beauty-industry news into cited, catalog-linked product insights for a cosmetics market-intelligence platform.
Beauty brands and retailers had no efficient way to track product news across dozens of outlets spanning multiple markets and languages. Monitoring was manual or relied on expensive analyst reports, and the coverage that matters is scattered thinly across general-news sites that have no beauty section. The AsiaCosmePulse platform already had a product catalog and a dashboard, but no automated pipeline to feed it timely, structured, source-cited news tied to its real brands and products.
We built a backend engine that ingests beauty news through a config-driven collection framework — sitemap, RSS and HTML-crawl strategies plus headless-browser rendering for JavaScript sites — and normalizes every article into a common shape with full provenance. Mistral models classify relevance, summarize each article, and extract typed entities (brands, ingredients, products, companies); a deterministic engine scores trend momentum while embeddings cluster same-topic articles into merged stories. A relevance gate keeps only beauty-product content, so downstream cost and noise stay bounded. Results are projected into the client's existing platform database using its exact table schema, matching each article to real catalog brands and products by name and creating rows only for genuine gaps. Discovery, enrichment, delivery (a FastAPI read API plus Markdown reports) and an event-driven webhook path are independent, idempotent stages.
The obvious way to add Korean coverage, where the client's catalog is strongest, was to scrape the big Korean English dailies. We onboarded five (Korea Herald, Korea Times, JoongAng, MK) via sitemap and HTML-crawl and got zero beauty articles out of roughly fifty — beauty is scattered across general news with no dedicated section. A paid search API would solve it but needs a key, so we found a keyless path: Google News RSS search plus a headless browser to resolve its JavaScript redirect links to the real publisher article. That yielded 13 of 15 relevant K-beauty articles, and article-to-catalog brand match jumped from 28% to 79%.
We planned to create the dashboard tables fresh. On connecting to the DigitalOcean database we found it was the client's live platform DB, already holding populated brands, markets and products_master, plus a sources table whose name collided with ours. We redesigned to a two-database model: internal pipeline tables in a separate database, and a publisher that creates only the missing News Pulse tables and links articles to the existing catalog by name — matching where possible, creating catalog rows only for real gaps, and never dropping or altering an existing table.
Ranking which products are trending is the sellable output, so it has to be reproducible and explainable to a buyer who asks why something is ranked first. We used the LLM only for extraction and narrative, and computed momentum with a fixed formula over volume, cross-source breadth, velocity against the prior window, and sentiment. This is cheaper, deterministic, and defensible — the model never decides the ranking.
The first collector was hand-written for a single publication. Facing a target of 100-plus heterogeneous sources, we generalized it into a YAML registry with pluggable discovery (sitemap/RSS/HTML-crawl) and extraction (JSON-LD/readability) strategies, and an orthogonal render flag that swaps in a headless browser for JavaScript sites. Adding a source became a config entry, and a fleet runner isolates each source so one broken site never halts the run. Live testing then caught bugs the offline tests missed — CDATA-wrapped sitemap URLs with stray whitespace, and a browser wait strategy that hung on ad-heavy pages until we switched off network-idle.
A staged, idempotent pipeline: discover beauty news across many source types, extract and enrich it with Mistral, score trends and cluster related stories, then publish into the client's platform schema.
Mistral (chat), Mistral embeddings (mistral-embed), Python 3.12, FastAPI, SQLAlchemy 2, Alembic, httpx, Pydantic, trafilatura, Playwright, PostgreSQL, SQLite (tests), Docker Compose, GitHub Actions CI, DigitalOcean Managed Postgres, Ruff, pytest