BandTelusur
A discovery platform for Indonesian bands, and a deliberate exercise in treating LLM features as software: versioned prompts, measured output, logged cost.
- Client
- Personal product
- Year
- 2025—2026
- Role
- Full-stack / AI Engineer
- Team
- Solo
Context
Most AI features are demos that worked once. The interesting engineering problem is not getting a model to produce something impressive — it is knowing, six weeks later, whether it still does.
This project was built to answer that: every AI call is versioned, logged and evaluated, so quality regressions are detectable rather than anecdotal.
Constraints
- Solo, serverless
- No ops team. Everything runs on Cloudflare Workers with Prisma Postgres in Singapore for latency from Indonesia.
- Model cost is real
- Every call costs money, so caching and rate limiting are product requirements rather than optimisations.
- Non-deterministic output
- The same prompt does not return the same thing twice, which makes conventional testing close to useless.
- User-submitted content
- Anything a stranger uploads needs moderating before it is public.
Architecture
A multi-step submission agent: scrape the source, extract structure with an LLM, score confidence per field, then route anything uncertain to a human.
- 01
Scrape
Fetch the submitted URL
- 02
Extract
LLM pulls structured fields
- 03
Score
Per-field confidence
- 04
Moderate
Vision + similarity check
- 05
Review
Human-in-the-loop on low confidence
- Chat routes across three tools — keyword search, semantic search over pgvector, and detail lookup — rather than stuffing everything into one prompt.
- Embeddings are 1536-dimension text-embedding-3-small, queried with pgvector alongside the relational data rather than in a separate vector service.
- Every AI call writes model, token count, latency and prompt version to the database. That table is what makes quality measurable.
Key decisions
Tool routing instead of one large prompt
Splitting into searchBands, semanticSearch and getBandDetail makes each call inspectable — when an answer is wrong, you can see which tool made it wrong.
More round trips, and the router itself becomes a component that can be wrong.
Confidence scoring per field, not per submission
A submission is rarely wholly right or wholly wrong. Scoring each extracted field routes only the uncertain parts to a human.
More schema and more prompt engineering than a single accept/reject score would need.
Human-in-the-loop as a pipeline stage, not an escape hatch
Moderation runs fetch → vision analysis → similarity check → human. The model narrows the queue; it does not have final say on what goes public.
Throughput is capped by human review, which is the correct trade for user-generated content.
A prompt versioning registry with every call logged
Prompt version, model, tokens and latency in the database means a quality regression can be traced to the change that caused it.
Write overhead on every call, and a schema that has to be maintained alongside the prompts.
LLM-as-judge for evaluation
Automated scoring catches drift between releases without a human grading every output.
The judge is itself a model and drifts too. Without a fixed reference set, you end up measuring one moving thing against another.
generateObject with Zod schemas, cached for 7 days
Band Insights return typed, validated structures instead of prose to be parsed, and the cache keeps repeat views free.
Insights can be up to a week stale — acceptable for editorial content, not for anything live.
Outcome
The result worth pointing at is not a feature — it is that cost per call and output quality are both observable, which is what makes an AI feature maintainable rather than merely impressive.
What I’d revisit
LLM-as-judge needs a small human-labelled golden set to calibrate against. Without one, the evaluation drifts alongside the thing it is evaluating and quietly stops meaning anything.
The KV sliding-window rate limiter is approximate under concurrency. Fine at this traffic level, wrong at a larger one — and it is the kind of thing that only reveals itself when it matters.
Diagrams are drawn from scratch and module names describe the domain. No client assets, interfaces or internal naming appear here.