When I built a natural-language property search, the naive version worked on day one: take the query, send it to an LLM, parse the structured result. Accurate. Also, at scale, about $138K a year.
That number is the whole story. A feature the business can’t afford to run isn’t a feature.
The pattern
Most queries are boring. “3 bed house in Brunswick under 900k” doesn’t need a frontier model — a few well-written rules parse it perfectly. The hard queries — vague locations, unusual phrasing, implied constraints — are where a model earns its cost.
So I routed:
- Rules first. If a query matches a confident rule-based parse, it never touches the LLM.
- Model for the hard cases. Only genuinely ambiguous queries fall through to the model.
- A confidence boundary between them, tuned against real queries, not vibes.
The result
- ~90% cost reduction — roughly $13.8K vs $138K/yr at scale, because the model only sees the queries that need it.
- +20% location-extraction accuracy — handling suburbs and fuzzy phrasing deliberately beat throwing everything at one model.
- 32 unit tests, 95% coverage — so I can change the routing without fear.
The lesson
The interesting engineering wasn’t the model call. It was deciding when not to make one. That decision is where production AI lives.