Cutting Input Tokens by 82%: Optimizing an Agentic RAG Pipeline

The short version: Our agentic RAG pipeline burned ~12,000 input tokens per turn. We cut it to ~2,200 on average — an 82% reduction — while keeping citations and confidence scoring. A deliberately dumb one-shot path we built as a comparison runs at ~760 input tokens — 94% cheaper than where we started.

Here's the story: the problem, the fix, and the rules we'd follow building it again.

The Problem: Agentic RAG Bleeds Tokens

Frontdesk is a self-hostable, multi-tenant RAG system. Tenants upload PDFs; an async pipeline parses, chunks, embeds, and stores them; a chat endpoint retrieves tenant-scoped context and streams an LLM answer. The chat answer was produced by a graph of small LLM steps:

classify → reformulate → retrieve → assess → compress → generate → cite

Every step was a separate LLM call — and every call re-read the whole picture:

• Each node re-sent the retrieved context as full text. Step N+1 re-tokenized everything step N already saw.
• Sequential calls compound fixed overhead. A few hundred tokens of system prompt × 7 calls is a few thousand tokens before you've answered anything.
• We couldn't even see where the cost was. No per-node accounting, no usage on the stream, no way to measure. We were optimizing blind.

The result was an awful ratio: ~12,000 input tokens in for ~800 output tokens out.

The Objective

1. Cut input tokens per turn dramatically — ideally down to a small multiple of "one good prompt."
2. Keep what made the graph worth it: answer quality, citations, confidence scoring.
3. Measure first, so every change was verified against real numbers, not vibes.

Phase 0: Measure Before You Optimize

The most important step — and the one we'd do first next time — was instrumenting before touching any behavior.

We built a token tracker that wraps every graph step and ships a per-step usage breakdown in the final event of the chat stream. Then a measurement harness: a fixed set of questions fired at a running API, producing a table of input/output tokens per question plus per-step totals. It paces requests and handles rate limits, because free-tier token caps will silently corrupt your data if you let them.

Lesson #1: capture a baseline before you change anything. We did — the "12K" figure came straight from Groq's usage metadata for that model — and it still haunted every comparison after.

Phase A: Slim the State, Merge the Calls

The insight: the graph was a state machine, but we were passing content through it. Whatever traveled in state got re-tokenized by every downstream call whether or not it was needed.

A1 — The graph now carries IDs, not text. Retrieval returns chunk references; content is fetched on demand only when a step actually needs it — and even then, without dragging along heavy vector data that nobody reads. A normal question now flows through most of the graph as just the question plus a handful of IDs.

A2 — Merged two adjacent calls into one. We had one step that assesses whether retrieved context is relevant and another that compresses it for generation. Together they re-sent the same context twice. We fused them into a single structured-output call that returns relevance score, confidence, and compressed context in one shot.

A3 — Dropped the checkpointer. The graph persisted full state between steps. With ID-only state, recomputing is cheap, so the serialization was pure overhead — gone.

Phase B: Smarter Retrieval (Not Just Cheaper Calls)

Once state was slim, the remaining cost was how much context each call saw. Three changes:

• LLM-as-reranker: instead of a heavyweight cross-encoder or a new dependency, a lightweight call scores the candidates and keeps the best few. Cheap to run, no infra added.
• Dynamic top-k by question type: a classifier tags each question (single-fact, comparison, multi-part) and retrieval pulls more candidates only when the question needs to fan out.
• Multi-part fan-out: for questions with several facets, we retrieve per facet and merge — so a simple question no longer pays for a complex one's context.

We also discovered the biggest cost multiplier in the whole system: reformulation loops. One question that went through two reformulation rounds jumped from ~1,400 to ~4,575 input tokens — a 227% increase. We capped iteration hard.

Phase C: Guard the Generation Input

Two safeguards on the output side:

• A hard context cap on the generator. Compression only helps if the compressed text is actually what reaches the model — we made that a guarantee, not a hope.
• A citation-validator step that verifies each cited source actually supports the claim, so citations stay truthful instead of confidently fabricated.

Phase D: Better Chunks at Ingest

Chunk boundaries decide how much useless text retrieval drags in. We replaced fixed-size splitting with semantic chunking: a single embedding pass and similarity-based boundaries, so chunks break where topics actually change instead of mid-paragraph. Chunks also carry a human-readable source label — answers can cite "Page 1" instead of "chunk 0x3f9". Re-embedding the test document changed its chunk counts, proof the new boundaries were meaningfully different, not cosmetic.

The Control Group We Almost Skipped

Halfway through, we built a deliberately dumb one-shot RAG path: embed the question → grab top chunks → one prompt call → stream. No reranking, no compression, no citations, no reformulation. It wasn't a feature — it was the answer to "does the graph actually earn its tokens?"

The Numbers

Same document, same 4 questions, measured back-to-back:

Three findings worth keeping:

1. Input per turn fell ~82% from baseline, and clean questions run at ~1.4K.
2. The assess-and-compress step alone accounts for ~70% of the graph's input budget — the "quality" machinery now costs more than generation itself. That's a deliberate trade — and the thing to watch as prompts or context sizes drift.
3. Reformulation is the real tax. A single question needing two rounds cost +227% over a clean one. Capping loops is non-negotiable.

And the uncomfortable truth the control group exposed: for a small corpus, the graph costs ~190% more than the dumb path (2,218 vs 761) for answers that are only slightly better — sometimes just terser. The graph earns its tokens as the corpus and the questions grow: multi-part queries, cross-document synthesis, verified citations.

Things to Keep in Mind Building Something Like This

1. Measure first; record a baseline. A starting number you can't reproduce is a hole in every future comparison.
2. Know what your "tokens" are. A chars/4 estimate is fine for before/after, but it's not the provider's bill. Pick one method, never mix.
3. Carry IDs through graph state, never content. Content in state gets re-tokenized by every downstream call.
4. Every LLM call has fixed overhead. System prompts are paid per call, per node. Merging adjacent calls is the cheapest win available.
5. Loops are the real multiplier. Retry and reformulation loops multiply all per-call costs. Cap them hard.
6. Rerankers are expensive — price them honestly. An LLM reranker can cost more input than generation. It's only worth it if it changes which sources the generator sees.
7. Compression is theater without a cap. A compressor whose output never reaches the model is just extra tokens.
8. Rate limits corrupt measurements. Free-tier caps mean back-to-back asks produce errors, not data. Build pacing and retry into your measurement tool.
9. The corpus is a moving target. Chunking and re-indexing change retrieval results. Pin the measurement corpus or your before/after numbers lie.
10. Build the dumb baseline. A one-shot path answers "is this complexity paying for itself?" on real data — let the numbers, not the architecture diagram, decide.
11. Stream usage to the client. Token counts in the stream's final event mean cost regressions surface the day they ship, not the month after.

Bottom Line

Input tokens are the dominant cost in agentic RAG, and the architecture — not the model — was the waste. By measuring first, carrying IDs instead of content, merging sequential calls, capping context, and pricing every node against a one-shot baseline, we took Frontdesk from ~12,000 to ~2,200 input tokens per turn (−82%) while keeping citations and confidence scoring.

The last lesson is the most useful one: the graph is a product decision, not a default. Measure it against the dumbest thing that works, on your real data — and let the numbers decide when it's worth the tokens.

Per-turn averages from the measurement harness against a single test document, token estimate = chars/4.