Real-time game AI lives or dies on responsiveness. Players feel delays above ~200ms in chatty NPCs, and anything over a second reads like “loading.” The tricky part is that the same knobs that reduce latency often increase cost (and vice versa). This article breaks down a practical way to think about tokens, caching, and streaming so you can hit a target experience without burning your budget.
1) Latency isn’t one number: decompose the pipeline
When a player sends a message, your end-to-end time typically includes:
- Client → server network + queuing
- Context assembly: memory, state, tools, retrieval (if any)
- Model time-to-first-token (TTFT): the pause before any output arrives
- Token generation: how fast the model emits the rest of the response
- Post-processing: safety checks, formatting, tool results
A useful rule: optimize for TTFT first (perceived responsiveness), then optimize for total tokens (cost), then tune throughput (scalability).
2) Tokens drive cost—and they also drive latency
Most deployments pay by tokens, and tokens map directly to work. But they also affect speed in two ways: (1) longer prompts increase prefill time, and (2) longer answers increase generation time. To control both, treat your prompt like a budget.
Token budget checklist
- Keep system + policy text compact; move rarely-needed guidance into tools or short “modes.”
- Summarize chat history into a rolling “memory” instead of replaying everything.
- Prefer structured state (JSON) over verbose prose when passing game state.
- Set explicit max output tokens; use follow-up turns for depth.
If you’re using retrieval for lore or patch-note grounding, see RAG for Game Studios for strategies that keep retrieved chunks short and relevant.
3) Caching: the simplest lever with the biggest payoff
Caching reduces both latency and cost when players repeat similar actions. In games, repetition is common: tutorial hints, town NPC greetings, item descriptions, quest recaps, and “what can I do next?” queries. The key is choosing what to cache and how to key it.
What to cache
- Static prompt prefixes: safety policy, game rules, style guide.
- Canonical answers: help text, glossary definitions, patch explanations.
- Computed context: retrieved lore snippets, compiled tool outputs, condensed memory.
Cache keys that won’t betray you
A good cache key includes mode (dialogue vs. hint vs. recap), locale, build/version, and a normalized player intent. Avoid raw player text as a key—use intent classification or a hashed/cleaned representation to improve hit rate without leaking PII.
{
"mode": "npc_dialogue",
"npc_id": "alchemist_07",
"quest_state": "q142:stage3",
"build": "1.9.2",
"intent": "ASK_RECIPE_HINT"
}4) Streaming: performance you can feel
Streaming doesn’t reduce model time, but it changes perception. If a player sees text arriving quickly, they tolerate longer total completion times. For NPC dialogue, streaming also enables UX flourishes like “typing,” voice-lip sync alignment, or early interruption when the player walks away.
Two practical tips:
- Front-load the intent: ask the model to start with the key answer first, then optional detail.
- Stop conditions: enforce short clauses and allow cancel/retry if player state changes mid-stream.
If your NPCs are fully conversational, pair streaming with tight lore constraints; otherwise you’ll stream mistakes faster. The canon-consistency angle is covered in Building a Lore Bible for LLMs.
5) Balancing cost vs. latency with a simple playbook
| Goal | Do this | Trade-off |
|---|---|---|
| Lower TTFT | Shrink prompt, cache prefixes, precompute retrieval | More engineering + careful invalidation |
| Lower cost | Cap output, summarize memory, reuse templates | Less richness per turn |
| Better feel | Stream, show partials, allow interrupts | Need robust cancellation + state checks |
6) A note for tabletop and club-style “real-time”
Even if you’re using AI for tabletop recaps or meetup logistics (not twitchy gameplay), the same principles apply: keep prompts short, reuse cached context (weekly formats, host templates), and stream long summaries so readers see progress. For broader design patterns, Top Use Cases for LLMs in Gaming is a good companion piece.
Want more like this? Browse the full archive on Blog.