Decide which prompt prefix to cache vs re-run to reduce LLM latency

#ai #llm #performance

Implementation summary

This post gives a practical decision flow for splitting an LLM prompt into a reusable prefix and a volatile suffix so you can use provider-side prompt caching safely and effectively. The included helper finds the longest contiguous leading stable prefix, produces a conservative token estimate for that prefix, and compares it to a configurable minimum cacheable token threshold. Use the server-inserted example to plug this helper into tests and CI.

Expected behavior

The helper treats only a contiguous leading block of parts marked stable as a single cacheable prefix. Stable parts appearing after the first volatile message do not extend that leading prefix for reuse.
If the estimated prefix token count is below your configured minimum, the helper marks the prefix as not cacheable so you avoid treating tiny stable blocks as worth caching.
For modern models (GPT-5.6+), the helper recommends explicit-only mode when later stable parts exist or when the prefix is borderline; it suggests implicit mode only for robust leading stable blocks with no later stable sections.
Use cache keys to isolate reuse by customer/session/workspace and pick TTLs aligned with expected reuse: shorter TTLs for frequently changing prefixes and longer TTLs for durable reference material.

Limitations

The token estimator is intentionally small and conservative; it is a planning tool, not the provider tokenizer. For production measurement, use a tokenizer that matches the actual model.
The helper does not guarantee provider-side cache hits or quantify latency reduction. You must instrument the real system to measure cache hit rate, warm vs cold latency, and whether the prefix choice improves user-visible performance.
Prompt caching is sensitive to prompt shape: append-only updates or compaction can invalidate reuse even when much of the text looks similar.

Use this post as a checklist: identify ownership (what changes each request), measure prefix size before committing to cache, pick keys and TTLs to separate scope, prefer explicit-only when later stable blocks exist, and verify with real instrumentation. The server will append the canonical URL and insert the verified implementation example.

Working example

export function simpleTokenEstimate(text) {
  if (!text) return 0;
  // Very small, deterministic estimator: split on whitespace and punctuation groups.
  // This is intentionally conservative and inexpensive; real apps should use a tokenizer
  // matching the actual model.
  return String(text)
    .trim()
    .split(/\s+|(?=[.,;:!?()\[\]{}"'])|(?<=[.,;:!?()\[\]{}"'])/)
    .filter(Boolean).length;
}

export function analyzePromptParts(parts, { model = 'gpt-5.6', minCacheableTokens = 64 } = {}) {
  // parts: array of { role: 'system'|'developer'|'user'|'assistant', text: string, stable?: boolean }
  // Determine the longest leading contiguous stable prefix (from start) that can be cached.
  if (!Array.isArray(parts)) throw new TypeError('parts must be an array');

  let prefixParts = [];
  let prefixTokenCount = 0;
  for (const p of parts) {
    if (!p || typeof p.text !== 'string') throw new TypeError('each part must have a text string');
    const isStable = Boolean(p.stable);
    if (!isStable) break; // only contiguous leading stable parts are cacheable as one prefix
    prefixParts.push(p);
    prefixTokenCount += simpleTokenEstimate(p.text);
  }

  const prefixText = prefixParts.map(p => p.text).join('\n');
  const isCacheable = prefixTokenCount >= minCacheableTokens;

  // Model-specific guidance: on gpt-5.6+, explicit-only is recommended when suffixes are volatile.
  const major = String(model).toLowerCase();
  const prefersExplicit = major.includes('gpt-5.6') || major.includes('gpt-6') || major.includes('gpt-5.');

  // Suggest cache mode: if there are trailing stable parts later (non-leading), explicit is safer.
  const hasLaterStable = parts.slice(prefixParts.length).some(p => p.stable);
  const suggestedMode = (!isCacheable || hasLaterStable) ? 'explicit' : (prefersExplicit ? 'implicit' : 'explicit');

  const contiguousLeadingStableCount = prefixParts.length;

  return {
    prefixText,
    prefixTokenCount,
    isCacheable,
    minCacheableTokens,
    suggestedMode,
    // top-level convenience for tests and callers
    contiguousLeadingStableCount,
    reason: {
      contiguousLeadingStableCount,
      hasLaterStable,
      model,
    }
  };
}

export function chooseCacheKey(baseKey, { scope = 'global', id = null } = {}) {
  // Simple helper to build a cache-accounting key. Do not include secrets here.
  if (typeof baseKey !== 'string') throw new TypeError('baseKey must be a string');
  const safeScope = String(scope).replace(/[^a-z0-9_:-]/gi, '_');
  if (id == null) return `${safeScope}:${baseKey}`;
  return `${safeScope}:${id}:${baseKey}`;
}

export default { simpleTokenEstimate, analyzePromptParts, chooseCacheKey };

Read the full article: https://chriseugenerodriguez.com/blog/prompt-prefix-caching-decision

DEV Community

Decide which prompt prefix to cache vs re-run to reduce LLM latency

Working example

Top comments (0)