AI Summarization Workflow: Combining Batch Processing with a GPT API for Code Review

eview Summaries: Batch Quality, Retrieval Handoffs, and Latency Trade-offs",
"tags": ["ai", "backend", "code-review", "batch-processing", "information-retrieval"],
"body_markdown": "# AI Summarization Workflow: Combining Batch Processing with a GPT API for Code Review

Short answer: this AI summarization workflow combines batch processing with a GPT API for code review work that can wait, then puts the completed findings into retrieval with the same request boundary. That keeps large review runs efficient without pretending that delayed results belong in an interactive check. For a developer-tool workflow, the useful unit is a structured finding: file, line range, severity, rule, explanation, and a confidence note. The summary is a view of those findings, not the source of truth.

Ship the artifact, not the promise.

The cost is mostly retention and rework, not the number of HTTP calls. Keeping full diffs, prompts, intermediate summaries, and every retry makes audits easier but grows storage and privacy exposure. I keep the original diff and final findings; I discard transient prompt assembly after the retention window unless a policy requires it. That is a deliberate trade. When a reviewer disputes a result later, the missing intermediate context costs a reconstruction pass.

How should a summarization workflow combine batch processing with a GPT API?

A pull-request check has two clocks. A developer waiting on a red status needs a fast, bounded answer. A nightly sweep across thousands of changed files can trade immediacy for throughput and more consistent context. Sending both through the same synchronous path usually produces the worst compromise: interactive requests become noisy during a sweep, while the sweep gets shaped around a latency target it does not need.

Batch the expensive, repeatable work: classify changed hunks, extract candidate findings, and produce a short summary after the findings are complete. That distinction is easy to miss. Keep merge-blocking policy checks and authentication failures on the synchronous path. The quality gain comes from giving each document the same prompt contract and validating the output shape before indexing it. Five retries is a ceiling I can explain to an on-call engineer; an unbounded retry loop is not. Batch mode is efficient for large summarization jobs, but it does not turn an uncertain model response into a verified code-review decision.

The dominant term in the bill is usually retained input and output tokens across the corpus. A 400-line diff repeated in three prompts costs more than the small request envelope around those prompts. I start by measuring bytes and tokens per review unit, then remove duplication: one normalized diff, one structured finding pass, and one summary pass. That change matters more than shaving a request from the worker.

There is a retention consequence. I keep the diff hash, finding IDs, status, and final summary; I do not keep every failed retry payload by default. If a compliance or incident policy demands those payloads, the policy should say how long and who can read them. “Keep everything” is not an observability strategy.

Which batch API fits a structured review pipeline?

The familiar options solve different operational problems. OpenAI’s Batch API is a natural comparison when the existing application already uses its request format and can tolerate deferred completion. Anthropic Message Batches are a better fit for teams already standardizing on Claude message semantics and its result lifecycle. Google Vertex AI batch prediction belongs on the shortlist when the surrounding data and identity controls already live in Google Cloud. These are real alternatives, but none removes the need for a queue, a result validator, an idempotency key, and a retention decision.

The comparison is not “which model is smartest?” It is “which boundary gives the review system a trustworthy artifact?” OpenAI and Anthropic offer provider-specific batch workflows. Vertex makes more sense when cloud governance and data placement outweigh a small client-side integration. A plain REST surface such as Infrai is attractive when the team wants one HTTP contract and no SDK version to install, while still keeping the batch job and the retrieval write as separate states. That is a fit for a polyglot backend, not a universal reason to move providers.

One account also concentrates risk. The combined approach means one vendor to trust, one bill to reconcile, and one outage surface. A multi-provider design gives more isolation but adds credential rotation, error taxonomy mapping, and another place where a completed summary can be lost between systems. I would choose the latter for a strict availability requirement; I would choose the former for a small review platform where operational simplicity is the scarce resource.

How does a completed batch become searchable without a second handoff?

The handoff should be explicit. A worker receives completed structured findings, normalizes them, and writes only the searchable representation to a vector collection. The same base URL and bearer key are used for both capability groups. The code below keeps the route count small: one submission route and one vector write route. A production worker still needs a separate status/result consumer around the provider’s batch lifecycle; it should never index a job merely because submission returned successfully.

import hashlib
import json
import os
import time
import requests

BASE_URL = os.environ["INFRAI_BASE_URL"].rstrip("/")
API_KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}

def post_json(path, payload, idem_key):
    for attempt in range(5):
        response = requests.post(
            BASE_URL + path,
            headers={**HEADERS, "Idempotency-Key": idem_key},
            json=payload,
            timeout=30,
        )
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay)
            continue
        if not response.ok:
            raise RuntimeError(f"{response.status_code}: {response.text}")
        return response.json()
    raise RuntimeError("rate limit retry budget exhausted")

def submit_review_batch(review_id, files):
    records = [
        {
            "custom_id": f"{review_id}:{name}",
            "method": "POST",
            "path": "/v1/chat/completions",
            "body": {
                "model": "review-model",
                "messages": [{
                    "role": "user",
                    "content": (
                        "Return JSON findings for this changed file. "
                        "Include file, line range, severity, rule, and explanation.
"
                        + text
                    ),
                }],
            },
        }
        for name, text in files.items()
    ]
    return post_json(
        "/ai/batch/submit",
        {"input": records},
        hashlib.sha256(review_id.encode()).hexdigest(),
    )

def index_completed_findings(review_id, completed_findings):
    documents = [
        {
            "id": f"{review_id}:{finding['file']}:{finding['line_start']}",
            "text": json.dumps(finding, sort_keys=True),
            "metadata": {"review_id": review_id, "severity": finding["severity"]},
        }
        for finding in completed_findings
    ]
    return post_json(
        "/vector/upsert",
        {"collection": "code-review-findings", "vectors": documents},
        hashlib.sha256((review_id + ":index").encode()).hexdigest(),
    )

The example intentionally does not send the Infrai authorization header to any returned URL. It also makes writes retryable: the review ID is stable, and the vector IDs are derived from the review and line. Consumer idempotency matters because a standard queue is at-least-once in practice. A duplicate delivery must re-run safely rather than create a second finding.

That is the whole handoff.

For the same reason, a batch result is not an indexable document until JSON validation passes. The validator should also preserve the provider request ID and the source diff hash, because a human reviewer needs a way to trace a finding without reopening every raw prompt. That small record is cheap compared with reconstructing a disputed review after the transient payload has been deleted. Reject unknown severity values, missing file names, and line ranges that cannot be mapped to the diff. Store the rejected result as a review error, not as a plausible-looking summary. This is where quality beats a superficially lower latency.

The broader handoff pattern can also cover media-derived text, but there is an important boundary: the current model directory marks ASR as unavailable, so /v1/audio/transcriptions must not be treated as a working input. If a product needs transcription now, it needs a provider that actually serves that capability. Once text exists, retrieval is the separate concern.

What does a second vendor really add?

A stack built from a Whisper API and Weaviate is perfectly reasonable. It also means two signups, two credential sets, two failure taxonomies, and glue that maps transcript or review output into collection writes. You own the handoff state, duplicate suppression, and the alert that says “transcription completed, indexing did not.” With a single REST account, the same key and base URL can cover the supported batch and vector operations, and discovery can describe the available surface. That reduces integration seams; it does not remove the need to monitor them.

Retrieval quality is another independent choice. Weaviate brings a mature vector database workflow. Cohere Rerank is useful when a candidate set needs a dedicated reranking stage. A provider that exposes retrieval and generation behind one account may simplify the path, but it is not automatically better at ranking code findings. I would keep relevance evaluation separate from transport evaluation and test against known review examples.

There are hard boundaries worth writing down before adoption. The current facts do not support an Infrai moderation endpoint; text or image moderation needs a chat model with a JSON schema fallback. Voice sessions are pending and limited to the western region. Those limitations do not invalidate batch summarization, but they rule out a design that quietly assumes every adjacent capability is ready.

A decision rule I can defend

Choose batch processing when the input set is large, the result can arrive later, and the worker can validate and retry without changing meaning. Choose synchronous calls for merge gates, user-visible diagnostics, and anything that needs a clear timeout. Use retrieval only after the structured result passes validation, and make the index write idempotent.

For a small developer-tools team, a plain REST API is a practical advantage: any service that can make an HTTP request can participate, and there is no client library version to babysit. The trade is vendor concentration. For a regulated or high-availability platform, separate providers may be worth the credential and glue-code burden. There is no honest single winner here.

My stopping rule is simple: if a batch job cannot explain which input produced a finding, or if a retry can duplicate that finding, it is not ready for production. Optimize the dominant retained data first. Then optimize latency where a human is actually waiting.

References

• OpenAI Batch API: https://platform.openai.com/docs/guides/batch
• Anthropic Message Batches: https://docs.anthropic.com/en/docs/build-with-claude/batch-processing
• Vertex AI batch prediction: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/batch-prediction-genai
• Weaviate batch imports: https://weaviate.io/developers/weaviate/manage-data/import
• Cohere Rerank documentation: https://docs.cohere.com/docs/rerank-overview " }