Shopify turns production failures into daily training data for a cheaper GraphQL agent

Shopify has described a production AI loop that converts failed GraphQL-agent conversations into new training trajectories every day, then serves the resulting specialized model with vLLM. The company says the system surpassed its frontier-model baseline while reducing estimated annual serving cost from about $27 million to roughly $1 million—a 96% reduction.

Those figures are Shopify’s estimates, not an independently reproduced benchmark. Even so, the engineering pattern is notable because it treats production failures as training inputs rather than accumulating every fix in prompts, tools and orchestration code.

From a rubric to a reward signal

The loop begins with a scored rubric for completeness, execution, response quality and safety. Shopify has product experts annotate random production samples, measures their agreement with Cohen’s kappa, and revises ambiguous criteria before calibrating automated judges. The team then backtests those judges against earlier A/B tests and deliberately degrades specific behaviors to check whether the matching score responds.

Before changing model weights, an automated research loop proposes edits to prompts, tool definitions or harness code, evaluates each change against the judge, and keeps only improvements. Once that route plateaus, low-scoring anonymized conversations become “hard negatives.” Multiple frontier reasoning models critique each failure, an arbiter combines their proposed repairs, and the conversation is replayed. Passing replays become reinforcement-learning trajectories; unresolved cases go to human annotators.

Training daily, serving through vLLM

Shopify says training runs in two stages: supervised fine-tuning on complete repaired trajectories, followed by group relative policy optimization using the calibrated judge as the reward signal. A daily pipeline adds new trajectories, repeats full-parameter fine-tuning over new and previous examples, and runs reinforcement learning again. PyTorch supplies distributed training across tensor, context and data parallelism.

The production example is a GraphQL agent that writes and executes Shopify Admin API queries at up to 2,000 requests per minute. Shopify serves it through vLLM, citing continuous batching and suitability for tool-heavy workloads.

The team also trained a small set of “gist” token embeddings to replace much of the agent’s static system prompt while freezing the model weights. Shopify reports that this compressed roughly 6,000 prompt tokens to about 1,500 learned tokens with no measured loss on its judge. At 350 requests per minute, its load test showed about 19% lower time to first token and 38% lower end-to-end latency, alongside 16% more requests per second and an estimated 14% reduction in GPUs required for the same traffic.

What platform teams should take from it

The transferable idea is not a single optimizer. It is the control loop: define quality with human agreement, validate the judge against real outcomes, improve the harness first, promote only repaired failures into training data, and keep serving economics in the same measurement system as model quality. For teams running vLLM on an enterprise platform, Shopify’s account offers a concrete architecture to test—but its cost and quality gains should be treated as workload-specific until reproduced on their own traffic.