NeMo Guardrails gives LangGraph agents a separate safety proxy on OpenShift AI

Red Hat has published a deployable pattern for putting NVIDIA NeMo Guardrails between a LangGraph agent and its model endpoint on OpenShift AI. The design changes the agent’s BASE_URL to the guardrails service rather than embedding safety logic in the agent, letting LangGraph retain its conversation loop, system prompt and tool calls while the proxy inspects traffic in both directions. The accompanying banking example is available in Red Hat’s agentic starter kits repository. Red Hat Developer Example repository

Policy becomes a separately operated service

On OpenShift AI, the TrustyAI Operator provisions the service from a NemoGuardrails custom resource and ConfigMap. The example points its main role at an in-cluster vLLM endpoint and can assign dedicated NVIDIA NIM classifiers to content-safety and topic-control roles. Red Hat’s platform documentation confirms that NeMo Guardrails is deployed through the TrustyAI-managed CRD and supports input and output controls, standalone checks and transformations such as sensitive-data redaction. Red Hat Developer OpenShift AI documentation

The supplied configuration runs inexpensive regex checks first, then content-safety and banking-topic classifiers, and finally an output content-safety check. A failed rail short-circuits later checks and returns a configured refusal. Operators can replace the banking policy, add patterns or point each classifier role at a different model without rewriting the agent’s business logic. Example repository

Two traces expose different failure domains

The pattern also separates observability. MLflow records the LangGraph conversation, tool calls and the proxy as an ordinary model endpoint; OpenTelemetry exposes the individual rail spans, including which check blocked and how long each layer took. That split gives platform teams a way to distinguish agent behavior from policy behavior when tuning false positives or investigating refusals. Red Hat Developer

There is an operational caveat. OpenShift AI’s documentation warns that the /v1/chat/completions guardrails endpoint is not universally transparent and may modify or drop request parameters or response fields depending on configuration. The repository’s passthrough: true profile is designed for agent tool traffic, but teams should test their actual streaming, tool-call and error contracts before treating the proxy as a drop-in endpoint. Red Hat documents the standalone /v1/guardrail/checks path as an alternative when inference and policy enforcement need to remain discrete. OpenShift AI documentation

The practical change is architectural rather than cosmetic: safety policy becomes a versioned, observable OpenShift workload with its own models, configuration and failure behavior. That makes guardrail ownership separable from application-agent development—but also gives the platform team another service contract to operate and verify. Red Hat Developer