OpenShift AI field demo makes data sharding part of a distributed training pipeline

A new Red Hat AI Americas reference repository demonstrates an OpenShift AI 3.5 training workflow in which data preparation is part of the pipeline rather than an external prerequisite. The notebook generates synthetic images, a Kubeflow pipeline packages them into WebDataset tar shards, and the same pipeline starts a two-node distributed TrainJob.

The repository is an early field build, not a released or supported product. It currently contains a single initial commit, no tagged release and several environment-specific values that adopters must replace.

What the pattern shows

The example generates 20,000 labeled 256-by-256 JPEG images across 10 shape classes. Its packing step writes those samples as WebDataset tar archives, pairing each image with a class file, and records only a small manifest as pipeline state. The image bytes remain on a shared read-write-many volume.

The training step uses ResNet18 for one epoch with PyTorch Distributed Data Parallel. It launches two nodes with one GPU per node through OpenShift AI’s Trainer runtime and splits WebDataset input by node and worker. Training metrics return as a pipeline artifact written by the TrainJob.

This makes the handoff between data preparation and distributed execution explicit: the pipeline creates the shard layout first, then submits and waits for the training job. For teams building repeatable computer-vision experiments, that is more useful than a notebook that assumes a prepared data directory already exists.

The boundaries matter

The README assumes an OpenShift AI 3.5 cluster with MaaS, MLflow, Kueue, an RWX storage class, GPU nodes and the named CUDA/PyTorch Trainer runtime already installed. However, the demo does not call MaaS, does not place the project under Kueue and does not send metrics to MLflow. Those services are cluster assumptions rather than integrated parts of the workflow.

The checked-in manifests also name efs-csi and gp3-csi storage classes, an S3 bucket, an MLflow URL and a dashboard host from one sandbox. The repository explicitly tells users to change those values before applying the manifests elsewhere. WebDataset is installed manually onto the shared volume because the workbench package index used by the example does not provide it.

What to try

Platform teams can use the repository as a compact test of three interfaces: shared storage between workbench, pipeline and training pods; the pipeline’s ability to create and monitor a TrainJob; and multi-node access to the same sharded dataset. Before treating it as a reusable template, teams should parameterize the storage classes and external URLs, move dependency installation into a controlled image, and decide whether Kueue admission and MLflow tracking belong in the production version.