From GPU server to AI inference on OpenShift

A GPU server makes inference possible. A storage platform makes the model files available across deployments. OpenShift connects the two through its worker scheduling, Kubernetes storage interfaces, and Red Hat OpenShift AI model-serving components.

The important question is: what happens between downloading a model and running a prompt? This guide follows that path from bare metal to a serving endpoint.

1. Join the GPU server to the cluster#

A physical server joins OpenShift as a worker running Red Hat Enterprise Linux CoreOS (RHCOS). The exact procedure depends on how the cluster was installed. Supported options can include installer-managed bare-metal infrastructure or the node-image installation workflow on supported OpenShift releases.

Check the hardware compatibility, firmware, network connectivity, node installation method, registry access, and security requirements. In disconnected environments, the new node must reach internal image mirrors and cluster services. The GPU worker also needs enough host RAM and local disk space for containers and any model cache.

oc get nodes -o wide
oc get machineconfigpools
oc get csr

Verify the new worker's identity before approving any pending certificate requests. The worker should report Ready before installing accelerator software.

2. Expose NVIDIA GPUs to Kubernetes#

Install and configure Node Feature Discovery and a supported NVIDIA GPU Operator from the cluster's approved Operator catalog. Depending on the OpenShift and driver stack, kernel module support can involve Kernel Module Management. Confirm the compatibility of GPU model, driver branch, operating system, and OpenShift release.

The GPU Operator deploys the driver and device-plugin components. After they become healthy, Kubernetes can advertise GPUs as the extended resource nvidia.com/gpu.

oc get nodes -l nvidia.com/gpu.present=true
oc get clusterpolicy
oc describe node gpu-worker-01

For example, a worker with two unpartitioned GPUs can advertise nvidia.com/gpu: 2. Resource allocation differs when MIG or time-slicing is configured. Label dedicated accelerator nodes so hardware profiles and pod placement policies can select them.

oc label node gpu-worker-01 opsden.io/accelerator=nvidia

3. Connect the storage array and model repository#

The model artifacts contain weights, tokenizer files, and configuration data. Keeping these files on persistent storage prevents their loss when a model-serving pod is replaced. The source can be an object repository or a supported mounted filesystem.

An enterprise array can provide CSI-backed block volumes for workbenches and databases, and shared file volumes when its file service supports the required access mode. An S3-compatible repository needs an actual object-storage service. A block-only array does not become an S3 endpoint when a CSI driver is installed.

Example of a PVC using an existing StorageClass named enterprise-block:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-workspace
  namespace: ai-demo
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: enterprise-block
  resources:
    requests:
      storage: 250Gi

oc get storageclass
oc get pvc -n ai-demo

With HPE Alletra Storage MP B10000, a supported HPE CSI integration can provide PVC-backed storage over approved block protocols. An eligible NFS File Service can provide shared filesystem capabilities. Use the compatibility matrix for the actual array OS and CSI release. For a model registry, deploy an appropriate S3-compatible object service or use a compatible PVC-backed model source.

4. Install Red Hat OpenShift AI and choose a serving runtime#

Install Red Hat OpenShift AI through the Operator catalog. Configure the DataScienceCluster with the model-serving components needed for the installed version. KServe manages inference workloads, while a compatible vLLM runtime uses the NVIDIA GPUs for generative inference. Some serving modes have dependencies on other OpenShift components.

Create a hardware profile that requests the appropriate nvidia.com/gpu resource and selects GPU worker nodes. Size CPU, RAM, GPU count, and model runtime together. The installed version of OpenShift AI determines the exact dashboard labels and custom-resource fields.

oc get datasciencecluster
oc get pods -n redhat-ods-applications
oc get inferenceservices -A

5. Follow the model from storage into GPU VRAM#

When an inference service starts, KServe uses the configured model source to prepare artifacts for the runtime. With an object repository, model files are transferred before serving. With a mounted PVC, the runtime reads files through its filesystem path. vLLM then loads the model weights into GPU memory and prepares runtime state.

The array stores the model. GPU VRAM runs the model. Once the model is loaded, ordinary token generation mostly operates through GPU memory and serving software, rather than fetching weights from the array for each token.

A local NVMe cache can reduce repeated downloads when pods are replaced or scaled. The cache consumes worker disk space and still requires the runtime to load weights into VRAM. Cache support and configuration depend on the OpenShift AI version.

6. Estimate model memory before deployment#

The raw weight footprint depends on the number of model parameters and bytes per weight. An 8-billion-parameter model at BF16 needs about 16 GB for its weights alone. A 70-billion-parameter model at BF16 needs about 140 GB. Real deployments also consume memory for KV cache, runtime buffers, context length, and framework overhead.

This estimate is useful for screening model sizes. It does not prove that a particular multi-GPU deployment or quantized runtime configuration will start successfully.

7. Validate the whole data path#

Work from the underlying infrastructure toward the inference endpoint. Check node readiness, device-plugin allocation, storage provisioning, model artifact access, then serving readiness. When startup is slow, distinguish download time from model initialization time.

oc get nodes -o wide
oc describe node gpu-worker-01
oc get clusterpolicy
oc get pvc -A
oc get inferenceservices -A
oc describe pod <model-pod> -n ai-demo
oc logs <model-pod> -n ai-demo

If a deployment remains Pending, examine GPU allocation, resource limits, node selectors, and volume access modes. For model download failures, inspect repository URLs, credentials, internal CA certificates, and network routes. When a pod crashes while loading weights, check GPU memory and runtime compatibility first.

8. Plan for disconnected environments#

Disconnected clusters need mirrored OpenShift AI, GPU Operator, driver-related, and serving runtime images. Import model weights into an approved internal repository and verify the image pull secrets and certificate trust settings before starting deployment.

A model repository located inside the restricted environment also gives the organization a controlled process for approving model versions and managing storage use across multiple inference deployments.

Vendor references#

• Red Hat OpenShift AI documentation
• NVIDIA GPU Operator on OpenShift
• HPE Alletra B10000 CSI integration
• HPE Alletra B10000 NFS File Service