Learn how self-hosting NVIDIA Run:ai on private Kubernetes clusters unlocks dynamic fractional GPUs, gang scheduling, and automated workflow efficiency.
When enterprise engineering teams build private artificial intelligence infrastructure, they quickly collide with a frustrating physical reality. GPUs are extraordinarily fast, yet they sit idle for most of their operational lifecycle. In our work as a custom software development agency, we frequently audit client compute estates where multi-million-dollar clusters of NVIDIA H100 or H200 accelerators run at average utilization rates of just 15% to 25%. Data science teams lock full cards for interactive model exploration. Automated batch jobs fail to schedule due to fragmentation, and production inference deployments demand dedicated hardware to protect against out-of-memory crashes.
Cloud hyperscalers charge steep premiums for isolated GPU instances, while public cloud APIs raise continuous data privacy and compliance alarms for sensitive enterprise workflows. For financial services, healthcare, and sovereign enterprise clients, public cloud endpoints are simply not an option. Bringing hardware back on-premise or into private data centers solves data sovereignty, but it introduces a massive management layer. Vanilla Kubernetes lacks the internal scheduling primitives to slice GPUs, share device memory safely, or enforce cross-department quotas with priority borrowing.
This is why self-hosting NVIDIA Run:ai has emerged as the definitive architecture for enterprise AI platforms. Following NVIDIA's decision to open source the full NVIDIA Run:ai platform within the NVIDIA DSX AI Factory stack, engineering teams can now deploy production-grade GPU orchestration directly inside their private Kubernetes clusters without cloud lock-in. By deploying native control planes, KAI Schedulers, and CUDA-level GPU fractioning operators, organizations transform static server nodes into an elastic, multi-tenant compute fabric.
In this guide, we walk through the technical architecture, dynamic memory fractioning mechanisms, workflow automation integrations, and deployment strategies required to self-host NVIDIA Run:ai on private infrastructure. Drawing from client implementations across air-gapped environments and hybrid private clouds, we share the real operational numbers, architectural trade-offs, and scheduling configurations that drive private GPU utilization past 80%.
Self-hosting NVIDIA Run:ai means installing the NVIDIA Run:ai control plane, KAI Scheduler, and GPU fractioning operators directly on your private Kubernetes infrastructure. It replaces standard integer-based GPU allocation with dynamic memory slicing, gang scheduling, and automated over-quota borrowing. Organizations deploy self-hosted Run:ai on-premise or in air-gapped environments to ensure strict data privacy, maintain complete compliance, and maximize hardware efficiency across data science and business automation workloads.
According to benchmark telemetry from production deployments, dynamic GPU fractioning and memory swapping with NVIDIA Run:ai increase average cluster utilization by up to 3.2x while cutting first-token latency by over 40 times on shared inference nodes.
Standard Kubernetes was designed for web services, microservices, and stateless containerized applications. Its core scheduling engine, kube-scheduler, views compute hardware as a pool of CPU cores, system memory, and integer devices. When a pod requests compute resources, Kubernetes evaluates available nodes and binds the pod to a host that meets the CPU and RAM thresholds. When a container requests an accelerator, kube-scheduler interacts with the NVIDIA GPU Operator using a binary integer resource model. A pod either gets zero GPUs or one full physical device.
+-----------------------------------------------------------------------------------+
| Standard Kubernetes Scheduling |
| |
| [ Pod A: Jupyter Notebook ] ---> Requests 1 GPU ---> Allocates 100% H100 (80GB) |
| (Actual Usage: 4GB VRAM) |
| |
| [ Pod B: Small Inference ] ---> Requests 1 GPU ---> BLOCKED (No Integer GPU) |
+-----------------------------------------------------------------------------------+This integer-only abstraction creates severe structural waste across private clusters. Consider a typical corporate AI workspace where 30 data scientists require interactive Jupyter notebooks to inspect datasets and prototype code. Under vanilla Kubernetes, serving these 30 researchers requires 30 physical GPUs. During model setup, data ingestion, and code editing, those GPUs process almost no CUDA operations, leaving gigabytes of High Bandwidth Memory entirely unutilized. Yet, because each pod holds a exclusive device lock, no other workload can touch the hardware.
Inference microservices compound this inefficiency. An embedding model or a light classification service may require only 6GB of virtual memory and minimal compute cycles. Under native Kubernetes scheduling, that tiny service consumes a full 80GB H100 card, stranding 74GB of high-speed memory. If engineering teams attempt to manually co-locate containers on a single host without isolated device contexts, a memory spike in one process triggers an Out Of Memory error that crashes every process on the board.
Standard Kubernetes also lacks advanced queue mechanics. It has no native understanding of gang scheduling, where all parallel workers in a distributed training job must be allocated simultaneously or held in queue to avoid deadlocks. It offers no capacity borrowing across department quotas, meaning Team A's idle allocations cannot be automatically borrowed by Team B's high-priority training runs. To solve these core limitations, enterprise infrastructure requires a dedicated, GPU-native orchestration layer.
To build a resilient private deployment, you must understand how the core components of the NVIDIA Run:ai architecture interact within a private Kubernetes estate. The platform splits responsibilities between a centralized management surface and lightweight, in-cluster execution operators.
The architecture consists of three principal logical layers:
kube-scheduler for GPU-targeted pods, the KAI Scheduler is an open-source, high-throughput scheduling loop (maintained as a Cloud Native Computing Foundation sandbox project under Apache 2.0). KAI evaluates node topology, NVLink interconnects, device memory availability, gang constraints, and organizational priority rules before making placement decisions.+-----------------------------------------------------------------------------------+
| Self-Hosted NVIDIA Run:ai Architecture |
| |
| +-----------------------------------------------------------------------------+ |
| | Run:ai Control Plane | |
| | (REST / CLI APIs | Tenant Management | RBAC | Quota Policies) | |
| +-----------------------------------------------------------------------------+ |
| | |
| v |
| +-----------------------------------------------------------------------------+ |
| | KAI Scheduler Core Engine | |
| | (Gang Scheduling | Topology Awareness | Over-Quota Borrowing Rules) | |
| +-----------------------------------------------------------------------------+ |
| | |
| v |
| +-----------------------------------------------------------------------------+ |
| | GPU Fractioning Operator & Drivers | |
| | (CUDA Memory Slicing | Burstable Limits | Multi-Tenant Hardware Nodes) | |
| +-----------------------------------------------------------------------------+ |
+-----------------------------------------------------------------------------------+When an engineer or an automated workflow submits a workload through the CLI or API, the request bypasses generic Kubernetes scheduling rules. The KAI Scheduler calculates the exact memory footprint and compute ratio requested, checks the team's guaranteed quota allocation, and determines if idle resources from other departments can be borrowed. Once a placement decision is reached, the local cluster engine binds the container to the designated node and configures the driver runtime to enforce isolated execution bounds.
For engineering teams evaluating cloud-native enterprise platforms, combining this control surface with modern web interfaces simplifies operations. Our team frequently builds custom internal developer portals using our web application design & development practices to surface Run:ai cluster state directly to product teams alongside traditional web workloads.
The technical breakthrough behind self-hosting NVIDIA Run:ai lies in its dynamic GPU fractioning operator. Older methods of sharing GPUs, such as standard CUDA time-slicing, allow multiple processes to submit work to the same card, but they do not enforce hard VRAM isolation boundaries. If process A unexpectedly allocates more memory, process B experiences an out-of-memory crash. Hardware-level partitioning like Multi-Instance GPU (MIG) provides strict physical isolation, but MIG requires fixed, static repartitioning of the silicon that cannot instantly re-adjust when workloads start and finish.
NVIDIA Run:ai bridges this gap by implementing dynamic GPU fractions through software-defined memory management and native driver hooks. Workloads specify fractional resource requirements using Kubernetes Request and Limit syntax adapted specifically for GPU memory and compute cycles.
When deploying a containerized model, an engineer can declare a guaranteed memory request (for example, 0.25 GPU or 20GB VRAM) alongside a burstable memory limit (such as 0.80 GPU or 64GB VRAM). The KAI Scheduler guarantees that the container will always have access to its requested baseline memory allocation. If adjacent containers on the same physical card are operating below their limits, the scheduler permits the burstable container to expand its memory allocation up to its configured ceiling.
This dynamic memory behavior is essential for optimizing self-hosted operational costs. In our deep-dive analysis on optimizing local AI workflows with Qwen 3.8 27B and Claude models, we showed how local model performance hinges on efficient VRAM allocation. When running enterprise inference alongside interactive developer tools, dynamic memory fractioning eliminates hardware starvation without risking system stability.
For modern enterprise organizations, orchestrating GPU infrastructure is not just about supporting data science research. It is about powering high-throughput, automated business workflows. When enterprises deploy microservices such as NVIDIA NIM (NVIDIA Inference Microservices) or self-hosted autonomous agent frameworks, compute demands become volatile and event-driven.
Consider an enterprise invoice processing engine or an automated customer service resolution platform. During quiet operational hours, inference demand is minimal. During morning processing peaks, thousands of document pages or incoming customer requests arrive simultaneously. If each agent microservice requires a dedicated physical GPU, the operational expenses soar out of control.
By pairing self-hosted NVIDIA Run:ai with specialized model serving endpoints, enterprise systems achieve dense multi-tenant efficiency. NVIDIA NIM containers run seamlessly inside Run:ai fractional allocations. Small embedding models, guardrail checkers, and localized reasoning engines run on 0.10 GPU or 0.25 GPU fractions on shared H100 or H200 cards.
+-----------------------------------------------------------------------------------+
| Dense Multi-Tenant Workflow Execution on 1 Shared GPU |
| |
| +-----------------------------------------------------------------------------+ |
| | Physical NVIDIA H100 (80GB) | |
| | | |
| | [ NIM Guardrail (0.10 GPU / 8GB) ] [ NIM Embeddings (0.15 GPU / 12GB) ] | |
| | [ NIM Llama 3B (0.25 GPU / 20GB) ] [ Agent Task Worker (0.50 GPU / 40GB)] | |
| +-----------------------------------------------------------------------------+ |
+-----------------------------------------------------------------------------------+When building autonomous business tools, aligning model execution with dynamic resource orchestration prevents service degradation. Our practical guide to building custom AI agents for business outlines how modern agent architectures break complex queries into multiple sub-tasks. When those sub-tasks run against a self-hosted Run:ai cluster, the system dynamically spins up fractional worker containers, processes vector operations or local reasoning steps, and tears down execution contexts instantly.
Similarly, in our client engagements involving legacy ERP cloud integration for workflow automation, enterprise backends trigger batched GPU inference jobs every time inventory tables update or optical character recognition runs against incoming supply invoices. Self-hosted Run:ai schedules these background tasks with lower priority flags, allowing them to scavenge unused fractional VRAM from interactive team workspaces without interrupting active engineers.
Deploying NVIDIA Run:ai within a private, self-hosted environment requires careful preparation of host operating systems, driver stacks, storage subsystems, and Kubernetes control layers. The deployment follows a two-tier installation process: first deploying the administrative Control Plane application, followed by the Cluster Engine and KAI Scheduler operators.
Before launching the Run:ai installer, host worker nodes must meet precise hardware and software prerequisites. Operating systems such as Ubuntu 22.04 LTS or Red Hat Enterprise Linux 8.8+ must run on high-performance kernel profiles with Nouveau drivers completely disabled.
+-----------------------------------------------------------------------------------+
| Node Preparation Flow |
| |
| [ Bare-Metal Node (RHEL / Ubuntu) ] ---> [ Disable Nouveau & Enable IOMMU ] |
| | |
| v |
| [ Install NVIDIA Driver & Container Toolkit ] ---> [ Deploy GPU Operator & Helm ]|
+-----------------------------------------------------------------------------------+Key system prerequisites include:
The control plane can be installed on the same Kubernetes cluster as worker GPUs or on a separate, dedicated management cluster. The installation machine requires Helm 3.14+, Docker, and network access to the target host context.
Administrators execute deployment through the runai-installer CLI utility, feeding in a structured configuration values file that defines TLS termination, Fully Qualified Domain Name (FQDN) endpoints, single sign-on (SSO) integration, and database storage backends.
Once the control plane application is running and validated, the cluster engine is registered by applying the Run:ai Cluster Operator Helm chart to the target GPU worker cluster. This step deploys the KAI Scheduler, registers custom resource definitions (CRDs) for projects and workloads, and establishes secure TLS communication back to the control plane.
| Deployment Component | Role and Function | Hardware Minimum | Network / Storage Requirements |
|---|---|---|---|
| Control Plane App | Identity, RBAC, GUI, global policy definitions | 10 CPU Cores, 12GB RAM, 110GB Storage | Port 443 HTTPS ingress, FQDN, OIDC SSO |
| KAI Scheduler | Dynamic placement, gang scheduling, queueing | Runs on K8s Control Plane nodes | High-speed control network, low latency |
| GPU Fractioner | CUDA memory slicing and limit enforcement | Deployed on all GPU Worker nodes | NVIDIA Container Runtime, NVLink hooks |
| Cluster Engine | Metric collection, pod status reporting | System pods across worker nodes | Outbound access to Control Plane API |
For teams planning complex, enterprise-grade deployments, partnering with experienced infrastructure architects prevents misconfigurations. Through our custom software development services, our engineers handle host provisioning, helm chart customisation, storage mounting, and cluster validation to ensure high-availability operations from day one.
Managing compute in a multi-tenant enterprise environment requires balancing strict departmental budget boundaries with maximum cluster throughput. Traditional static resource quotas prevent teams from exceeding their allocated hardware limits, but they cause severe resource starvation when one department has urgent compute needs while another team's allocation sits completely empty.
NVIDIA Run:ai solves this through hierarchical quota management, dynamic priority scoring, and automated borrowing mechanisms.
Administrators structure the cluster into Departments and Projects, assigning each group a guaranteed baseline share of GPUs (for example, 24 GPUs to Financial Analytics and 16 GPUs to Supply Chain AI). Under normal operations, each department is guaranteed access to its baseline share.
However, if Customer Experience is currently using only 16 of its 24 allocated GPUs, the KAI Scheduler identifies 8 idle cards. If Financial Analytics submits a large batch fine-tuning job that exceeds its 24-GPU allocation, the scheduler marks the job as Over-Quota and grants it temporary access to the 8 idle GPUs.
When Customer Experience eventually submits a new workload that requires its guaranteed quota back, the KAI Scheduler executes priority preemption. It gracefully sends a terminate or pause signal to Financial Analytics' lowest-priority over-quota pods, frees the physical devices, and restores guaranteed capacity to Customer Experience within seconds.
For large-scale deep learning tasks running across multiple nodes using PyTorch distributed frameworks, partial scheduling causes catastrophic deadlocks. If a training job requires 16 GPUs across 2 nodes, but the cluster only has 12 available GPUs, launching half the job locks those 12 GPUs indefinitely while waiting for the remaining 4 cards. KAI Scheduler resolves this by implementing gang scheduling. The entire job array is evaluated as a single unit. The scheduler holds the job in queue until all 16 requested GPU slots are simultaneously available, preventing stranded compute deadlocks.
Designing these organizational policies and mapping them to automated workflows requires cross-functional planning. In our client work on workflow automation for business operations, establishing fair-share queuing mechanisms ensures background operational pipelines never starve interactive development teams.
While self-hosting NVIDIA Run:ai delivers massive efficiency gains and absolute data sovereignty, it is not the correct solution for every organization or project scale. Operating private orchestration layers carries significant technical complexity, capital expenditure, and maintenance responsibilities.
Building and running a self-hosted private GPU cluster involves several capital and operational cost buckets:
Organizations should skip self-hosting NVIDIA Run:ai if they meet any of the following criteria:
The single most common mistake engineering teams make during private Run:ai deployments is neglecting storage throughput and I/O limits. Slicing one H100 GPU into 4 or 8 fractional execution contexts allows 8 separate containers to run concurrent inference or training pipelines.
However, if all 8 containers pull heavy tensor weights and training data from a standard SATA SSD array or a slow NFS share, the disk I/O bottleneck chokes processing speeds. GPU cores end up stalled waiting for memory reads, completely negating the compute efficiency gained by fractional scheduling. Private clusters must pair dynamic GPU orchestration with parallel file systems or local NVMe storage arrays capable of multi-gigabyte per second read speeds.
To prevent these pitfalls, many growing organizations engage our team for tech partnership & consultation early in their platform planning. We assess network topology, storage throughput, and team workflows before a single rack is ordered.
In high-security enterprise sectors such as defense, banking, healthcare, and government operations, internal infrastructure must run in strictly isolated networks with zero internet connectivity. Self-hosted NVIDIA Run:ai fully supports air-gapped installation mode, allowing total control over data ingress and egress.
+-----------------------------------------------------------------------------------+
| Air-Gapped Security Architecture |
| |
| [ Private Registry (Harbor / Nexus) ] <--- (Pre-Loaded NGC Images & Helm Charts) |
| | |
| v |
| [ Air-Gapped Kubernetes Cluster ] <--- (Local PKI / OIDC SSO Authentication) |
| | |
| v |
| [ Run:ai Control Plane + Worker Nodes ] ---> Zero Outbound Egress Allowed |
+-----------------------------------------------------------------------------------+Deploying in an air-gapped environment requires staging all installation artifacts in local private registries prior to execution:
runai-installer utility and associated Helm repository archives are delivered via secure physical media or verified internal file transfers.Role-Based Access Control (RBAC) rules inside Run:ai ensure that data science teams are isolated from underlying cluster management commands. Users interact strictly with their assigned project spaces, preventing unauthorized access to adjacent team datasets, model checkpoints, or persistent volumes.
Once the air-gapped cluster is running, ongoing operational stability depends on proactive monitoring and driver management. Through our long-term maintenance & customer support services, we help enterprise teams handle zero-downtime upgrades, security patching, and capacity planning across critical private infrastructure.
Key takeaways
- Solve Stranded Compute: Vanilla Kubernetes integer scheduling strands VRAM, keeping private GPU utilization under 25%. Self-hosting NVIDIA Run:ai raises average utilization past 80%.
- Dynamic Memory Slicing: Software-defined fractional GPU allocations allow containers to share physical cards safely with guaranteed requests and burstable memory limits.
- Automated Quotas & Borrowing: Hierarchical team quotas guarantee baseline capacity for data science teams while allowing idle GPUs to be borrowed by background workflow automation.
- Air-Gapped Readiness: Full self-hosted control planes run entirely within private networks, providing zero-egress security and compliance for sensitive enterprise AI workflows.
- Requires Scale: Self-hosting makes financial sense for clusters of 16 or more GPUs with dedicated Kubernetes engineering support and high-speed NVMe storage backends.
Self-hosting NVIDIA Run:ai gives organizations complete operational sovereignty, ensuring that control plane metadata, user logs, project structures, and cluster telemetry remain inside private data centers. This is essential for air-gapped networks, financial institutions, and healthcare providers that cannot connect internal compute clusters to external public SaaS control planes due to strict compliance rules.
NVIDIA Run:ai uses a specialized GPU fractioning operator that integrates with the NVIDIA Container Toolkit to partition physical VRAM and execution contexts at the driver level. Workloads define a guaranteed memory request alongside a burstable memory limit, allowing containers to expand VRAM usage into idle capacity without causing out-of-memory crashes for adjacent processes.
The self-hosted control plane requires a dedicated Kubernetes node or system pool with a minimum of 10 CPU cores, 12GB of system RAM, and 110GB of persistent storage. The installer machine must run Helm 3.14+ and Docker, with HTTPS ingress and FQDN resolution configured across the internal enterprise network.
Yes, NVIDIA Run:ai offers full air-gapped deployment packages via the NVIDIA NGC Catalog. Platform administrators mirror all control plane container images, operator charts, and installer binaries to a private container registry, allowing cluster deployment and user authentication without any outbound internet access.
Gang scheduling ensures that distributed multi-node AI training jobs are only launched when all required physical GPU resources are simultaneously available. If a training run requests 16 GPUs across 2 nodes, KAI Scheduler holds the job in queue until all 16 slots are ready, preventing deadlocks where partial allocations lock hardware while waiting for remaining nodes.
Multi-Instance GPU (MIG) is a hardware-level feature present on A100 and H100 cards that physically divides silicon into fixed instances, requiring manual repartitioning. Run:ai fractional GPUs operate in software at the driver level, providing instant, dynamic memory slicing and burstable limits without requiring fixed hardware reconfigurations or node reboots.
When a department is not actively using its guaranteed baseline GPU quota, the KAI Scheduler reallocates those idle cards to other teams that have submitted over-quota jobs. If the owning department submits new workloads, the scheduler preempts the borrowed jobs according to priority rules and restores guaranteed capacity to the rightful owner.
NVIDIA NIM (Inference Microservices), vLLM, Triton Inference Server, and TGI run exceptionally well inside fractional allocations. Small embedding models, guardrail checkers, and light local LLMs can run on 0.10 GPU or 0.25 GPU fractions, allowing dozens of model endpoints to share a single physical accelerator card efficiently.
Maximizing private GPU efficiency is the defining technical challenge for enterprise AI platforms. As AI workflows transition from experimental research to mission-critical business automation, allowing multi-million-dollar accelerator hardware to sit idle under legacy scheduling models is an unacceptable operational tax.
Self-hosting NVIDIA Run:ai provides the architectural bridge between hardware isolation and cloud-like agility. By replacing integer device locks with dynamic memory fractioner operators, open-source KAI Schedulers, and automated over-quota borrowing, engineering leaders transform static racks into an elastic compute fabric capable of powering both interactive data science and dense agent workflows.
Deploying and fine-tuning these complex orchestration stacks requires deep expertise across Kubernetes networking, driver hooks, storage subsystems, and custom application pipelines. If your organization is evaluating private GPU infrastructure or building automated enterprise solutions, explore our comprehensive custom software development services or reach out to our engineering team to discuss your architectural strategy.
01 · RelatedAnalyze JetBrains Mellum 2.1, the 12B Mixture-of-Experts local coding model. Compare its local agentic performance, architecture, and cost against cloud enterprise APIs.
Read post
02 · RelatedExplore NVIDIA's new native CUDA Rust toolchains, cuda-oxide and cutile-rs. Learn how compile-time-safe GPU programming accelerates cloud infrastructure without sacrificing performance.
Read post
03 · RelatedExplore why Dhaka is rapidly becoming a global technology hub. Learn about the real costs, developer talent pool, and key strategies for outsourcing software development in Bangladesh in 2026.
Read postWe will reply in plain English within one business day, NDA on request. Discovery call is free.
We design and engineer software, mobile, and web products end-to-end. Send the brief, we will reply within one business day.
Start a projectWe send a short email whenever we publish a new field note or ship a studio update. No fixed schedule, no filler.
Unsubscribe in one click. We never share your address.