Introduction

The Challenge

The company’s existing VM-based infrastructure was unable to scale efficiently and required high overprovisioning to accommodate peak loads. As they planned a new AI-enabled product iteration, they needed GPU capabilities, modern automation, and a scalable platform. They also aimed to reduce reliance on Microsoft technologies and prepare for future hybrid cloud flexibility, all while keeping sovereignty and independence at the core of their architecture strategy.

They aimed to:

• Stay sovereign and independent by remaining on-premises for now, while keeping the application stack as portable as possible to enable a future move to public or hybrid cloud
• Make use of their existing NVIDIA DGX appliances in combination with officially supported NVIDIA GPU Operators
• Avoid vendor lock-in and minimize license costs by investing in open-source technology and internal engineering
• Transition away from Microsoft dependencies

The existing team lacked deep Kubernetes knowledge, and the business model required elastic resource management while maintaining high security standards.

The Solution

The solution provider architected and delivered a greenfield, dual-datacenter Kubernetes setup using Rancher RKE2, integrating NVIDIA DGX hardware via GPU Operators. Key architectural decisions included:

• Cilium for networking and security
• Cilium Cluster Mesh for multi-site resilience and workload distribution
• ArgoCD-based GitOps for full automation, consistency and security
• Standard observability stack with Prometheus, Grafana, Loki, and Alloy
• CIS Benchmark-based hardening for security best practices

The platform was built to be portable, cost-efficient, and easy to operate, with strong support for both production and development needs.

Implementation Process

• Started with a collaborative architecture workshop and MVP cluster
• Migrated workloads from VMs to containers (customer-led, solution provider supported)
• Rolled out multiple environments via GitOps automation
• Enabled GPU isolation and inference support within shared prod/dev cluster
• Delivered training, DevOps enablement, and Day 2 operational support in a hybrid collaboration model

Results Achieved

• Scalable platform ready for AI inference workloads
• Improved portability for future cloud transitions
• Reduced complexity to enable faster customer onboarding to the new platform and boost developer productivity.
• Security posture elevated with CIS Benchmarks and GitOps practices
• Deployment times reduced significantly
• Developer experience and autonomy significantly improved
• Operations streamlined with a leaner ops footprint
• Maintenance windows reduced from 8 hours to just 1 hour an 87.5% reduction
• Better developer experience with easier, faster deployments

Lessons Learned

• The greenfield, iterative delivery model works extremely well when customers are open to collaboration
• NVIDIA GPU integration is seamless—until operator updates break things. Good vendor communication is key
• Joint team ownership with clear domain knowledge (business + engineering) delivers the best results