ML Cluster Operations Engineer in Las Vegas

Energy Jobline is the largest and fastest growing global Energy Job Board and Energy Hub. We have an audience reach of over 7 million energy professionals, 400,000+ monthly advertised global energy and engineering jobs, and work with the leading energy companies worldwide.

We focus on the Oil & Gas, Renewables, Engineering, Power, and Nuclear markets as well as emerging technologies in EV, Battery, and Fusion. We are committed to ensuring that we offer the most exciting career opportunities from around the world for our jobseekers.

Job DescriptionJob Description

ML Cluster Operations Engineer (Slurm / K8s)

At TensorWave, we’re leading the charge in AI compute, building a versatile cloud platform that’s driving the next of AI innovation. We’re focused on creating a foundation that empowers cutting-edge advancements in intelligent computing, pushing the boundaries of what’s possible in the AI landscape.

About the Role:

We are seeking an exceptional Machine Learning Engineer who has made training and AI workload scheduling a specialty. This is a senior-level role for someone who has significant experience managing distributed machine learning workloads at scale using Slurm and/or Kubernetes.

As a technical visionary and hands-on expert, you will lead the evolution of our managed Slurm and Kubernetes offerings, as well as internal health checking and cluster automation.

Key Responsibilities:

• Manage and iterate our containerized Slurm (Slurm-in-Kubernetes) solution, including customer configuration and deployment.

• Work closely with our engineering team to develop and maintain CI and automation for managed offerings.

• Ensure healthy cluster operations and uptime by implementing active and passive health checks, including automated node draining and triage.

• Help profile and debug distributed workloads, from small inference jobs to cluster-wide training.

• Establish best practices for running jobs at scale, including monitoring, checkpointing, etc.

• Mentor and upskill ML engineers in best practices.
  

Qualifications:

Must-Have:

• 5+ years of experience in cloud infrastructure, HPC, or machine learning roles.

• Significant hands-on experience with Slurm in production HPC/ML environments, including understanding of setup/configuration, enroot (pyxis), modules, and MPI.

• Strong knowledge of distributed ML and frameworks, such as Python, PyTorch, Megatron, c10d, MPI, etc.

• Understanding of node lifecycle, including health checks, prolog / epilog scripts, and draining.

• Deep understanding of security, compliance, and resilience in containerized workloads.

Nice-to-Have:

• 3+ years of hands-on Kubernetes experience, including deep knowledge of the Kubernetes API, internals, networking, and storage.

• Proficiency in writing Kubernetes manifests, Helm charts, and managing releases.

• Experience with DAGs using K8s tools such as Argo Workflows.

• Foundation in networking, especially as it pertains to RDMA, RoCE, and Infiniband.

• Experience with low level kernel libraries, such as CUDA and Composable Kernel.

• Contributions to open-source projects or ML/AI tooling.

What Success Looks Like

• A production-grade integrated Slurm platform that can support thousands of GPUs, with self-healing, scaling, and strong observability.

• Infrastructure is resilient, secure, resource-optimized, and compliant.

• Best practices and tooling are well-documented, standardized, and continuously improved across the company.

• Make GPUs go Brrrrrrr
  

What We Bring:

Stock Options

100% paid Medical, Dental, and Vision insurance

Life and Voluntary Supplemental Insurance

Short Term Insurance

Flexible Spending Account

401(k)

Flexible PTO

Paid Holidays

Parental Leave

Mental Health Benefits through Spring Health

If you are interested in applying for this job please press the Apply Button and follow the application process. Energy Jobline wishes you the very best of luck in your next career move.