Senior Software Engineer, Datacenter Automation Platforms

Details

Collaborate with NVIDIA professionals responsible for building platforms that operate AI factories at scale. The team drives product roadmaps, architecture, cross-functional alignment, and implementation across multiple open-source and internal-facing products. It partners with executive leadership to align platform strategy, drive adoption of innovative software development approaches, mentor engineering talent, and turn successful domain solutions into reusable platform capabilities used across the organization.

Responsibilities

• Build self-service SDKs, workflow templates, reference implementations, and catalogs of reusable agents, tools, and connectors.
• Prioritize automation based on customer impact, frequency, operational risk, and reduction of human effort.
• Translate operational intent into versioned, testable policies covering triggers, prerequisites, permitted actions, approval requirements, maintenance windows, capacity limits, and recovery or blocking issue paths.
• Define common contracts for tools, state, events, and workflow execution.
• Establish versioning and compatibility standards, guide migration from fragmented implementations to shared services, and select runtime and durable execution architectures.
• Build agents that correlate hardware diagnostics, logs, metrics, topology, configuration, and incident history to investigate failures, propose evidence-backed diagnoses, and produce executable remediation plans.
• Enforce authorization independently of model output, retain action histories, and support human review, bounded retries, cancellation, and compensating actions.
• Automate provisioning validation, node quarantine, workload-aware maintenance, repairs, upgrades, and return to service.
• Integrate with Kubernetes, fleet management systems, schedulers, and relevant hardware interfaces.
• Address fleet-scale problems including partial failures, duplicate events, stale state, conflicting remediations, noisy telemetry, control-plane bottlenecks, and correlated rack or cluster failures.
• Establish shared evaluation, CI/CD, staged rollout, and observability capabilities across teams.
• Drive cross-team adoption through build reviews, migration plans, documentation, and hands-on partnership.
• Align infrastructure, networking, storage, security, and workload teams on shared priorities and contribution standards.

Requirements

• 8 or more years of experience in software engineering or a related field.
• Bachelor's degree in Computer Science, Engineering, Physics, Mathematics, or a comparable field, or equivalent experience.
• Expert-level knowledge of a systems programming language such as Go, Python, and/or Rust.
• Solid understanding of data structures and algorithms.
• Strong background in Linux system administration and management.
• Experience with cluster orchestration tools, including Kubernetes and/or SLURM.
• Understanding of software development patterns for agents and current security practices for agentic systems.
• Understanding of performance, security, and reliability in complex distributed systems, including system-level architecture, data synchronization, fault tolerance, and state management.

Preferred Qualifications

• Experience with open-source software projects, licensing, and governance.
• Proficiency in understanding and independently managing large-scale distributed systems.
• Deep knowledge of datacenter operations and GPU hardware.
• Advanced hands-on experience with cluster management systems, including Kubernetes and SLURM.
• Hands-on experience in Machine Learning Operations, training, and/or security.

Compensation and Benefits

• Expected base salary for Level 4: USD 184,000–287,500 per year.
• Expected base salary for Level 5: USD 224,000–356,500 per year.
• Eligible for equity and benefits.
• Applications accepted at least until November 14, 2026.
• NVIDIA is an equal opportunity employer committed to an inclusive work environment.