Agent Evaluation and Self-Improvement Engineer

Details

Autonomous agents now do real work across NVIDIA, and every piece underneath them keeps changing: models, harnesses, tools, skills, and plugins. Evaluation is how a team knows whether a change made an agent better or quietly made it worse. This role will set evaluation strategy at every layer, from individual skills and MCP servers to whole agents and products, and design how agents use evaluation results to improve themselves.

Agents can already rewrite the skills and plugins they run on, then test whether the change helped. The role focuses on making that loop trustworthy and determining how agents can improve autonomously while proving that improvements are real.

Responsibilities

• Build evaluation that works consistently across harnesses, with automated checks on every change to catch regressions before users do.
• Mine production traces for failures, new usage patterns, missing skills and tools, cost and latency hot spots, and successful runs. Turn them into evaluation cases and improvement signals.
• Evaluate evaluation datasets for real-world usage coverage, duplicate cases, and similarity to production data.
• Build self-improvement loops for evolving harnesses, allowing agents to propose changes to their skills, plugins, and harness configuration. Use sandboxed evaluations to test each change, ship only measurable gains, require human sign-off, and enforce limits agents cannot change.
• Create realistic enterprise environments, including sandboxed replicas of issue trackers, code repositories, chat, and internal APIs, populated with synthetic or anonymized data so agents can safely perform long, end-to-end tasks.
• Measure multi-agent swarms as a team, including work distribution, handoff success, permission enforcement at every hop, task duration, and token usage.
• Keep evaluations honest by calibrating model and agent judges against human labels.
• Detect agents that cheat, such as working backward from expected answers or accessing data and tools they should not access.
• Collaborate with product, platform, and hardware teams across NVIDIA.
• Help shape open evaluation standards and benchmarks across the industry.

Requirements

• Bachelor’s or master’s degree in Computer Science, Engineering, or a related field, or equivalent experience.
• 12 or more years of experience building large-scale systems, including evaluation systems for machine learning or agents and the data pipelines that support them.
• Hands-on experience building and evaluating agents on an open or closed agent harness.
• Strong understanding of experiment design, including statistical significance, variance across repeated runs, and A/B testing.
• Experience setting technical direction across teams as a senior individual contributor.
• Proficiency in Python, Go, Rust, or a similar programming language.

Preferred Qualifications

• Experience building self-improving or self-evolving agent systems that improve from their own evaluation results.
• Experience shipping an agent benchmark, evaluation framework, or simulated test environment adopted by other teams.
• Experience detecting reward hacking or evaluation gaming, or building oversight for systems that modify themselves.
• Experience building domain evaluations, such as evaluations for chip design or hardware verification.

Compensation and Benefits

The base salary is determined by location, experience, and the pay of employees in similar positions. The base salary range is USD 224,000–356,500 for Level 5 and USD 272,000–431,250 for Level 6. The position also includes eligibility for equity and benefits.

Applications will be accepted at least until October 20, 2026. This posting is for an existing vacancy. NVIDIA uses AI tools in its recruiting processes. NVIDIA is an equal opportunity employer committed to an inclusive work environment.