Senior Machine Learning Engineer

Senior ML Systems & Agentic Software Engineer (Contract)

Remote - United states

12+ Months Contract

Must-Have Qualifications & Eligibility:

• Work Authorization & Seniority: Open exclusively to US-based candidates currently authorized to work in the US without current or future sponsorship; requires a minimum of 10+ years of professional software experience (5+ years in production Python).
• Distributed ML & GPU Stacks: Mandatory hands-on experience debugging multi-node GPU cluster jobs, rank communication failures, and production proficiency with PyTorch alongside large-scale frameworks (Megatron-LM, DeepSpeed, TorchTitan, or vLLM).
• Profiling & Agent Engineering: Proven track record delivering production LLM agentic/tool-calling systems combined with direct experience in GPU profiler/trace analysis and cluster failure diagnosis.

Role Overview

CLIENT is building production-grade agentic software to streamline how engineering teams develop, optimize, deploy, and operate machine learning workloads at scale on CLIENT Instinct™ GPUs.

In this senior individual contributor contract role (equivalent to SDE 5), you will architect and deliver autonomous, production-ready Python agents deeply integrated into the CLIENT ML stack. Your systems will automate high-value, complex engineering workflows: analyzing distributed cluster failures, parsing telemetry and trace profiles, diagnosing throughput regressions, and eliminating operational toil across multi-node GPU deployments.

Key Responsibilities

• Agent Architecture & Frameworks: Design and implement modular Python agent runtimes, orchestration loops, custom tool adapters, state/session management, and robust CLI/service interfaces.
• LLM Orchestration & Reliability: Build deterministic, production-grade workflows using frameworks such as LiteLLM, DSPy, LangChain, or LlamaIndex. Enforce strict JSON Schema outputs, budget/token caps, allowlisted tool calling, and grounded, evidence-backed root cause reports.
• Client Ecosystem Integration: Connect intelligent agents into the Client’s ROCm™ ecosystem, PyTorch, distributed training/serving runtimes, telemetry feeds, cluster log collectors, and hardware/software profiling utilities.
• Evaluation & CI/CD Pipelines: Design comprehensive evaluation infrastructure, including golden fixture datasets, regression suites, LLM-as-a-judge benchmarks, and automated validation gates before rollout.
• Cluster Operations & Diagnostics: Develop agents that autonomously triage multi-node distributed training and inference issues—such as collective communication hangs (NCCL/RCCL), OOMs, silent numerical divergence, host-device bottlenecks, and Slurm/Kubernetes job crashes.
• Packaging & Security: Deliver production-ready artifacts utilizing Docker and Kubernetes, complete with comprehensive runbooks, defense-in-depth egress filtering, and secure credential handling suitable for customer-isolated environments.

Required Qualifications

• Education: Bachelor’s degree or higher in Computer Science, Computer Engineering, Electrical Engineering, or equivalent practical industry experience.
• Experience: 10+ years of professional software engineering experience, with 5+ years dedicated to building and scaling production Python distributed systems.
• Distributed ML Expertise: Hands-on experience debugging and running multi-node training or high-throughput inference across GPU clusters, including log/metric correlation across nodes and ranks.
• Framework Mastery: Deep hands-on proficiency with PyTorch and large-scale acceleration/serving stacks (e.g., DeepSpeed, Megatron-LM, TorchTitan, vLLM, TensorRT-LLM, or TGI).
• Failure Mode & Performance Diagnostics: Demonstrated track record resolving real-world cluster anomalies: throughput degradation, memory leaks/fragmentation, collective communication deadlocks, launch misconfigurations, and profiler trace analysis (PyTorch Profiler, Perfetto).
• Agentic Systems Delivery: Proven history of shipping agentic workflows, autonomous tool-use systems, or complex LLM chains into production or near-production environments.
• Engineering Rigor: High standards for clean design, test-driven development, typing (mypy), CI automation, and clear technical documentation.

Preferred Qualifications

• Direct experience with the CLIENT ROCm™ software platform and ROCm profilers/debuggers (Omnitrace, Rocprof).
• Open-source contributions to major ML or LLM orchestration repositories (e.g., vLLM, PyTorch, Hugging Face, DSPy).
• Advanced performance tuning background: Model Flops Utilization (MFU) optimization, 3D parallelism strategies (TP/PP/DP/EP), roofline modeling, and custom kernel/communication overlap analysis.
• Experience with HPC cluster orchestrators (Slurm, Kubernetes/KubeFlow) and high-speed interconnect telemetry (InfiniBand, RoCE).
• Knowledge of agent security posture: prompt injection mitigation, network isolation, least-privilege tool execution, and local VPC execution.

Scope Boundaries (What This Role Is Not)

• Not Low-Level Driver Development: Does not involve Linux kernel drivers, KFD, or bare-metal graphics firmware.
• Not Pure AI Research: Focused on systems engineering, software reliability, and operational tooling rather than training foundational models from scratch.
• Not Junior/Mid-Level: Requires end-to-end architectural independence, ambiguity management, and senior technical leadership.