Machine Learning Performance Engineer (Offboard Training & Inference)

• We are looking for a performance engineer who specializes in making large-scale machine learning workloads fast and cost-efficient in the datacenter
• This role is focused on distributed training runs spanning many nodes, and high-throughput batch inference sweeping petabytes of real-world autonomy logs for auto-labeling, data mining, ground-truth generation, and evaluation
• The optimization target here is not tail latency on a vehicle - it is throughput, cluster goodput, and cost per unit of data processed. A training run that wastes 30% of its GPU-hours on stalled data loaders, or an offline inference sweep that takes a week instead of a day, directly slows down how fast the whole company can iterate
• You will own the gap between what our fleet of accelerators is theoretically capable of and what our workloads actually achieve: profiling across the stack, finding where the compute and the wall-clock time actually go, and closing the difference
• You will work at the intersection of accelerators, ML frameworks, and large-scale data infrastructure, partnering with the teams who own each layer to land wins that show up in training time-to-result and offline processing cost
• At Applied, we encourage all engineers to take ownership over technical and product decisions, closely interact with users to collect feedback, and contribute to a thoughtful, dynamic team culture
• Profile and optimize distributed training end to end - data loading and preprocessing, augmentation, kernel execution, gradient communication, and checkpointing
• Optimize large-scale offline and batch inference over petabyte-scale sensor logs: batching and scheduling strategies, quantization and low-precision execution, graph optimization, and accelerator saturation across long-running sweeps
• Establish roofline and performance models for our workloads, quantify the gap between achieved and theoretical performance, and stack-rank optimization opportunities by impact and effort
• Improve multi-node scaling efficiency: sharding and parallelism strategies, collective communication, interconnect utilization, and memory-bandwidth and kernel-fusion bottlenecks
• Drive cluster goodput - reduce GPU idle time from input pipeline stalls, storage and network I/O, scheduling gaps, stragglers, and failure recovery on long-running jobs
• Build the benchmarking, observability, and regression-detection tooling that keeps performance from silently degrading as models and code evolve
• Collaborate with engineers across functions to solve complex data and compute problems at scale
• Contribute to a team culture that values effective collaboration, technical excellence, and innovation

Benefits

• Health Insurance: Applied covers 100% of the cost and offers medical, dental, vision, and life insurance
• Fitness Stipend: Our wellness program supports your health and fitness goals. We pay for your trainers, gym memberships, massages, and more!
• 401(k) Match: 50% employer matching to help you save money for retirement
• Learning Stipend: We invest in your learning and growth by reimbursing you for educational and training expenses
• Parental leave: 12-week, fully paid parental leave to bond with a newborn child or new adopted or foster child
• Catered lunch, expensed dinner, on-site meals and snacks- A deep understanding of machine learning foundations, and the ability to develop technical solutions for problems with no established playbook
• Experience with distributed multi-node training at scale (FSDP, DeepSpeed, Megatron, NCCL, or equivalent), including diagnosing scaling inefficiency as node count grows
• Experience with high-throughput or batch inference systems (NVIDIA Triton Inference Server, TensorRT, ONNX Runtime, Ray, or similar)
• Excellent debugging, analytical, and problem-solving skills
• Fluency in Python and proficiency in C++ or another systems language
• Hands-on ML performance engineering experience: profiling, roofline analysis, throughput optimization, and root-cause investigation in production systems
• Deep familiarity with GPU or accelerator performance concepts - memory bandwidth, kernel launch overhead, occupancy, quantization, collective communication
• GPU kernel development experience: CUDA, Triton, CUTLASS, or hand-tuned attention implementations
• Experience with profiling toolchains such as Nsight Systems/Compute, PyTorch Profiler, or perf
• Experience with GPU scheduling and orchestration on Kubernetes, Slurm, or Ray, including multi-tenant cluster utilization
• Experience with fault tolerance and elastic training for long-running jobs - checkpointing strategy, straggler mitigation, preemption recovery
• Familiarity with autonomy or robotics data (ROS, OpenCV, multi-sensor log formats)
• Don’t meet every single requirement? If you’re excited about this role but your past experience doesn’t align perfectly with every qualification in the job description, we encourage you to apply anyway. You may be just the right candidate for this or other roles