Software Engineer (Inference Performance)

• We’re looking for inference performance engineers who want to make the world’s most demanding AI workloads run faster and more efficiently
• You’ll work across the stack, from the inference engine and runtime through scheduling, serving, and routing
• Along the way you’ll apply techniques like prefill/decode disaggregation, speculative decoding, and KV-cache management
• You’ll reason from first principles about where time and memory go, find what’s holding performance back, and close the gap
• Your work directly impacts how fast our customers’ models run and how efficiently we serve them
• You’ll get to work on these types of projects as an Inference Performance engineer:
• Agentic Kernels in Production
• How we built the new fastest API for GLM-5.2
• Live draft model training for speculative decoding
• Making Kimi K3 Tokenization 18x faster
• The Baseten Inference Stack
• Driving model performance optimization
• Implement and productionize cutting-edge inference techniques, working deep in runtime internals. That includes quantization, speculative decoding, KV-cache reuse, chunked prefill, LoRA, guided generation for structured outputs, and custom scheduling and routing algorithms
• Profile and optimize inference end to end, from kernel launch overhead and memory layout up to request scheduling, prefill/decode disaggregation, and cache-aware routing. Run cross-layer investigations, such as tracing a tail-latency regression from request timing through routing and batching down to a kernel
• Turn performance into cost savings. Improve tokens per GPU-hour, raise utilization, and give customers and internal teams clear latency/throughput/cost tradeoffs
• Bring up and tune new model architectures on new hardware quickly, often in the same week they’re released
• Build benchmarking frameworks that measure real-world performance across model architectures, batch sizes, sequence lengths, and hardware configurations
• Contribute upstream to open-source inference engines (vLLM, SGLang, TensorRT-LLM), and partner closely with model, infrastructure, and customer-facing teams to ship wins

Benefits

• Remote-first work environment. The Baseten team is welcome to work from wherever they want; fully remote, in our San Francisco office, or a mix of both. Today, our team (including our founding team) is spread across the United States, Canada, and Armenia. We provide a $1,000 stipend for you to make your home-office comfortable and productive
• Regular in-person team summits. We get together as a team three times a year to plan, workshop, and most importantly, get to know each other better
• Unlimited PTO. We ask that everyone take at least 4 weeks of vacation. And we have a company-wide break between Christmas and New Year’s Day
• Full healthcare coverage. Medical, dental and vision insurance for you and your family
• Paid parental leave. 16-weeks fully paid parental leave (adoptive and non-birth parents included) and flexibility with schedules while returning to work
• Company-sponsored 401(k) for you to contribute to
• Learning and development budget. We encourage you to take classes, attend conferences, and invest in your craft and we’ll cover expenses to make it happen- This role is ideal for someone who thrives in a fast-paced startup environment and is eager to make significant contributions to the exciting field of LLM inference
• Deep understanding of GPU architecture
• Experience with one or more general-purpose programming languages, such as Python or C++
• Bachelor’s, Master’s, or Ph.D. degree in Computer Science, Engineering, Mathematics, or related field
• Strong familiarity with ML libraries, especially PyTorch, TensorRT, or TensorRT-LLM
• Familiarity with LLM optimization techniques (e.g., quantization, speculative decoding, continuous batching)
• Demonstrated interest and experience in LLMs
• Proficiency in enhancing the performance of software systems, particularly in the context of large language models (LLMs)
• Contributed to vLLM, SGLang, TensorRT-LLM, or another inference engine
• Worked on large-scale distributed serving: autoscaling, load balancing, multi-region or multi-cloud capacity
• Worked on quantization (FP8/FP4, AWQ, GPTQ) or speculative decoding in production
• Written or optimized GPU kernels (CUDA, Triton, CUTLASS, or similar)
• Deep understanding of software engineering principles and a proven track record of developing and deploying AI/ML inference solutions