Senior LLM Inference Engineer

Position: AI Inference Solutions Developer

Location: San Francisco Bay Area or Remote

Duration: Long-term

Job Summary:

We are seeking a highly skilled AI Inference Solutions Developer to design, build, and optimize a scalable, hardware-agnostic LLM inference platform across heterogeneous accelerators, including GPUs and other AI processors. The role focuses on disaggregated inference, KV cache management, cache-aware routing, and high-performance multi-node inference using open-source engines such as vLLM or SGLang. The ideal candidate will have deep expertise in distributed systems, RDMA networking, Kubernetes, accelerator programming, and performance optimization. This is a hands-on engineering role involving architecture, development, production deployment, and performance validation.

Must-Have Skills:

• LLM Inference Engines: Deep hands-on experience with vLLM or SGLang internals, including scheduling, KV cache management, prefix caching, and connector APIs.
• Disaggregated Inference: Understanding of prefill/decode separation, KV cache transfer, and multi-tier caching architectures.
• Multi-Node Accelerator Workloads: Production experience with NCCL or equivalent technologies, tensor parallelism, expert parallelism, and kernel/network-level debugging.
• Programming: Strong Python development skills, along with production-quality experience in Go, C++, or Rust.
• RDMA Networking: Hands-on experience with RoCEv2 or InfiniBand networking.
• Kubernetes: Strong systems-level knowledge of Kubernetes and multi-node inference deployments.
• Accelerator Programming: Experience with CUDA, ROCm, Triton, or vendor-specific accelerator SDKs, including understanding differences between accelerator architectures.
• Distributed Systems: Strong knowledge of stateful distributed systems, caching, scheduling, and scalable service architecture.
• Performance Engineering: Experience with profiling, benchmarking, bandwidth optimization, latency analysis, and performance troubleshooting.
• Cache-Aware Routing: Knowledge of prefix-aware routing, cache-aware request scheduling, and service-level objective (SLO) admission.
• Agentic Development: Regular hands-on use of AI coding tools such as Claude Code, with strong code review and debugging capabilities.
• Software Engineering Practices: Strong testing, CI/CD, observability, and clean, versioned API development.
• Technical Communication: Ability to analyze performance measurements, make architecture decisions, and document technical findings clearly.

Responsibilities:

• Design, develop, and optimize disaggregated LLM inference serving using vLLM or similar open-source engines, including prefill/decode pools, expert parallelism, and KV cache transfers.
• Build multi-tier KV cache management solutions, including connector integrations, cache offloading, storage policies, and shared storage admission.
• Develop cache-aware routing capabilities with prefix-aware scoring, hash consistency, and SLO-based request admission using open-source routing frameworks.
• Design accelerator and engine adapter interfaces, build conformance tests, and deploy scalable multi-node inference workloads on Kubernetes.
• Optimize inference performance through benchmarking, profiling, and validation of throughput, cache hit rates, tenant isolation, and failover.
• Troubleshoot accelerator and network-level issues, contribute improvements to open-source projects, and support production deployment and infrastructure lifecycle operations.