LLM KV Cache Systems Engineer

About the role

We are looking for an LLM KV Cache Systems Engineer to design and build KV Cache subsystems for large-language-model inference. You will work on memory allocation, lifecycle management, reuse, compression, transfer, tiering, and KV-aware scheduling across single-node and distributed serving systems.

This is a systems role. Direct source-level experience with an LLM KV Cache subsystem is more important than familiarity with serving-framework APIs alone.

What you will do

• Design block/page allocators, KV Cache lifecycle management, fragmentation control, eviction, prefix caching, cross-request reuse, and preemption/recovery mechanisms.
• Optimize KV layouts, GQA/MQA handling, quantization, compression, or sparsity while balancing accuracy, capacity, memory bandwidth, and compute overhead.
• Build KV transfer and disaggregated-serving mechanisms, including prefill/decode separation, RDMA/NVLink/CXL transport, and GPU/CPU/SSD memory tiering or offload.
• Develop KV-aware request routing, batching, admission control, scheduling, and multi-tenant isolation under latency and SLA constraints.
• Integrate and productionize KV technologies in systems such as vLLM, SGLang, TensorRT-LLM, LMDeploy, Mooncake, or DistServe, with observability and failure handling.

What we are looking for

• Master’s degree or Ph.D. in Computer Science, Computer Engineering, Distributed Systems, High-Performance Computing, or a related field.
• Deep understanding of Transformer attention, prefill/decode execution, memory behavior, and modern LLM serving architectures.
• Direct source-level design, implementation, or research experience in an LLM KV Cache subsystem.
• Proficiency in C++/Python, Linux, CUDA or equivalent accelerator development, and distributed-system performance engineering.
• Demonstrated results in at least one KV-specific metric: capacity, hit or reuse rate, fragmentation, transfer bandwidth, TTFT, TPOT, throughput, latency, memory cost, or service-level efficiency.

Preferred qualifications

• Experience with disaggregated inference, distributed KV transfer, cache-aware routing, or GPU/CPU/SSD tiered KV storage.
• Core open-source contributions, publications, patents, or large-scale production deployment in LLM serving or KV Cache systems.