GreenNode

We are looking for a
Senior Inference Enginee
r with a strong foundation in software engineering, distributed systems, and performance optimization
to build and optimize inference engines for large-scale LLM serving systems
. You will work across both research and production environments, ensuring our LLM serving systems are fast, scalable, and efficient. The role spans the entire inference stack — from kernel and runtime to scheduling, memory management, and distributed execution
Key Responsibilities:
<ul>
<li>Profile, benchmark, and analyze bottlenecks for LLM inference workloads across multiple layers: kernel, memory, networking, and scheduler</li>
<li>Optimize inference engines (vLLM, SGLang, TensorRT-LLM) for throughput, latency, memory efficiency, GPU utilization, and cost</li>
<li>Implement and fine-tune inference optimization techniques including batching, KV-cache management, quantization, speculative decoding, parallelism strategies, and disaggregated serving</li>
<li>Build instrumentation and profiling tools to identify bottlenecks</li>
<li>Ensure the reliability of the inference pipeline through A/B launches, rollback, model versioning, and fault tolerance</li>
<li>Collaborate with the Platform Engineering team to improve serving architecture based on performance findings</li>
<li>Document and share knowledge, contributing to internal best practices and AI open-source projects whenever possible</li>
</ul>
Job Requirements:
<ul>
<li>Minimum 5 years of experience as a Software Engineer, Performance Engineer, or equivalent roles</li>
<li>Strong foundation in software engineering, distributed systems, and performance optimization</li>
<li>Proficient in Python; experience with C/C++, Go, or other low-level programming languages is a plus</li>
<li>Experience with high-performance systems such as high-throughput backends, distributed services, large-scale serving systems, or equivalent</li>
<li>Knowledge of or hands-on experience with AI/ML serving systems, LLM inference, or GPU workloads</li>
<li>Understanding of GPU architecture and the CUDA programming model</li>
<li>Hands-on experience with at least one inference engine (vLLM, SGLang, TensorRT-LLM, Triton Inference Server) or equivalent systems</li>
<li>Understanding of inference optimization techniques including batching, KV-cache optimization, quantization, speculative decoding, tensor/pipeline parallelism, or disaggregated serving</li>
<li>Experience with profiling, bottleneck analysis, and performance tuning for production systems</li>
<li>Strong systems thinking, a high sense of ownership, and willingness to dive deep into complex technical challenges</li>
</ul>
Nice to Have:
<ul>
<li>Hands-on production experience with CUDA programming or NVIDIA libraries such as cuBLAS, cuDNN, and NCCL</li>
<li>Open-source contributions to inference-related projects (vLLM, SGLang, TensorRT-LLM) or AI infrastructure projects</li>
<li>Deep experience with the NVIDIA inference stack including Triton, CUTLASS, TensorRT, or CUDA profiling tools (Nsight Systems / Nsight Compute)</li>
<li>Experience with distributed inference, request routing, or inference orchestra</li>
<li>Experience with LLMOps, RAG systems, or AI agtionents</li>
<li>Experience with observability stacks (Prometheus, Grafana, OpenTelemetry) for ML or distributed systems</li>
<li>Published research papers or open-source contributions in the field of ML systems or inference optimization</li>
</ul>

Senior Inference Performance Engineer (AI)

Senior Inference Performance Engineer (AI)

This job is no longer available

Talents

Recruiters

About