LLM Inference Engineer

1. Deployment

• Set up NVIDIA B300 GPU servers, including drivers, CUDA, NCCL, RDMA, Docker, and Kubernetes.
• Deploy vLLM, SGLang, or TensorRT-LLM and build container images.
• Convert, quantize, and shard open-source models for multi-node, multi-GPU inference.
• Provide OpenAI-compatible APIs and integrate gateways, authentication, rate limits, quotas, and caching.

2. Performance Optimization

• Optimize batching, KV cache, prefix cache, FP8, and distributed inference.
• Improve throughput and reduce time to first token (TTFT).
• Identify GPU, memory, network, and gateway bottlenecks.
• Optimize 128K–256K long-context workloads.

3. Testing and Delivery

• Test API functions, model quality, performance, capacity, and stability.
• Validate streaming, tool calls, structured output, multimodal input, caching, error handling, and security.
• Run benchmarks such as AIME 2026, GPQA Diamond, MMLU-Pro, and LongBench v2.
• Analyze THVV reports, fix issues, and prepare final test reports.

4. Operations

• Monitor GPU usage, memory, TPM/RPM, TTFT, success rates, errors, and cache hit rates.
• Resolve OOM errors, GPU failures, rate-limit errors, timeouts, network issues, and service hangs.
• Maintain at least 99.9% monthly SLA.
• Run regular stress tests and regression tests.
• Manage customer credits, quotas, and rate limits.

Requirements

• Bachelor’s degree or above in Computer Science or a related field.
• At least one year of experience in LLM inference deployment.
• Experience with bare-metal GPU servers and Docker/Kubernetes.
• Strong knowledge of vLLM, SGLang, or TensorRT-LLM.
• Familiarity with CUDA, NCCL, RDMA/InfiniBand, KV cache, FP8, and distributed inference.
• Good Python, Linux, and shell scripting skills.
• Experience with THVV, EvalScope, or similar testing tools.
• Familiarity with OpenAI-compatible APIs, streaming, tool calls, and structured output.