Inference Intern

• We are seeking talented Fall ‘26, Spring ‘27, and Summer ‘27 Inference Architecture interns to join our team and contribute to the design of next-generation AI accelerators.
• This role focuses on developing and optimizing compute architectures that deliver exceptional performance and efficiency for transformer workload
• You will work on cutting-edge architectural problems and performance modeling over the course of your internship
• Support porting state-of-the-art models to our architecture. Help build programming abstractions and testing capabilities to rapidly iterate on model porting
• Assist in building, enhancing, and scaling Sohu’s runtime, including multi-node inference, intra-node execution, state management, and robust error handling
• Contribute to optimizing routing and communication layers using Sohu’s collectives
• Utilize performance profiling and debugging tools to identify bottlenecks and correctness issues
• Develop and leverage a deep understanding of Sohu to co-design both HW instructions and model architecture operations to maximize model performance
• Implement high-performance software components for the Model Toolkit
• 12-week paid internship

Benefits

• Competitive Salary and Equity - We offer a competitive salary and equity package that aligns with industry standards
• Benefits - Your wellbeing is important to us. We offer unlimited PTO, life insurance, and a 401(k)
• Full Insurance Coverage - Our insurance includes full medical, dental, and vision with 100% of premium covered- Deep knowledge of transformer model architectures and/or inference serving stacks (vLLM, SGLang, etc.)
• Understanding of performance-sensitive or complex distributed software systems, e.g. Linux internals, accelerator architectures (e.g. GPUs, TPUs), Compilers, or high-speed interconnects (e.g. NVLink, InfiniBand)
• Proficiency in Python, C++
• Ported applications to non-standard accelerator hardware or hardware platforms
• Progress towards a Bachelor’s, Master’s, or PhD degree in computer science, computer engineering, applied mathematics, or a related field
• Solid grasp of Transformer architectures, particularly Mixture-of-Experts (MoE)
• Familiarity with PyTorch or JAX
• Math competitions (AIME, AMC, etc)
• Low-latency, high-performance applications using both kernel-level and user-space networking stacks
• Built applications with extensive SIMD (Single Instruction, Multiple Data) optimizations for performance-critical paths
• Proficiency in Rust
• Deep understanding of distributed systems concepts, algorithms, and challenges, including consensus protocols, consistency models, and communication patterns
• We encourage you to apply even if you do not believe you meet every qualification