Foundation Model Builder

Role & Responsibilities

• Design, implement, and experiment with advanced Foundation Model and LLM architectures, including Transformers, Mixture-of-Experts (MoE), State-Space Models, and diffusion-based models.
• Develop and scale large-scale pre-training and distributed training pipelines across multi-GPU/multi-node environments.
• Work with PyTorch, JAX, DeepSpeed, Megatron-LM, FSDP, or equivalent frameworks.
• Curate, clean, deduplicate, tokenize, and manage large-scale pre-training datasets.
• Monitor training runs and troubleshoot loss spikes, numerical instability, GPU failures, and performance bottlenecks.
• Implement post-training techniques such as SFT, RLHF/RLAIF, DPO, PPO, and GRPO.
• Develop model evaluation and benchmarking frameworks covering reasoning, accuracy, hallucination, safety, and domain adaptability.
• Optimize models for compute efficiency, memory usage, inference latency, and scalability.
• Collaborate with AI infrastructure teams to improve GPU utilization and distributed training performance.
• Apply quantization, compression, and other optimization techniques for production deployment.
• Conduct root-cause analysis of model failures and continuously improve model quality.

Preferred Candidate Profile

• 4+ years of hands-on experience in AI/ML, Deep Learning, LLM, or Foundation Model development.
• Strong proficiency in PyTorch, JAX, or similar deep-learning frameworks.
• Proven experience training or fine-tuning large-scale LLMs/Foundation Models, preferably 10B+ parameter models.
• Strong understanding of Transformer architecture, attention mechanisms, tokenization, and positional encoding.
• Hands-on experience with distributed multi-GPU/multi-node training and frameworks such as Megatron-LM, DeepSpeed, or PyTorch FSDP.
• Strong Python programming skills with good systems engineering fundamentals.
• Experience with SFT, RLHF/RLAIF, DPO, PPO, GRPO, or preference optimization.
• Experience with large-scale dataset preparation and pre-training pipelines.
• Knowledge of CUDA/C++ and GPU optimization will be an advantage.
• Experience with Mixture-of-Experts (MoE) or multimodal AI such as Vision-Language, Audio, or Video is highly desirable.
• Strong analytical and problem-solving skills with the ability to diagnose model and training failures.
• Master's/Ph.D. in Computer Science, AI/ML, Applied Mathematics, or a related field is preferred.