Senior Software Engineer Java LLM Evaluation

Were looking for experienced Senior Software Engineers / Tech Leads with strong Java expertise to contribute to an exciting project at the intersection of Software Engineering and AI/LLM research.

About the Project

We’re building LLM evaluation and training datasets based on realistic software engineering problems. The project involves working with public GitHub repositories to create verifiable SWE tasks that challenge and evaluate modern AI models.

What You’ll Work On

• Analyze and triage GitHub issues across trending open-source libraries
  • Set up and configure code repositories, including Dockerization and development environments
  • Evaluate unit test coverage and quality
  • Modify and run real-world codebases locally to assess LLM performance in bug-fixing scenarios
  • Collaborate with researchers to identify repositories and issues that are challenging for LLMs
  • Potential opportunities to lead and collaborate with junior engineers

What We’re Looking For

• 2–6 years of professional experience in Java/software engineering
  • Strong experience with Java
  • Proficiency with Git and Docker
  • Understanding of basic software pipeline setup
  • Ability to navigate and understand complex codebases
  • Comfortable running, modifying, and testing real-world projects locally
  • Experience contributing to or evaluating open-source projects is a plus

Nice to Have

• Previous experience with LLM research or evaluation projects
  • Experience building or testing developer tools or automation agents

Role Details

Location: Remote
Role Type: Contractor
Commitment: 40 hours/week
Benefits: Contractor assignment; medical benefits and paid leave are not included

If you’re a Senior Java Engineer / Tech Lead interested in working on cutting-edge AI