Technical Program Manager, AI Benchmarking

Job Summary

Title: Technical Program Manager, AI Benchmarking

Location: New Delhi, India

Work Authorization: Current and ongoing work authorization in India

Position: Full-time

Role Overview

PxD is looking for a Technical Program Manager to lead delivery of a national initiative benchmarking Indian AI models for agricultural advisory. Government advisory platforms are increasingly powered by large language models, and a growing set of Indian foundational models, fine-tuned variants, and voice-based systems have emerged as potential alternatives - but little rigorous evidence exists on how they perform inside live advisory workflows. You will run a multi-partner benchmarking program that produces decision-ready evidence for the Ministry of Agriculture and Farmers Welfare on which models can be safely and effectively integrated into existing advisory systems.

PxD will co-design the benchmark framework and act as consortium facilitator and government coordination partner for this work, convening government agencies, academic technical evaluators, benchmark partners, and model providers. You will be the person who holds that consortium together day to day: setting the benchmarking architecture with government, contracting and managing academic and technical partners, overseeing the evidence pipeline from technical testing through farmer-facing user research, and turning the results into recommendations a ministry can act on. This position is based in New Delhi, with travel to partner institutions and project states.

What You'll Bring

Required

• Masters degree (or equivalent experience) in Computer Science, Data Science, Engineering, Public Policy, Economics, Agriculture, Development Studies, Management, or another relevant field.
• 5-8 years of progressively responsible experience managing complex technical programs or projects.
• Working fluency with modern AI systems - LLMs, retrieval-augmented generation, prompting, model evaluation, and the practical trade-offs between accuracy, latency, safety, and cost.
• Demonstrated experience managing multi-stakeholder projects involving government agencies, academic institutions, and external technical partners.
• Strong project management skills with the ability to run parallel workstreams against fixed donor milestones.
• Ability to translate technical evidence into clear, decision-oriented recommendations for non-technical senior audiences.
• Strong stakeholder management and relationship-building abilities.

Responsibilities

Government Stakeholder Management

• Serve as PxD's day-to-day point of contact for the benchmarking program, working through PxD's embedded PMU team to align the Ministry of Agriculture and Farmers Welfare on priority use cases, target languages and geographies, candidate models, benchmarking baselines, and approval pathways.
• Establish the governance framework for the consortium - partner roles, decision rights, steering cadence, and escalation routes - and keep it functioning as the program moves through its stages.
• Negotiate system and data access, including access to government advisory environments for substitution testing, and maintain a permissions tracker across government, academic, and advisory data sources.
• Facilitate workshops and consultations with government counterparts, model providers, and domain experts; manage the funder relationship and reporting.

Benchmarking Design Technical Delivery

• Own the benchmarking framework end to end: advisory use cases, benchmark tasks, datasets, human-rating rubrics, safety and language criteria, tool-use and source-grounding assessments, cost comparison templates, and decision thresholds.
• Work with existing benchmark initiatives to adapt and localise established assets rather than building a parallel benchmark.
• Coordinate technical benchmarking across three layers - standalone model evaluation, substitution testing within the live government advisory stack, and a neutral benchmarking harness where direct system access is unavailable - alongside assessment of speech recognition, translation, text-to-speech, and multilingual performance.
• Manage latency, reliability, and total-cost-of-ownership analysis, and oversee the cloud compute, GPU, and API resources that represent the single largest cost line in the program.
• Safeguard the independence and neutrality of the benchmarking by maintaining clear separation between benchmark development, model evaluation, and model development roles of partner organisations, with transparent disclosure of potential conflicts of interest.

Evidence Generation Synthesis

• Oversee pre-deployment user evidence generation with agronomists, extension personnel, and compensated farmer participant testers, assessing whether advisory outputs are comprehensible, actionable, trustworthy, and usable in local languages.
• Manage IRB submission and third-party field vendor selection and oversight for user testing.
• Consolidate findings across technical performance, user evidence, safety, operational readiness, and cost into a single decision framework, and translate them into recommendations a non-technical government audience can act on.
• Author the final evidence synthesis and decision report identifying models ready for pilot deployment, models requiring further optimization, and key implementation risks.

Grants, Contracts Partner Management

• Contract and manage academic and technical partners, sandbox and compute hosts, domain-expert consultants, and field vendors, monitoring performance against agreed deliverables.
• Ensure contractual and donor compliance throughout implementation, including data-processing agreements and dataset licenses.
• Coordinate with finance and operations teams on budgets, procurement, and financial reporting against grant milestones.

Team Knowledge Management

• Work alongside research and field staff supporting the program, providing technical direction and day-to-day coordination.
• Document benchmarking tools, protocols, and methodology so future evaluations across agricultural and other public-sector advisory systems can build on this work.
• Contribute to PxD's institutional knowledge management and cross-country learning on AI advisory.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.