Senior Analyst-GenAI

Roles and Responsibilities :

• Design, develop, and maintain complex data pipelines using Python to extract insights from large datasets.
• Collaborate with cross-functional teams to identify business requirements and design solutions that meet those needs.
• Develop and implement machine learning models using retrieval augmented generation techniques to improve data quality and accuracy.
• Troubleshoot issues with existing systems, identifying root causes and implementing fixes to ensure optimal performance.

Job Requirements :

• 5 - 8 years of overall experience in software engineering, data science, or AI/ML development, with at least 3+ years focused on AI/LLM/GenAI testing or agent-based systems.

• Python expertise in scripting, automation, and debugging.

• Strong PySpark experience in distributed testing, data validation, and pipeline testing.

• Hands-on knowledge of GenAI concepts, including LLMs, prompting, context management, RAG pipelines, agent tool-calling, and multi-agent orchestration.

• Experience with agent development and deployment frameworks such as LangGraph, AutoGen, CrewAI, Copilot Studio Agent SDK, and Vertex AI/OpenAI agent frameworks.

• Solid understanding of agent architecture covering skills, tools, connectors, memory, guardrails, and observability.

• Familiarity with modern GenAI/agent evaluation frameworks such as Langsmith evaluation, AutoGen agent-behavior assessment utilities etc. for benchmarking reliability, grounding, tool-use correctness, and multi-agent performance.

• Strong foundation in functional and regression testing, scenario and edge-case testing, LLM safety and hallucination testing, and workflow validation.

• Experience creating evaluation datasets and defining success criteria for AI behavior.

• Good understanding of API testing frameworks, data engineering concepts, cloud workflow execution (Azure/GCP/AWS), and CI/CD pipelines for test automation.

Excellent communication skills for cross-functional collaboration. .