Software & DevOps Engineer AI/ML Evaluation & MLOps

Role Overview

We are seeking a Software & DevOps Engineer with experience in software engineering, MLOps, and AI/ML model evaluation. The role will focus on developing and maintaining solutions that assess AI model capabilities, establish evaluation benchmarks, and support risk and governance requirements through objective, data-driven analysis.

Key Responsibilities

• Design, develop, and support solutions for evaluating AI/ML models and their capabilities.
• Build and maintain benchmarking frameworks and automated evaluation pipelines.
• Analyze model performance and assessment results to provide actionable insights.
• Collaborate with Model Risk Management (MRM) and other stakeholders to align evaluation approaches with governance requirements.
• Implement and manage solutions within DevOps and MLOps environments, ensuring scalability and reliability.
• Document evaluation methodologies, results, and technical designs.

Required Skills & Experience

• Strong software engineering experience with modern development and DevOps practices.
• Hands-on experience with MLOps, model lifecycle management, and model performance evaluation.
• Strong analytical and data interpretation skills.
• Experience working with Model Risk Management (MRM) or similar governance functions.
• Ability to develop scalable solutions and automated testing/evaluation frameworks.

Preferred Skills

• Experience with AI/LLM evaluation frameworks such as DeepEval or similar tools.
• Exposure to AI model benchmarking and validation methodologies.
• Familiarity with cybersecurity concepts, AI risk assessment, or responsible AI practices.

Key Deliverables

• Develop and own a solution that evaluates whether AI models demonstrate advanced cyber capabilities.
• Establish and maintain benchmark frameworks for model assessment.
• Execute evaluations and generate clear outputs on model capabilities and associated risks.
• Support governance and decision-making through repeatable, data-driven evaluation processes.

Education

• Bachelor's or Master's degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, or a related field.

What do we offer:

• Cooperation on a B2B contract basis.
• Opportunity to use foreign languages and the newest technologies on a daily basis.