Data Engineer - (DevOps and ML)

Job Summary

We are seeking an experienced DevOps / MLOps Engineer to support a large-scale data and AI platform initiative on Google Cloud Platform (GCP). This role is responsible for building and operating the CI/CD, infrastructure automation, and machine learning lifecycle capabilities that allow data science and engineering teams to move models and data pipelines from experimentation to production reliably, securely, and at scale. The role requires strong expertise in Azure DevOps, cloud infrastructure, containerization, automation, and ML model operations, along with the ability to deliver production-grade platforms aligned with business requirements and project timelines.

Key Responsibilities

• Design, build, and maintain CI/CD pipelines in Azure DevOps (YAML pipelines, Azure Repos, Azure Artifacts, service connections, environments, and approval gates) for data pipelines, ML models, and application services deployed to GCP.

• Provision and manage GCP infrastructure using Infrastructure as Code (Terraform) executed through Azure Pipelines, covering GKE, Cloud Run, BigQuery, Cloud Storage, Vertex AI, Composer, and networking/IAM.

• Build and operate end-to-end MLOps workflows on Vertex AI (or equivalent), including feature stores, training pipelines, model registry, automated evaluation, and deployment to batch and online endpoints, triggered and governed through Azure DevOps.

• Containerize and orchestrate workloads with Docker and Kubernetes (GKE), including Helm charts, autoscaling, and resource optimization for training and inference jobs.

• Implement model monitoring for drift, data quality, performance degradation, and cost, with automated alerting and retraining triggers.

• Establish reproducibility and governance practices: experiment tracking (MLflow / Vertex Experiments), data and model versioning, lineage, branching strategies, and promotion gates across dev, UAT, and production environments.

• Implement observability across platforms and services using Cloud Monitoring, Cloud Logging, Prometheus, and Grafana, and define SLOs and incident response processes.

• Embed security and compliance into the delivery lifecycle: secrets management (Azure Key Vault / GCP Secret Manager), IAM least privilege, vulnerability scanning, image signing, and policy-as-code within pipelines.

• Support migration of on-premises data and ML workloads to GCP, including redesign of build, deployment, and orchestration patterns where required.

• Optimize cloud cost and performance across compute, storage, and ML serving resources.

• Collaborate with data scientists, data engineers, and application teams to standardize pipeline templates, development environments, and deployment patterns; manage Azure Boards work items and release planning where applicable.

• Participate in UAT, release management, production support, and documentation of platform standards and runbooks.

  Qualifications

• Strong hands-on expertise in:

  • Azure DevOps (YAML pipelines, Azure Repos, Azure Artifacts, environments, approvals, service connections, pipeline templates)
  • GCP services (GKE, Cloud Run, Vertex AI, BigQuery, Cloud Storage, Composer, IAM)
  • Terraform and Infrastructure as Code practices
  • Docker, Kubernetes, and Helm

• Proven experience deploying to GCP from Azure DevOps, including Workload Identity Federation or service account integration and multi-environment release strategies.

• Strong proficiency in Python and shell scripting; working knowledge of SQL.

• Proven experience operationalizing ML models, including pipeline orchestration (Vertex AI Pipelines, Kubeflow, or Airflow), model registry, and serving (batch and real-time).

• Solid understanding of ML lifecycle concepts: experiment tracking, feature engineering, model validation, drift monitoring, and retraining strategies.

• Experience with monitoring and logging stacks (Cloud Monitoring, Prometheus, Grafana, ELK or equivalent).

• Experience with DevSecOps practices: secrets management, container scanning, policy enforcement, and audit readiness.

• Proven experience in cloud migration projects, including:

  • On-premise GCP platform transition
  • Migration of ML or data workloads to managed cloud services

• Familiarity with data engineering tools in the GCP ecosystem (Dataflow, Dataproc, Pub/Sub) is preferred.

• Google Cloud Professional certifications

• Exposure to Azure cloud services or multi-cloud environments

• Experience with LLM / GenAI deployment patterns (RAG, vector databases, model gateways)

• Experience with GitOps (Argo CD) and service mesh technologies

• Knowledge of aviation, travel, or large-enterprise regulated environments

• Ability to work closely with data science and business stakeholders to translate requirements into platform capabilities.

• Excellent communication and collaboration skills.

• Ability to manage multiple platforms, environments, and release scenarios effectively.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.