DataOps Engineer

5-10 year(s)

AWSAzure DatabricksDevOps

Description

We're hiring a DataOps Engineer to own the reliability layer behind our data pipelines and platforms — the automation, CI/CD, observability, and quality controls that keep data trustworthy in production across client engagements. This is a client-facing delivery role: you'll work directly with international stakeholders and embed DataOps discipline into every pipeline you touch. AWS experience is mandatory for this role, given our delivery partnership with an AWS Advanced Partner firm and the AWS-centric nature of our client environments

Responsibilities

• Design, build, and maintain scalable, automated data pipelines (batch and streaming) on AWS for client data platforms.
• Implement CI/CD pipelines for data workflows using tools like Jenkins, GitLab CI, or AWS CodePipeline.
• Apply Infrastructure-as-Code (Terraform/CloudFormation) to provision and manage data infrastructure.
• Set up monitoring, alerting, and observability for pipelines — data quality checks, SLA/SLO tracking, and incident response.
• Build and manage orchestration workflows (Airflow, dbt, or AWS Step Functions) to schedule and monitor pipeline execution.
• Manage and optimize data warehouse/lake solutions on Redshift, S3, Glue, EMR, or Athena.
• Design and manage lakehouse architectures using AWS Lake Formation, handling data cataloging, fine-grained access control, and governance across S3-based data lakes.
• Build real-time data processing solutions using Kafka, Kinesis, or MSK where client use cases require low latency.
• Implement data version control, lineage tracking, and governance practices.
• Work directly with international clients to understand data reliability and delivery requirements, and translate them into platform designs.
• Ensure data security, access controls (IAM/RBAC), and compliance across pipelines.
• Troubleshoot and resolve pipeline failures, minimizing downtime and maintaining client SLAs.
• Document architecture, runbooks, and operational processes for the data platform.

Requirements

• 5+ years of experience in DataOps, Data Engineering, or DevOps with a strong data focus.
• Mandatory: Hands-on expertise with AWS services — S3, Glue, Redshift, EMR, Lambda, Kinesis/MSK, IAM, CloudWatch.
• Experience with AWS Lakehouse architecture and AWS Lake Formation, including data cataloging, permissions, and governance for data lakes.
• Strong programming skills in Python and SQL; working knowledge of Scala or Java is a plus.
• Production experience with an orchestration tool (Apache Airflow, dbt, or AWS Step Functions).
• Proficiency with containerization (Docker) and orchestration (Kubernetes/EKS).
• Solid understanding of CI/CD practices and Infrastructure-as-Code (Terraform preferred).
• Experience implementing data quality frameworks and monitoring/observability tooling.
• Strong troubleshooting and incident management skills.
• Strong verbal and written communication — comfortable working directly with international clients across cultures and time zones.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Preferences

• AWS Certification (Data Analytics Specialty, Solutions Architect, or DevOps Engineer).
• Experience with big data frameworks (Spark, Hadoop) for distributed processing.
• Exposure to Databricks, Snowflake, BigQuery, or other cloud data platforms alongside AWS.
• Experience with data governance/catalog tools (Atlan, Collibra, or similar).
• Familiarity with MLOps or supporting data science workflows.
• Prior experience in an IT services or consulting environment delivering to overseas clients.