Data Engineer - Scala | Kafka | Spark | GCP

We are looking for an experienced Data Engineer with 6+ years of experience in designing, developing, deploying, monitoring, and optimizing scalable batch and streaming data pipelines. The ideal candidate should have strong hands-on expertise in Scala, Apache Spark, Kafka, Airflow, BigQuery, Dataproc, and GCP.

Key Responsibilities

• Design, develop, deploy, monitor, and optimize ETL/ELT workflows for both batch and real-time/streaming data processing.
• Develop scalable data processing solutions using Scala and Apache Spark.
• Build and manage streaming pipelines using Kafka, including handling high-volume data and back-pressure scenarios.
• Develop and maintain Airflow DAGs for complex data orchestration, dependency management, scheduling, retries, and idempotent execution.
• Work extensively with Google Cloud Platform (GCP) services including BigQuery and Dataproc.
• Implement Medallion Architecture (Bronze, Silver, Gold) with appropriate data quality validations, transformations, aggregations, and summarization.
• Perform performance tuning and optimization of Spark jobs, Kafka pipelines, BigQuery queries, and ETL workflows.
• Implement monitoring and observability mechanisms to identify and troubleshoot pipeline failures, performance issues, and data-quality problems.
• Create and maintain runbooks and operational documentation for production support and continuous optimization.
• Troubleshoot runtime data issues independently and ensure timely resolution of production incidents.
• Follow best practices for data governance, security, access control, and information security.

Required Skills

• 6+ years of experience as a Data Engineer.
• Strong hands-on experience with Scala and Apache Spark.
• Strong knowledge of Kafka and streaming data processing.
• Experience with Apache Airflow and workflow orchestration.
• Hands-on experience with Google Cloud Platform (GCP).
• Strong experience with BigQuery and Dataproc.
• Experience building both batch and streaming ETL pipelines.
• Good understanding of Medallion Architecture and data quality frameworks.
• Strong understanding of Spark performance optimization and distributed data processing.
• Knowledge of streaming concepts such as back-pressure handling, fault tolerance, checkpointing, and recovery.
• Experience with monitoring, observability, troubleshooting, and production support.
• Good understanding of idempotency, dependency handling, retries, and failure recovery in Airflow.

Good to Have

• Exposure to data governance and information security.
• Experience with GCP data services beyond BigQuery and Dataproc.
• Experience implementing automated data quality checks and reconciliation.
• Knowledge of CI/CD and infrastructure-as-code practices.
• Experience creating production runbooks and operational standards.

Ideal Candidate

A strong candidate should be capable of owning end-to-end data pipelines, from development and deployment through production monitoring, troubleshooting, performance optimization, and continuous improvement, across both batch and real-time data environments.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.