GCP+Pyspark+Scala Professional

Role Overview

We are looking for an experienced GCP Data Engineer with 5+ years of experience in data engineering and hands-on expertise in Google Cloud Platform. The candidate will be responsible for implementing, managing, monitoring, and troubleshooting data ingestion pipelines using GCP services.

The role requires strong experience with GCS, BigQuery, Dataflow, Cloud Composer, Data Fusion, and Dataproc, along with proficiency in Python, Terraform, and Git. The ideal candidate should be comfortable working independently on data ingestion, ETL/ELT, migration, and production support activities.

Key Responsibilities

• Configure and execute data ingestion pipelines using reusable GCP-based frameworks.
• Develop and manage ETL/ELT workflows using GCS, BigQuery, Dataflow, Composer, Data Fusion, and Dataproc.
• Manage ingestion parameters, job configurations, metadata, and runtime dependencies.
• Monitor daily data pipeline executions and ensure agreed SLAs are met.
• Troubleshoot and resolve data, parameter, configuration, and job execution issues.
• Analyze runtime failures and independently identify and implement appropriate solutions.
• Escalate framework-level defects to the core development team when required.
• Support data engineering and migration projects on GCP.
• Develop scripts and automation using Python.
• Maintain infrastructure configurations and deployment scripts using Terraform.
• Follow Git-based version control and deployment best practices.
• Collaborate with cross-functional teams to ensure smooth delivery and production operations.
• Identify opportunities to improve ingestion frameworks, processes, and operational efficiency.
• Contribute to framework enhancements and automation initiatives where applicable.
• Work with TWS/TWSd or other enterprise scheduling tools for job scheduling and monitoring, where required.

Mandatory Skills

• 5+ years of experience as a Data Engineer or in a similar role.

• Strong hands-on experience with Google Cloud Platform (GCP).

• Hands-on experience with:

  • Google Cloud Storage (GCS)
  • BigQuery
  • Dataflow
  • Cloud Composer
  • Cloud Data Fusion
  • Dataproc

• Strong Python skills for scripting, debugging, and automation.

• Experience with Terraform / Infrastructure as Code (IaC).

• Good understanding of Git and version control.

• Strong understanding of data ingestion, ETL/ELT workflows, and file formats.

• Ability to independently troubleshoot and resolve production/runtime data issues.

• Strong analytical and problem-solving skills.

Good to Have

• Experience with TWS/TWSd or other enterprise job scheduling and monitoring tools.
• Understanding of BigQuery SQL optimization.
• Experience in GCP data migration projects.
• Exposure to reusable data ingestion frameworks and production data operations.

Preferred Candidate Profile

The ideal candidate should be a hands-on GCP Data Engineer who can independently manage data ingestion pipelines, troubleshoot production issues, work with multiple GCP data services, and support ETL/ELT operations. Strong Python, Terraform, and Git knowledge is expected, along with the ability to work effectively in a production environment.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.