Manager | GCP Data Engineer | Ahmedabad | Engineering | Data Modernization & Migration

Role Summary

We are looking for a GCP Data Engineering Manager / AD with 8 to 15+ years of experience to lead the delivery and optimization of cloud-based data platforms. The role combines strong expertise in GCP data services with hands-on performance engineering for large-scale Spark and Dataproc workloads.

Key Responsibilities

GCP Data Engineering and Platform Delivery

• Design and deliver data solutions using Cloud Run, Cloud Run Functions with Python, Cloud SQL PostgreSQL, Pub/Sub, GCS and BigQuery.
• Build and manage orchestration workflows using Cloud Composer / Managed Airflow.
• Drive Airflow and Composer optimization, including FinOps, capacity planning and cost governance.
• Support Databricks-based data processing and integration.
• Design and integrate REST APIs across cloud applications and data platforms.
• Provide technical leadership, architecture guidance and delivery oversight to engineering teams.

Performance Engineering and Dataproc Optimization

• Execute workload test waves and enhance Python-based execution templates.
• Troubleshoot Spark, YARN and Dataproc failures using Spark UI and platform diagnostics.
• Analyze approximately 60 workload profiles and capture cost-performance benchmarks.
• Define N1/N2 mapping strategies and translate custom machine requirements into standard GCP machine types.
• Recommend VM sizing, schema design and performance improvements for Big Data workloads.

Required Skills

• 8 to 15 years in Data Engineering, with strong experience on GCP.
• Python, Spark, Dataproc, GCS, BigQuery and Cloud Data Architecture.
• Cloud Run, Cloud Functions, Cloud SQL PostgreSQL and Pub/Sub.
• Cloud Composer / Airflow administration, optimization and FinOps.
• Spark UI debugging, YARN diagnostics and workload benchmarking.
• Strong stakeholder management, team leadership and solution design capabilities.