GCP Databricks and Data Engineer

Client: Dun & Bradstreet Corp
Experience: 6-8 Years
Location: Hyderabad /

We are seeking a highly skilled GCP Databricks Data Engineer to design, develop, and optimize scalable data solutions on Google Cloud Platform. The ideal candidate will have strong expertise in Databricks, Apache Spark, PySpark, BigQuery, and cloud-native data engineering practices to support enterprise-scale analytics and data modernization initiatives.

Key Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines using Databricks and PySpark.
• Build and optimize ETL/ELT workflows for structured and semi-structured datasets.
• Develop data ingestion frameworks from multiple source systems into GCP.
• Create and maintain data lakes, data warehouses, and curated datasets in BigQuery.
• Implement data transformation, cleansing, validation, and enrichment processes.
• Optimize Spark jobs, partitioning strategies, and query performance.
• Collaborate with data architects, analysts, and business stakeholders to deliver scalable solutions.
• Monitor pipeline performance, troubleshoot failures, and ensure data quality.
• Implement CI/CD and automation for data engineering workflows.
• Ensure adherence to governance, security, and compliance standards.

Required Skills

• Strong experience with Google Cloud Platform (GCP).
• Hands-on expertise in Databricks, Apache Spark, and PySpark.
• Strong knowledge of BigQuery, Cloud Storage (GCS), Dataproc, Dataflow, Pub/Sub.
• Advanced SQL and Data Modeling skills.
• Experience building large-scale ETL/ELT pipelines.
• Strong Python programming skills.
• Experience with workflow orchestration tools such as Airflow/Cloud Composer.
• Understanding of Data Warehousing concepts, Star Schema, Snowflake Schema.
• Experience with Git, CI/CD, and Agile methodologies.

Good to Have

• Delta Lake architecture and optimization.
• Streaming data processing.
• Experience with Looker/Tableau/Power BI.
• Exposure to ML/AI data pipelines.
• Knowledge of Terraform or Infrastructure as Code.