Senior Data Engineer - GCP

About Quantiphi:

Quantiphi is an award-winning, AI-First global digital engineering company that helps the world’s leading Fortune 1000 organizations transform bold ideas into measurable business impact. We go beyond building innovative AI technologies—we solve the problems that matter most to our clients.

Since our founding in 2013, Quantiphi has built a proven track record of turning complex challenges into meaningful outcomes across industries.

Headquartered in Boston, with more than 4,000 professionals worldwide, we partner with global enterprises to deliver large-scale digital, cloud, and AI-driven transformation. #SolvingWhatMatters

We are an Elite and Premier partner to Google Cloud, AWS, NVIDIA, Snowflake, and other leading technology platforms, and our work has been recognized across the industry, including:

• 21 Google Cloud Partner of the Year awards in the past 10 years
• 3 AWS AI/ML Partner of the Year awards
• 3 NVIDIA Partner of the Year awards
• 3 Snowflake Partner of the Year awards
• Rated Leaders by Gartner, Forrester, IDC, ISG, Everest Group and other leading analyst firms

Quantiphi delivers First-in-class AI solutions across Life Sciences, Healthcare, Banking, Financial Services, CPG, Manufacturing, Energy, High-Tech, Telecommunications, etc., powered by cutting-edge Generative AI and Agentic AI accelerators.

We are also proud to be certified as a Great Place to Work —reflecting our commitment to our people and our culture.

For more details, visit: Website or LinkedIn Page

Job Description -

In this role, you will design, build, and scale robust data pipelines, data platforms, and analytical solutions on cloud-native infrastructure, working closely with data scientists, analysts, ML engineers, and business stakeholders.

Key Responsibilities -

• Design, develop, and maintain scalable data pipelines and ELT/ETL workflows using Databricks (GCP & Azure) and DBT
• Build and manage Delta Live Tables (DLT) pipelines and leverage Auto Loader for incremental data ingestion
• Architect and manage data lakehouse solutions using Databricks — including Unity Catalog for data governance and Delta Lake
• Develop and maintain data solutions on Google Cloud Platform (GCP) — GCS, Pub/Sub, GCP Storage Transfer Service (STS), BigQuery, Cloud Composer, and Dataflow
• Write efficient, production-grade code in Python and PySpark/Spark for distributed data processing and automation
• Build and maintain Kafka-based streaming pipelines for real-time data ingestion and processing
• Develop complex SQL queries for data extraction, transformation, aggregation, and validation
• Lead and execute cross-cloud migration initiatives (e.g., migrating workloads between GCP, Azure, AWS)
• Implement data quality checks, monitoring, and alerting across data pipelines
• Ensure compliance with data governance, security, and privacy best practices
• Mentor and guide junior data engineers, conduct code reviews, and champion engineering best practices
• Document data flows, architecture decisions, and pipeline designs

Must-Have Skills -

• Databricks on GCP: Hands-on experience with Databricks on Google Cloud Platform — clusters, jobs, workflows, and integrations
• Azure Databricks: Experience deploying and managing Databricks workspaces on Microsoft Azure
• Unity Catalog: Proficiency in configuring and managing Unity Catalog for unified data governance across workspaces
• Delta Live Tables (DLT): Experience building and managing DLT pipelines for declarative ETL on Databricks
• Auto Loader: Hands-on experience with Databricks Auto Loader for scalable, incremental file ingestion
• GCP Storage Transfer Service (STS): Experience using GCP STS for large-scale data transfers across storage systems
• Google Cloud Storage (GCS): Strong hands-on experience with GCS for data lake storage, lifecycle policies, and integrations
• Python: Strong programming skills for data engineering, scripting, and automation
• PySpark / Spark: Experience building distributed data processing applications using PySpark and Apache Spark
• Kafka: Experience designing and managing Kafka-based real-time streaming pipelines
• SQL: Advanced SQL skills including query optimization, window functions, and complex joins
• Pub/Sub: Hands-on experience with Google Cloud Pub/Sub for event-driven and streaming architectures
• Cross-Cloud Migration: Proven experience migrating data workloads and pipelines across cloud platforms (GCP, Azure, AWS)

Good-to-Have Skills -

• DBT (Data Build Tool) — models, tests, macros, and project management
• Apache Airflow / Cloud Composer for workflow orchestration
• Terraform or other Infrastructure-as-Code (IaC) tools
• Dataflow for batch and streaming data processing
• Data cataloging and governance tools (Dataplex, Collibra, Alation)
• MLOps or feature engineering pipeline support
• CI/CD pipelines for data engineering (GitHub Actions, Cloud Build)
• Apache Iceberg or Apache Hudi table formats
• BigQuery for large-scale analytics workloads

Qualifications -

• Bachelor's or Master's degree in Computer Science, IT, Engineering, or related field.
• 5+ years of professional experience in data engineering.
• Proven track record of delivering end-to-end data pipeline solutions in production.
• Strong understanding of data warehousing, dimensional modeling, and data lakehouse architecture.
• Experience in Agile/Scrum environments.
• Excellent problem-solving, communication, and collaboration skills.