Developer — GCP Data Engineering

Pune

About Us

We empower enterprises globally through intelligent, creative, and insightful services for data integration, data analytics and data visualization.

Hoonartek is a leader in enterprise transformation, data engineering and an acknowledged world-class Ab Initio delivery partner.

Using centuries of cumulative experience, research and leadership, we help our clients eliminate the complexities & risk of legacy modernization and safely deliver big data hubs, operational data integration, business intelligence, risk & compliance solutions and traditional data warehouses & marts.

At Hoonartek, we work to ensure that our customers, partners and employees all benefit from our unstinting commitment to delivery, quality and value. Hoonartek is increasingly the choice for customers seeking a trusted partner of vision, value and integrity

How We Work?

Define, Design and Deliver (D3) is our in-house delivery philosophy. It’s culled from agile and rapid methodologies and focused on ‘just enough design’. We embrace this philosophy in everything we do, leading to numerous client success stories and indeed to our own success.

We embrace change, empowering and trusting our people and building long and valuable relationships with our employees, our customers and our partners. We work flexibly, even adopting traditional/waterfall methods where circumstances demand it. At Hoonartek, the focus is always on delivery and value.

Job Description

Senior Developer — GCP Data Engineering

Roles & Responsibilities

Owns the end-to-end tHands-on implementer responsible for building and testing individual pipelines: Datastream CDC/OTL configuration, Cloud Composer DAGs, PySpark-to-BigQuery SQL conversions, materialized view creation, and report connector reconfiguration. Works under a Tech Lead within a pod, executes the implementation plan, and produces evidence (logs, recon outputs) for QA sign-off.echnical design for the ClickHouse-to-BigQuery migration: medallion architecture (Bronze/Silver/Gold), Datastream/CDC topology, schema-drift handling strategy, and the Technical Design Document (TDD). Acts as the single point of technical authority during the 8-week engagement, signs off on architecture decisions, and resolves cross-pod design conflicts (Postgres, MongoDB, PySpark, reporting).

Primary Skillset

• Configuring and operating Google Cloud Datastream streams (one-time load + CDC) for PostgreSQL (Cloud SQL) and/or MongoDB Atlas sources into BigQuery

• Writing and maintaining Cloud Composer (Airflow) DAGs for scheduled/micro-batch ingestion, using parameterised/templated patterns

• Converting PySpark batch jobs to BigQuery SQL — using BQ-native Spark-to-SQL conversion utilities for standard jobs, and manually re-implementing complex UDFs/window functions where required

• Writing BigQuery SQL for Bronze/Silver/Gold transformations: cleansing, deduplication (via BQ Data Engineering Agent or SQL), and materialized view definitions

• Implementing Dataflow (Apache Beam) jobs to flatten nested MongoDB documents into BigQuery tables, handling document schema variability (SAFE/COALESCE patterns)

• Configuring BigQuery native features: column-level PII masking policies, scheduled exports to GCS (including Iceberg format), and basic Cloud Monitoring/Audit Logging setup

Secondary Skillset

• Reconfiguring BI tool connectors (e.g., Metabase JDBC connector switch from ClickHouse to BigQuery) and validating report output

• Writing automated reconciliation scripts (row-count, checksum comparisons between source and BigQuery)

• Exposure to ClickHouse (reading schemas/data for migration purposes — not requiring ClickHouse admin expertise)

• Shell scripting / Python for automation glue (recon scripts, batch reconfiguration helpers)

• Git-based version control and CI basics for pipeline code

• Awareness of data security/compliance concepts (PII handling, audit logging) sufficient to implement to spec

Job Requirement

Senior Developer — GCP Data Engineering

Roles & Responsibilities

Owns the end-to-end tHands-on implementer responsible for building and testing individual pipelines: Datastream CDC/OTL configuration, Cloud Composer DAGs, PySpark-to-BigQuery SQL conversions, materialized view creation, and report connector reconfiguration. Works under a Tech Lead within a pod, executes the implementation plan, and produces evidence (logs, recon outputs) for QA sign-off.echnical design for the ClickHouse-to-BigQuery migration: medallion architecture (Bronze/Silver/Gold), Datastream/CDC topology, schema-drift handling strategy, and the Technical Design Document (TDD). Acts as the single point of technical authority during the 8-week engagement, signs off on architecture decisions, and resolves cross-pod design conflicts (Postgres, MongoDB, PySpark, reporting).

Primary Skillset

• Configuring and operating Google Cloud Datastream streams (one-time load + CDC) for PostgreSQL (Cloud SQL) and/or MongoDB Atlas sources into BigQuery

• Writing and maintaining Cloud Composer (Airflow) DAGs for scheduled/micro-batch ingestion, using parameterised/templated patterns

• Converting PySpark batch jobs to BigQuery SQL — using BQ-native Spark-to-SQL conversion utilities for standard jobs, and manually re-implementing complex UDFs/window functions where required

• Writing BigQuery SQL for Bronze/Silver/Gold transformations: cleansing, deduplication (via BQ Data Engineering Agent or SQL), and materialized view definitions

• Implementing Dataflow (Apache Beam) jobs to flatten nested MongoDB documents into BigQuery tables, handling document schema variability (SAFE/COALESCE patterns)

• Configuring BigQuery native features: column-level PII masking policies, scheduled exports to GCS (including Iceberg format), and basic Cloud Monitoring/Audit Logging setup

Secondary Skillset

• Reconfiguring BI tool connectors (e.g., Metabase JDBC connector switch from ClickHouse to BigQuery) and validating report output

• Writing automated reconciliation scripts (row-count, checksum comparisons between source and BigQuery)

• Exposure to ClickHouse (reading schemas/data for migration purposes — not requiring ClickHouse admin expertise)

• Shell scripting / Python for automation glue (recon scripts, batch reconfiguration helpers)

• Git-based version control and CI basics for pipeline code

• Awareness of data security/compliance concepts (PII handling, audit logging) sufficient to implement to spec