Python Software Developer

Role & responsibilities

GCP Engineering Skills

• Google Cloud Platform (GCP)

• BigQuery

• Dataproc / Apache Spark

• Bigtable

• Dataplex / Data Catalog

• Python (ETL/ELT pipelines and automation)

• SQL (data transformation, validation, reconciliation)

• Databases:

• MySQL

• Hive

• MongoDB

• Cloud Composer / Apache Airflow

• Kafka and Google Pub/Sub

• Apigee / REST APIs (data extraction and integration)

• GenAI (LLMs, embeddings, prompt engineering) and integration with data platforms

• Experience with Vertex AI, BigQuery ML, or GenAI-enabled GCP services (preferred)

Key Responsibilities

Migration & Modernization

• Design and execute comprehensive migration strategies from on-premises, legacy platforms, or other cloud environments to GCP.

• Migrate structured and semi-structured data to BigQuery, Bigtable, and other GCP storage solutions.

• Build, optimize, and maintain scalable migration pipelines using Python, SQL, and Dataproc (Apache Spark).

• Implement batch and streaming migration patterns using:

• Cloud Composer (Apache Airflow)

• Kafka

• Google Pub/Sub

Data Quality & Governance

• Perform schema mapping, data transformation, reconciliation, and validation to ensure accuracy, consistency, and completeness of migrated datasets.

• Enable metadata management, lineage, and governance using Dataplex and Data Catalog.

• Handle large-volume data transfers using best practices including:

• Parallel loading

• Incremental loads

• CDC (Change Data Capture)

GenAI Adoption

• Leverage GenAI and automation techniques to accelerate:

• Data discovery

• Schema understanding

• Mapping recommendations

• Migration assessments

• Apply GenAI-assisted:

• Data transformation

• Code generation

• Migration pattern optimization

• SQL conversion

• Pipeline scaffolding

• Test case generation

Preferred candidate profile