GCP Data Engineer

Role & responsibilities

• Build batch and streaming data ingestion pipelines using Pub/Sub, Dataflow, Datastream (CDC), and Storage Transfer Service
• Load and ingest data into BigQuery via batch loads, the BigQuery Storage Write API and external/BigLake tables
• Manage Cloud Storage (GCS) landing zones, buckets, lifecycle policies and storage classes
• Develop ETL/ELT workflows using Dataflow (Apache Beam), Dataproc (Spark/Hadoop) and Dataprep
• Build SQL-based transformations using Dataform and BigQuery scheduled queries
• Implement medallion/layered architecture (raw staging curated/gold)
• Design and optimize BigQuery data warehouses with partitioning, clustering, materialized views and cost controls
• Orchestrate pipelines using Cloud Composer (Airflow) and Workflows
• Schedule and automate jobs with Cloud Scheduler and Cloud Functions
• Manage access and security using IAM, service accounts and Workload Identity Federation
• Implement data governance, lineage and quality using Knowledge Catalog
• Apply encryption (CMEK/CSEK, Cloud KMS), VPC Service Controls and DLP via Sensitive Data Protection
• Ensure compliance with PII masking and column/row-level security in BigQuery
• Enable analytics consumption through Looker, Looker Studio and BigQuery BI Engine
• Monitor pipelines and resources with Cloud Monitoring, Cloud Logging, and Cloud Trace
• Optimize query performance, slot usage, and overall cloud cost
• Implement CI/CD using Cloud Build, Artifact Registry, and Cloud Source Repositories
• Manage infrastructure-as-code using Terraform and Deployment Manager
• Maintain version control, automated testing and pipeline reliability with alerting and SLAs