GCP Data Engineer

Role & responsibilities

• Design, develop, and maintain scalable data pipelines on Google Cloud Platform.
• Build ETL/ELT workflows using tools such as Cloud Dataflow, Dataproc, Cloud Composer, and BigQuery.
• Develop and optimize SQL queries and data models in BigQuery.
• Perform data ingestion from various sources including databases, APIs, files, and streaming systems.
• Work with Python, PySpark, and SQL for data processing and transformation.
• Implement batch and real-time data processing solutions using technologies such as Pub/Sub and Dataflow.
• Design data warehouses/data lakes using GCP services such as BigQuery, Cloud Storage, and Dataplex.
• Ensure data quality, accuracy, security, governance, and compliance.
• Optimize pipeline performance and GCP resource/cost utilization.
• Monitor, troubleshoot, and resolve issues in production data pipelines.
• Collaborate with Data Scientists, BI teams, Architects, and business stakeholders to understand data requirements.
• Implement CI/CD, version control, testing, and deployment practices for data engineering solutions.
• Create and maintain technical documentation for data pipelines, architecture, and data models.