GCP Data Engineer

Roles and Responsibilities :

• Design, develop, test, deploy and maintain large-scale data pipelines using Data Flow, Dataproc, PubSub on GCP.
• Collaborate with cross-functional teams to gather requirements for ETL processes and implement solutions that meet business needs.
• Develop scalable data warehousing architectures using BigQuery and Spark/Python for efficient data processing.
• Troubleshoot issues related to pipeline failures or errors in real-time.

Job Requirements :

• 4-7 years of experience in designing and developing large-scale data pipelines on GCP platforms (Data Flow, Dataproc).
• Strong understanding of big data technologies such as Hadoop ecosystem (HDFS), Spark, Python programming language.
• Experience with cloud-based messaging queues like PubSub for event-driven architecture.
• Proficiency in building complex ETL pipelines using various tools like Airflow or similar technologies.