Big Data Engineer

Role & responsibilities

• Design, develop, and maintain scalable big data pipelines for batch and real-time data processing.
• Build and optimize data ingestion, transformation, and ETL/ELT workflows from multiple structured and unstructured data sources.
• Develop distributed data processing solutions using Apache Spark, PySpark, Hadoop, and related big data technologies.
• Implement real-time streaming solutions using Apache Kafka and other streaming platforms.
• Design and manage data lakes, data warehouses, and lakehouse architectures to support analytics and business intelligence.
• Develop and maintain data pipelines on cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP).
• Optimize data processing performance, storage utilization, query execution, and pipeline reliability.
• Implement data quality checks, validation, monitoring, security, and governance standards.
• Collaborate with data scientists, data analysts, software engineers, and business stakeholders to deliver reliable data solutions.
• Automate pipeline orchestration, scheduling, testing, and deployment using tools such as Apache Airflow and CI/CD frameworks.
• Troubleshoot production issues, resolve data processing bottlenecks, and ensure data availability and consistency.
• Document data architectures, workflows, technical specifications, and operational procedures.

Preferred candidate profile

• Bachelors degree in Computer Science, Information Technology, Software Engineering, Data Engineering, or a related technical discipline.
• 5 to 10 years of experience in big data engineering, data engineering, distributed computing, or large-scale data processing.
• Strong proficiency in Python, SQL, and preferably Scala or Java.
• Hands-on experience with Apache Spark, PySpark, Hadoop, Hive, Kafka, and distributed data processing frameworks.
• Proven experience designing and maintaining scalable batch and real-time data pipelines.
• Strong understanding of ETL/ELT processes, data modeling, data warehousing, data lakes, and lakehouse architectures.
• Experience with cloud data engineering services across AWS, Microsoft Azure, or Google Cloud Platform.
• Familiarity with Databricks, Snowflake, Delta Lake, Apache Airflow, Git, Linux, and CI/CD practices.
• Knowledge of data quality, data governance, security, access control, and pipeline monitoring.
• Strong analytical, debugging, problem-solving, and performance optimization skills.
• Ability to collaborate effectively with cross-functional teams and communicate complex technical concepts clearly.
• Relevant cloud or data engineering certifications are an advantage; a Master’s degree is preferred but not mandatory.