Senior Big Data Engineer

Role & responsibilities

• Design, develop, and maintain scalable big data pipelines for batch and real-time data processing.
• Build and optimize data processing solutions using Apache Spark, PySpark, Kafka, Hadoop, and Hive.
• Develop robust ETL/ELT pipelines and integrate data from multiple sources.
• Design and maintain data lakes, data warehouses, and distributed data platforms.
• Optimize Spark jobs, SQL queries, storage, and compute resources for performance and cost efficiency.
• Implement real-time/streaming data pipelines using Kafka and Spark Streaming/Structured Streaming.
• Work with cloud platforms such as AWS, Azure, or GCP and cloud-native data services.
• Develop data models and ensure data quality, reliability, security, and governance.
• Automate workflows and data pipelines using Apache Airflow and CI/CD practices.
• Troubleshoot production issues, perform root-cause analysis, and ensure high availability of data systems.
• Collaborate with data scientists, analysts, software engineers, architects, and business stakeholders.
• Participate in technical design, code reviews, documentation, and mentoring of junior engineers.

Preferred candidate profile

• 5 to 10 years of experience in Big Data Engineering, Data Engineering, or a related role.
• Strong hands-on expertise in Python/PySpark, SQL, and Apache Spark.
• Experience with Kafka, Hadoop, Hive, Airflow, and distributed data processing.
• Strong understanding of ETL/ELT, data modeling, data warehousing, data lakes, and data pipelines.
• Professional experience with at least one major cloud platform: AWS, Azure, or GCP.
• Experience with technologies such as Databricks, Snowflake, AWS S3/EMR/Glue, Redshift, or Azure Data Factory is advantageous.
• Strong understanding of distributed systems, performance optimization, scalability, and fault tolerance.
• Experience developing batch and real-time data processing solutions.
• Proficiency with Git, Linux, CI/CD, Docker, and Kubernetes is preferred.
• Strong problem-solving, analytical, communication, and collaboration skills.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related technical discipline; postgraduate qualifications are a plus.
• Experience working in large-scale production data environments and handling high-volume datasets.