Senior Data Engineer

**Job Description:**We are seeking an experienced and motivated Data Engineers to join our growing team. The ideal candidate will have a minimum of 4 years of hands-on experience in data engineering, with a strong focus on Spark.

As a Data Engineer, you will play a pivotal role in designing, developing, and maintaining our data infrastructure, as well as leading a team of data engineers to deliver high-quality solutions

**Experience Range:**4 - 7 years

Educational Qualifications:****-B.Tech/B.E in CS/IT/AIML

Job Responsibilities:

• Lead a team of data engineers in designing, building, and maintaining robust and scalable data pipelines and ELT/ ETL processes using Spark.
• Collaborate closely with cross-functional teams including data scientists, analysts, and software engineers to understand data requirements and deliver effective solutions.
• Architect and optimise data storage and processing solutions for performance, reliability, and scalability.
• Design and implement data quality and data validation processes to ensure the accuracy and integrity of our data.
• Stay up-to-date with the latest industry trends and best practices in data engineering and big data technologies.
• Mentor and guide team members, providing technical leadership, code reviews and knowledge sharing.
• Participate in the full development lifecycle, from requirements gathering to deployment and monitoring.
• Troubleshoot and resolve data pipeline issues, performance bottlenecks, and other technical challenges.
• Develop and maintain documentation for data pipelines, processes, and procedures.

**Skills Required:**Apache Spark , Python , Java , AWS , ETL , BigQuery, Data Warehousing

Candidate Attributes:

• Strong expertise in Apache Spark for processing and analysing large datasets.
• Experience of working with EMR and Dataproc is a plus.
• Strong coding experience in programming languages: Python or Java.
• Solid understanding of distributed computing concepts and data partitioning techniques.
• Experience with cloud platforms such as AWS, GCP, or Azure and their data services.
• Proven track record of designing and implementing ELT/ ETL processes and data pipelines.
• Experience with data modelling, data warehousing using BigQuery, and data architecture principles.
• Experience or Familiarity with LakeHouse architecture with Apache Delta files and Iceberg.
• Experience with Apache Kafka, Apache Airflow, Debezium and Kafka Connect on scale.
• Experience with zero code or low code data pipelines using open source.
• Experience of deploying data pipelines using Containers and Kubernetes.
• Excellent problem-solving skills and the ability to work in a fast-paced, collaborative environment.
• Strong leadership and communication skills, with the ability to mentor and guide team members.
• Experience with version control systems (e.g., Git) and agile development methodologies.
• Knowledge of data security and privacy best practices is a plus.