Lead Data Engineer

Your Role:

• Provide technical leadership on data engineering (acquisition, transformation, distribution), across multiple projects and squads
• Guide, nurture, and coach a team of data engineers, architects and database administrators working in an agile, fast-paced environment
• Drive adoption and evolution of frameworks and products based on best-in-class data management technologies, such as ETL/ELT tools, open-source frameworks, workflow management tools, etc.
• Provide suggestions and recommendation on how to improve the overall data engineering architecture towards a scalable, cost effective ecosystem

Your Deliverables:

• Design, implement, and manage data pipelines using Apache Airflow.
• Develop, test, and maintain custom Airflow operators and DAGs.
• Monitor and troubleshoot Airflow workflows to ensure optimal performance and reliability.
• Collaborate with data engineers and data scientists to understand data requirements and translate them into scalable workflows.
• Optimize and tune Airflow configurations for performance and scalability.
• Ensure data quality and integrity across all workflows.
• Document processes, workflows, and configurations.
• Develop, test, and maintain DBT infrastructure that is executed by Airflow

Your Qualities:

• Inherent curiosity and empathy for customer success
• Obsessive about solving customer problems
• Think long and Act short
• Collaborative with your peers, partners and your team.
• Excited about the mission and milestones not titles and hierarchies
• Nurture an environment of experimentation and learning
• Effective in communicating with both business and technical teams

Your Expertise:

• Excellent understanding of enterprise data warehousing principles of ETL/ELT and the tools/technologies used to implement them
• Proven experience with Apache Airflow, including designing and maintaining complex workflows.
• Strong knowledge of Python programming.
• Experience with SQL and relational databases.
• Familiarity with cloud platforms (e.g., AWS) and their data services.
• Experience with version control systems