Lead Data Engineer

Role & responsibilities

• Work with different scrum teams to develop all the quality database programming requirements of the sprint.

• Experience in Azure cloud platforms like Advanced Python Programming, Databricks, Azure SQL , Data factory (ADF), Data Lake, Data storage, SSIS.

• Create and deploy scalable ETL/ELT pipelines with Azure Databricks by utilizing PySpark and SQL.

• Create Delta Lake tables with ACID transactions and schema evolution to support real-time and batch processing.

• Experience in Unity Catalog for centralized data governance, access control, and data lineage tracking.

• Independently analyse, solve, and correct issues in real time, providing problem resolution end-to-end.

• Develop unit tests to be able to test them automatically.

• Use SOLID development principles to maintain data integrity and cohesiveness.

• Interact with product owner and business representatives to determine and satisfy needs.

• Sense of ownership and pride in your performance and its impact on companys success.

• Critical thinker and problem-solving skills.

• Team player.

• Good time-management skills.

• Great interpersonal and communication skills.

Preferred candidate profile

• 9-14 years of experience as a Lead Data Engineer.

• Self-driven with minimal supervision. • Proven experience with T-SQL programming, Azure Databricks, Spark (PySpark/Scala), Delta Lake, Unity Catalog, ADLS Gen2

• Microsoft TFS, Visual Studio, Devops exposure.

• Experience with cloud platforms such as Azure or any.

• Analytical, problem-solving mindset.