Lead Data Engineer

Role & responsibilities

• Lead data engineering projects to build and operationalize data solutions for business using Azure services in combination with custom solutions Azure Data Factory, Azure Data Flows, Azure Databricks, Azure Data Lake Gen 2, Azure SQL etc
• Proven experience on leading a team of data engineers providing technical guidance and ensuring alignment with agreed architectural principles
• Experience in migrating on-premise data warehouses to data platforms on AZURE cloud
• Designing and implementing data engineering, ingestion and transformation functions using ADF, Databricks
• Proficient in Py-Spark
• Experience in building Python based APIs on Azure Function Apps
• Experience on Azure Logic apps
• Experience in Lakehouse/Datawarehouse implementation using modern data platform architecture
• Capacity Planning and Performance Tuning on ADF & Databricks pipelines
• Support data visualization development using Power BI
• Exposure across all the SDLC process, including testing and deployment
• Experience in relational and dimensional modelling, including big data technologies
• Experience in Azure DevOps – Build CI/CD pipelines for ADF, ADLS, Databricks, Azure SQL DB etc
• Experience of working in secured Azure environments using Azure KeyVaults, Service Principals, and Managed Identities
• Good to have knowledge on Apigee (Googles API Management)
• Understanding of data masking, encryption and other practices used in handling sensitive data
• Ability to interact with Business for requirement gathering and query resolutions
• Working on off shore office based development teams, collaborating within a team environment and participating in typical project lifecycle activities such as requirement analysis, testing and release
• Develop Azure Data skills within the team through knowledge sharing sessions, articles, etc.
• Adherence to organisations Risk & Controls requirements
• Should have skills for Stakeholder management, process adherence, planning & documentationss

Key Stakeholder Management

Internal

Business Teams

Project Manager

Architects

Data Scientists

Team members

External

Knowledge, Skills, Experience & Educational Qualification

Knowledge & Skills:

• Azure Data Factory,
• Azure Data Lake Storage V2
• Azure SQL
• Azure DataBricks
• Pyspark
• Azure DevOps
• Power BI Report
• Technical leadership
• Confidence & excellent communication

Experience:

• Overall 9+ years of experience
• 5 + Experience on Azure data engineering
• 5 + experience of managing data deliveries

Educational Qualification:

• Graduate/Post-graduate. Preferably with specialisation in Computer Science, Statistics, Mathematics, Data Science, Engineering or related discipline

Microsoft Azure certification (good to have)

Preferred candidate profile

LGBQT++