Lead Data Engineer

Posted on: 13-10-2026

Experience: 8-12 years

India-Bengaluru

Job ID #1002284

Role Requirements

• Skills: SQL Server, Azure SQL Database, T-SQL, Stored Procedures, Complex Views, CTEs, Window Functions, Indexing, Query Plans, Azure Data Factory, SSIS, ETL/ELT, Python, PowerShell, Data Modeling, Data Reconciliation, Data Validation, JSON, Git, CI/CD, GitHub Actions, Azure DevOps, Azure Blob Storage, Azure Key Vault, Azure Monitor, Log Analytics, dbt, Databricks, Microsoft Fabric, Terraform, Bicep, Claude Code, GitHub Copilot
• Own the end-to-end migration strategy, including extraction, mapping, transformation, load, reconciliation, and cutover.
• Profile and reverse-engineer legacy SQL Server databases, views, and materialized view outputs with little or no documentation.
• Build source-to-target mappings from the legacy model into the new operational and reporting databases on Azure SQL.
• Design and build repeatable, idempotent ETL/ELT pipelines for a 10-year historical data load.
• Define data quality rules, cleansing logic, and exception handling for bad, duplicate, and orphaned records.
• Build automated reconciliation using row counts, checksums, aggregates, and record-level comparisons between legacy and target systems.
• Run parallel legacy-versus-platform cycles, including comparison of JSON report outputs, and drive defects to closure.
• Plan and rehearse production cutover, including runbooks, mock migrations, timing, rollback, and sign-off criteria.
• Work with the Solution Architect, Business Analysts, and client SMEs to resolve data rules and business definitions.
• Lead and review the work of a Data Migration Engineer and set coding and testing standards.
• Tune load performance and query performance on large historical datasets.
• Report migration status, risks, and data issues to client stakeholders clearly and early.
• Have 8–12 years of experience in data engineering, with at least two large legacy-to-cloud migrations delivered end to end as the lead.
• Demonstrate expert T-SQL skills, including complex views, stored procedures, CTEs, window functions, indexing, and query plans.
• Have deep SQL Server experience and hands-on Azure SQL Database experience, including serverless or elastic tiers.
• Have strong ETL/ELT tooling experience with Azure Data Factory and/or SSIS, plus scripting in Python or PowerShell.
• Demonstrate data reconciliation and validation practices using automated checks rather than manual spot-checking.
• Have data modeling experience for transactional (OLTP) and reporting workloads.
• Be comfortable reading and producing JSON and comparing structured outputs between systems.
• Have cutover planning experience, including mock runs, rollback plans, and go/no-go criteria.
• Have experience with Git and CI/CD pipelines using GitHub Actions or Azure DevOps for database code and migration jobs.
• Have led or mentored at least 1–2 engineers.
• Demonstrate clear written and spoken English and the ability to work directly with US client stakeholders.
• Survey, assessment, or research data experience, including Qualtrics exports, respondent-level data, and longitudinal datasets, is good to have.
• Education-sector data experience, such as school or district reference data, is preferred.
• Experience with event-driven or microservices architectures and understanding how data flows through queues is preferred.
• Experience with Azure Blob Storage, Key Vault, Azure Monitor/Log Analytics, dbt, Databricks, or Microsoft Fabric is good to have.
• Experience with infrastructure as code using Terraform or Bicep is preferred.
• Experience working with AI coding assistants such as Claude Code or GitHub Copilot is good to have.
• Microsoft Azure Data Engineer Associate (DP-203) certification or equivalent is preferred.