Lead Data Engineer

Job Title: Data Engineering Technical Lead
Location: Hyderabad, Telangana
Work Model: Hybrid

The Technical Lead will drive the migration of our Microsoft-based data platforms (on-premise and Azure) to Databricks, providing hands-on mentorship and technical oversight for the data engineering team.

Please note: This is a pure technical leadership role, focused on architecture and execution rather than project management or Scrum facilitation.

The focus is on:

• Technical direction
• Engineering quality
• Delivery execution
• Removing technical blockers
• Ensuring consistency and scalability across teams

The Technical Lead will act as the bridge between architecture, engineering teams, and delivery, while remaining capable of hands-on contribution when needed.

Key Responsibilities
Technical Leadership & Coordination

• Act as the technical reference point for data engineers during migration and transformation activities
• Break down high-level migration objectives into clear technical tasks and implementation approaches
• Ensure alignment to target architecture and standards (Databricks, cloud-native patterns)
• Review designs, pipelines, and implementations to ensure quality and consistency
• Actively remove technical blockers for the engineering teams

Migration & Transformation Execution

Lead the migration of data pipelines from:
o   SQL Server / SSIS (on-prem)
o   Azure Synapse
o   Azure SQL / SQL Managed Instances into Databricks (Lakehouse architecture)

Define and enforce best practices for:
o   Data ingestion
o   Transformation patterns
o   Performance optimization
o   Cost awareness
Support coexistence scenarios during the transition phase

Hands-on Engineering (as needed)
Contribute directly to development when required:
o Databricks notebooks (PySpark / SQL)
o Ingestion pipelines
o Refactoring legacy logic
Build reference implementations and examples for the team
Support troubleshooting and performance tuning

Team Enablement

• Mentor data engineers and help upskill them towards Databricks and modern data engineering practices
• Facilitate technical discussions, refinement sessions, and solution reviews
• Ensure development stays on track, even across multiple parallel workstreams

What This Role Is NOT

• Not a Project Manager
• Not a Scrum Master
• Not a pure Architect
• Not 100% hands-on developer
  It is a delivery-focused technical leadership role.

Required Technical Skills
Core Platform Knowledge

• Strong experience in Data Engineering
• Proven experience leading engineers in complex data environments
• Hands-on experience with:
  o Databricks (mandatory)
  o PySpark & Spark SQL
  o Lakehouse concepts (Bronze / Silver / Gold)
  Microsoft & Azure Background (must-have due to migration context)

Solid experience with:
o SQL Server
o Azure SQL / SQL Managed Instances
o Azure Synapse

• Ability to translate legacy ETL logic into modern Spark-based pipelines

Cloud & Engineering Practices

• Experience with Azure cloud environments
• Understanding of:
• CI/CD for data pipelines
• Git-based development
• Infrastructure-as-code concepts (nice to have)
• Strong SQL skills
• Performance tuning and troubleshooting experience

Qualifications
Bachelors or Master’s degree in Computer Science, or a strictly related technical discipline.
8-12 years of specialized experience in Data Engineering.
Preferred background in the Retail sector, with an understanding of industry-specific data challenges.

Technical Leadership
Lead the DE Team/Owning the DE tasks (Design, build, maintain entire pipelines)
Data quality/Integrity/availability across the system