Lead Data Engineer

Lead Data Engineer

Location: Chennai | Department: Data Engineering / Data Platform | Reports To: Data Engineering Manager

About the Role

We are lookingfor an experienced Lead Data Engineer to drive the design, development, andmanagement of our data infrastructure while leading a team of 4 engineers. Thisrole combines strong hands-on technical expertise with people management anddelivery ownership and requires someone comfortable using modern AI-assisted("vibe coding") tools to rapidly build and ship data products, whilemaintaining the infrastructure that powers them.

Key Responsibilities

Technical Leadership &Development

Architect, build, and optimizescalable data pipelines using Databricks, PySpark, and Python

Write and review complex SQLqueries for data transformation, validation, and performance tuning

Design and maintain data modelsand ETL/ELT workflows across Databricks environments

Support or lead development ofML models/pipelines for data products (feature engineering, model training,deployment, and monitoring)

Build data product applicationsusing AI-assisted "vibe coding" tools (Claude Code, Cursor, orsimilar) to rapidly prototype, develop, and ship internal tools, dashboards,and data-driven applications

Establish and enforce bestpractices for code quality, version control, and CI/CD using Git

Champion the adoption ofAI-assisted development tools across the team to improve developer velocity,code review efficiency, and documentation

Conduct code reviews,architecture reviews, and technical design sessions

Infrastructure & PlatformOwnership

Own and maintain theinfrastructure underlying data products compute clusters, transactionaldatabases, job orchestration, environments, deployment pipelines, andmonitoring

Ensure data pipelinereliability, observability, and cost optimization across Databricks

Manage access controls,environment configuration, and platform scaling for data product apps

Troubleshoot infrastructureissues end-to-end (compute, storage, networking, connectivity)

Team & People Management

Lead, mentor, and grow a teamof 4 engineers, providing technical guidance and career development support

Conduct 1:1s, performancereviews, and skill-development planning for team members

Foster a culture of continuouslearning and rapid upskilling in response to evolving tech/business needs

Encourage and enable the teamto adopt new tools and frameworks quickly

Delivery & Process Management

Own sprint planning, backloggrooming, and sprint retrospectives for the team (Agile/Scrum)

Break down businessrequirements into well-scoped technical tasks and user stories

Track sprint velocity, manageblockers, and ensure timely delivery of data engineering initiatives

Coordinate withcross-functional stakeholders (analytics, product, business teams) to alignpriorities

Required Skills & Experience

8+ years of experience in dataengineering, with at least 2+ years in a technical leadership or team lead role

Strong hands-on expertise inDatabricks (notebooks, clusters, job orchestration, Delta Lake)

Proficiency in PySpark andPython for large-scale data processing

Advanced SQL skills (queryoptimization, stored procedures, performance tuning)

Experience with Microsoft SQLServer (T-SQL, database design, troubleshooting)

Experience with OCR data extraction(Azure Document Intelligence, AWS Textract)

Solid understanding of Gitworkflows (branching strategies, PR reviews, merge conflict resolution)

Hands-on experience buildingapplications using AI coding tools / vibe coding (Claude Code, Cursor, GitHubCopilot, or similar)

Experience maintaining andtroubleshooting infrastructure supporting data products (compute,orchestration, deployment)

Working knowledge of MachineLearning concepts and lifecycle (model training, deployment, monitoring)

Proven experience running Agilesprint planning, backlog management, and team ceremonies

Experience managing andmentoring a small engineering team (3-5 members)

Demonstrated ability to quicklylearn and adopt new technologies in response to business needs

Preferred Qualifications

Experience with cloud platforms(Azure preferred, Databricks stack)

Familiarity with CI/CDpipelines (Azure DevOps, GitHub Actions, etc.)

Experience with data governanceframeworks and data quality tools

Bachelor's/Master's degree inComputer Science, Engineering, or related field

Soft Skills

Strong communication skills tobridge technical and non-technical stakeholders

Ability to balance hands-oncoding with people management responsibilities

High adaptability comfortablenavigating ambiguity and fast-changing tech/business priorities

Problem-solving mindset with afocus on scalable, maintainable solutions