Lead Data Engineer

Position: Lead Data Engineer Databricks or Snowflake
Experience: 5-8 years
Location: Hyderabad
Employment Type: Full-time

Role Overview

We are looking for a Lead Data Engineer with strong expertise in Databricks and Snowflake to design, build, and lead modern cloud data platforms. The role involves hands-on engineering, technical leadership, data architecture, and mentoring data engineering teams.

Key Responsibilities

• Lead the design and development of scalable data engineering and analytics platforms using Databricks and Snowflake.
• Build robust ETL/ELT pipelines for batch and streaming data.
• Develop solutions using Databricks, Apache Spark, PySpark, Delta Lake, and SQL.
• Design and optimize Snowflake data warehouses, schemas, queries, and data models.
• Implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.
• Develop data ingestion pipelines from APIs, databases, files, and cloud storage.
• Implement data quality, governance, lineage, security, and access controls.
• Optimize Spark jobs and Snowflake workloads for performance and cost.
• Build CI/CD pipelines for data engineering workloads.
• Work with cloud services such as Azure/AWS/GCP, depending on the environment.
• Provide technical direction, conduct code reviews, and mentor engineers.
• Collaborate with architects, product owners, analysts, and business stakeholders.
• Troubleshoot production data pipelines and establish monitoring and alerting.

Required Technical Skills

• Databricks: Spark, PySpark, Delta Lake, Unity Catalog, Workflows.
• Snowflake: SQL, Snowpipe, Streams & Tasks, Dynamic Tables, Time Travel, performance tuning.
• Strong Python and SQL skills.
• Strong experience with Apache Spark and distributed data processing.
• Experience with ETL/ELT and data warehouse architecture.
• Experience with Azure/AWS/GCP data services.
• CI/CD using tools such as Azure DevOps, GitHub Actions, Jenkins, or GitLab.
• Experience with orchestration tools such as Databricks Workflows, Airflow, Azure Data Factory, or similar.
• Strong understanding of data modeling, dimensional modeling, and data lake/lakehouse architecture.

Leadership Skills

• Lead technical design and architecture discussions.
• Break down complex requirements into engineering solutions.
• Mentor and guide junior/mid-level engineers.
• Establish coding, testing, deployment, and operational standards.
• Communicate effectively with both technical and non-technical stakeholders.
• Drive engineering best practices across the team.

Preferred candidate profile

• Experience migrating legacy data platforms to Databricks/Snowflake.
• Experience with real-time/streaming data using Kafka or similar technologies.
• Knowledge of Terraform/IaC.
• Experience implementing data governance and security.
• Databricks or Snowflake certifications are a plus.
• Experience working in Agile/Scrum environments