Technical Lead

QUALIFICATION

• B.Tech degree/MCA with computer science background or equivalent experience.

EXPERIENCE:

• 7 years of total and minimum 3 years of relevant experience in Databricks.
• Proven hands-on experience with Databricks , including designing and implementing scalable data lakehouse solutions.

SKILLS:

• Strong expertise in Python and PySpark for developing enterprise-grade data pipelines and distributed processing frameworks.
• Extensive experience with Snowflake, including data modeling, performance tuning, query optimization, and cost management.
• Hands-on experience with Apache Iceberg, Delta Lake, and modern open table formats for large-scale data lake implementations.
• Strong understanding of AWS/GCP cloud services, architecture design, deployment, and platform operations.
• Experience in building batch and real-time data processing pipelines using Spark, Kafka, and cloud-native services.
• Proven expertise in ETL/ELT design, data integration, and cloud migration initiatives.
• Experience leading development teams and delivering data modernization programs.
• Strong troubleshooting and analytical skills with the ability to resolve complex production issues.
• Excellent stakeholder management and communication skills, working across technical and business teams.
• Expert knowledge of Databricks, including Lakehouse Architecture, Delta Lake, Unity Catalog, Workflows, and Performance Optimization.
• Strong proficiency in Python, PySpark, Spark SQL, and distributed data processing.
• Extensive hands-on experience with Snowflake, including Snowpipe, Streams, Tasks, Data Sharing, and Query Optimization.
• Strong expertise in Apache Iceberg, open table formats, schema evolution, partitioning, and time-travel capabilities.
• Experience designing and implementing large-scale Data Engineering and ETL/ELT Pipelines.
• Hands-on experience with AWS/Azure/GCP cloud-native data services.
• Strong SQL and data modeling skills across OLTP and analytical environments.
• Experience with Real-Time Data Processing using Kafka, Event Hubs, Kinesis, or Pub/Sub.
• Knowledge of Data Governance, Metadata Management, Data Quality, and Security frameworks.

Good to Have

• Experience with Terraform, CloudFormation, or Infrastructure as Code (IaC).
• Knowledge of Docker, Kubernetes, and containerized data platforms.
• Experience with CI/CD tools such as GitHub Actions, Jenkins, Azure DevOps, or GitLab.
• Exposure to AI/ML, MLOps, and GenAI data platform integration.
• Knowledge of Data Mesh, Data Fabric, and modern enterprise data architectures.
• Databricks, Snowflake, and Cloud platform certifications.
• Expert knowledge of Databricks, including Lakehouse Architecture, Delta Lake, Unity Catalog, Workflows, and Performance Optimization.
• Strong proficiency in Python, PySpark, Spark SQL, and distributed data processing.
• Extensive hands-on experience with Snowflake, including Snowpipe, Streams, Tasks, Data Sharing, and Query Optimization.
• Strong expertise in Apache Iceberg, open table formats, schema evolution, partitioning, and time-travel capabilities.
• Experience designing and implementing large-scale Data Engineering and ETL/ELT Pipelines.
• Hands-on experience with AWS/Azure/GCP cloud-native data services.
• Strong SQL and data modeling skills across OLTP and analytical environments.
• Experience with Real-Time Data Processing using Kafka, Event Hubs, Kinesis, or Pub/Sub.
• Knowledge of Data Governance, Metadata Management, Data Quality, and Security frameworks.

Key Responsibilities

Technical Leadership

• Lead development and implementation of scalable data engineering solutions.
• Drive end-to-end data pipeline development using Databricks and Python.
• Design batch and real-time data processing frameworks.
• Review code and establish engineering best practices.
• Guide teams on performance optimization and reliability improvements.

Data Engineering & Analytics

• Build and maintain ETL/ELT pipelines using Databricks, Spark, and Python.
• Develop scalable data lakehouse architectures.
• Implement data quality, governance, and observability frameworks.
• Optimize Snowflake workloads and data models.
• Design Iceberg-based data lakes for large-scale analytics.

Cloud & Platform Engineering

• Work with AWS/GCP cloud services.
• Develop cloud-native data solutions using managed services.
• Awareness of CI/CD pipelines and Infrastructure as Code.
• Support platform modernization and cloud migration initiatives.

Stakeholder Management

• Partner with architects, product owners, and business stakeholders.
• Translate business requirements into technical solutions.
• Provide effort estimation and delivery planning.