Senior Data Engineer (Databricks Lead)

Overview We are looking for a Senior Data Engineer (Databricks Lead) with 12-16 years of experience to lead the design, development, and optimization of enterprise-scale data platforms. The ideal candidate will have deep expertise in Databricks, Apache Spark, Snowflake, Kafka, Python, SQL, and AWS , along with experience building scalable data architectures that support analytics, real-time processing, AI/ML, and Generative AI initiatives. .

Responsibilities

• Lead the architecture, design, and implementation of scalable, secure, and high-performance data platforms using Databricks, Snowflake, and AWS.
• Design and optimize large-scale batch and real-time data pipelines using Spark and Kafka.
• Develop and maintain robust ETL/ELT frameworks to ensure data reliability, scalability, and performance.
• Implement real-time data ingestion and processing solutions using Kafka and Spark Streaming.
• Build automated data integration frameworks leveraging REST APIs, GraphQL APIs, and Databricks APIs.
• Design cloud-native data solutions aligned with enterprise governance, security, and compliance standards.
• Develop data models using Star Schema, Snowflake Schema, Kimball, and Inmon methodologies.
• Provide technical leadership, architecture guidance, and mentorship to engineering teams.
• Collaborate with Analytics, Data Science, and AI teams to enable Machine Learning, Generative AI, and LLM-based solutions.
• Troubleshoot and resolve platform performance bottlenecks and scalability challenges.
• Drive engineering best practices, code quality standards, and data governance initiatives.

Requirements Technical Skills

• Strong expertise in Databricks and Apache Spark.
• Hands-on experience with Snowflake and modern data warehouse architectures.
• Proficiency in Kafka and real-time stream processing.
• Strong programming skills in Python and advanced SQL.
• Experience with AWS Cloud Services.
• Expertise in designing and implementing ETL/ELT architectures.
• Strong understanding of data modeling concepts and methodologies.
• Experience integrating systems using REST and GraphQL APIs.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• 12-16 years of experience in Data Engineering and Data Platform Architecture.
• Proven experience leading enterprise-scale data modernization and cloud migration initiatives.

Preferred Skills

• Experience with AI/ML data platforms, Generative AI, and LLM enablement.
• Exposure to Data Governance and Data Quality frameworks.
• Strong stakeholder management, communication, and technical leadership skills.