Senior Manager

Job Description Databricks Data Architect / Lead Data Engineer BFSI & Banking

Position Overview

We are looking for an experienced Databricks Data Architect / Lead Data Engineer with strong expertise in Data Warehousing, ETL, Databricks, Cloud Data Platforms, and Data Engineering. The ideal candidate should have extensive experience delivering large-scale data and analytics solutions, with strong hands-on expertise in Databricks, SQL, Python, PySpark, AWS/Azure, and modern data architecture.

The candidate must have strong experience in the BFSI (Banking, Financial Services & Insurance) domain, preferably with hands-on experience working on Banking projects, financial data platforms, regulatory reporting, customer/account data, transactions, risk, compliance, or related banking data solutions.

Roles and Responsibilities

• Design and develop modern Data Warehouse and Data Lakehouse solutions using Databricks and cloud platforms such as AWS and Azure.
• Define and implement scalable, high-performance data architecture and data engineering solutions aligned with business and analytical requirements.
• Provide forward-thinking and innovative solutions in the Data Engineering, Data Analytics, and Data Platform space.
• Collaborate with Data Warehouse, BI, Business, and Technical Leads to understand requirements for new ETL/data pipeline development.
• Design, develop, and maintain robust ETL/ELT pipelines for batch and near-real-time/streaming data processing.
• Develop data transformation processes using SQL, Python, PySpark, Spark, and Databricks.
• Build and optimize Delta Lake/Lakehouse architectures, including ingestion, transformation, storage, and consumption layers.
  Work closely with business stakeholders to understand reporting requirements and translate them into effective data models and reporting-layer solutions.
• Analyze and triage production issues, identify gaps in existing pipelines, perform root-cause analysis, and implement permanent fixes.
• Monitor and troubleshoot ETL/data pipelines, ensuring data quality, availability, reliability, and performance.
• Orchestrate data pipelines using Apache Airflow and integrate workflows with cloud and Databricks platforms.
• Implement data engineering solutions using batch and streaming technologies, including Kafka and AWS Kinesis, where applicable.
• Drive technical discussions with client architects, business stakeholders, engineering teams, and other technical leads.
• Participate in solution architecture, technical design, code reviews, performance optimization, and implementation discussions.
• Mentor and support junior/entry-level team members in resolving technical challenges and production issues.
• Provide technical guidance and help establish engineering best practices, coding standards, and reusable frameworks.
• Work within a DevOps/CI-CD environment using tools such as Git, Terraform, CircleCI, and related technologies.
  Implement and support data governance, data management, security, metadata, and data quality practices.
• Work with modern Databricks capabilities including Unity Catalog, Delta Lake, data sharing, and related Data & AI platform capabilities.
  Collaborate with cross-functional Agile teams and actively participate in Scrum ceremonies, sprint planning, estimation, technical discussions, and retrospectives.
  Support performance tuning of complex SQL queries, Spark jobs, ETL pipelines, and data warehouse processes.
  Ensure solutions adhere to enterprise architecture, security, compliance, and data governance standards, particularly within the BFSI/Banking environment.
  Work with modern Databricks capabilities including Unity Catalog, Delta Lake, data sharing, and related Data & AI platform capabilities.
• Collaborate with cross-functional Agile teams and actively participate in Scrum ceremonies, sprint planning, estimation, technical discussions, and retrospectives.
• Support performance tuning of complex SQL queries, Spark jobs, ETL pipelines, and data warehouse processes.
• Ensure solutions adhere to enterprise architecture, security, compliance, and data governance standards, particularly within the BFSI/Banking environment.
  Experience
• 13+ years of overall experience in Data & Analytics, with strong experience in Data Engineering/Data Warehousing.
• Experience delivering at least 2 large-scale, end-to-end Data Warehouse/Data Engineering implementations.
• Strong experience in BFSI/Banking domain projects is mandatory.
• Proven experience in technical leadership, solution design, architecture, and client-facing technical discussions.
• Strong communication, stakeholder management, presentation, and technical leadership skills.

Education

• Bachelor's and/or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
• Equivalent professional experience may be considered.Role & responsibilities.