Sr Data Engineer

Job Summary

We are hiring a Senior Data Engineer to design, build, and optimize enterprise scale data platforms using Databricks, Python, and PySpark. This role focuses on developing high-performance batch and real-time data pipelines, implementing modern data engineering frameworks, and delivering reliable, governed data solutions that power analytics, reporting, and AI-driven applications across financial services ecosystems.

Experience: 7 - 12 Years

Location: Bangalore

Mode of Work: Hybrid

Job Type: Full time

Must Have Skills

• Python (7+ years) for developing scalable, modular, and production-ready data engineering applications
• PySpark & Apache Spark (5+ years) including DataFrame API, Spark SQL, Structured Streaming, performance optimization, partitioning strategies, joins, caching, and handling data skew
• Databricks (4+ years) including Delta Lake, Unity Catalog, Databricks Workflows, notebooks, jobs, and end-to-end data engineering capabilities
• ETL/Data Engineering/Data Pipeline Development (7+ years) building large-scale batch and real-time data processing solutions
• Advanced SQL & Data Modeling including dimensional modeling, slowly changing dimensions (SCD), schema evolution, and query optimization
• Apache Airflow (3+ years) for workflow orchestration, DAG development, dependency management, scheduling, monitoring, retries, and backfills
• dbt (Data Build Tool) including layered architecture, incremental processing, snapshots, macros (Jinja), testing, documentation, and deployment best practices
• Cloud Platforms (AWS/Azure/GCP) with hands-on experience building and operating cloud native data solutions

Good to Have

• Microsoft Fabric for data integration, engineering, and analytics workloads
• AI-assisted development tools such as GitHub Copilot, Claude Code, or equivalent code generation platforms
• Delta Lake advanced optimization techniques and Lakehouse architecture expertise
• CI/CD pipeline implementation using Jenkins, GitHub Actions, Azure DevOps, or GitLab CI
• Containerization and orchestration using Docker and Kubernetes
• Infrastructure as Code (Terraform, CloudFormation, ARM Templates, or equivalent)
• Financial Services, Banking, Capital Markets, or AML domain expertise
• Data Quality, Data Observability, and Data Governance frameworks

Professional Attributes & Qualifications

• Education: BE/BTech/ME/MTech in Computer Science, Information Technology, Engineering, or a related discipline
• Leadership: Experience mentoring junior engineers, conducting code reviews, and contributing to technical design discussions
• Ownership: Strong track record of driving end-to-end delivery of data engineering solutions from architecture through production deployment
• Problem Solving: Excellent analytical and troubleshooting skills for distributed systems, large-scale data processing, and performance optimization
• Quality Focus: Commitment to engineering excellence, coding standards, testing automation, reliability, security, and maintainability
• Communication: Strong verbal and written communication skills with the ability to collaborate effectively with business, analytics, and engineering stakeholders
• Collaboration: Proven ability to work in cross-functional, globally distributed teams while influencing technical decisions

Key Responsibilities

• Design, develop, and optimize large-scale batch and streaming data pipelines using Python, PySpark, Databricks, and cloud-native technologies
• Build and maintain scalable data processing workloads in Databricks with a strong focus on performance, reliability, cost optimization, and maintainability
• Develop robust dbt models using layered architecture, incremental processing, snapshots, macros, testing frameworks, and documentation standards
• Design and maintain Apache Airflow DAGs for workflow orchestration, operational monitoring, dependency management, retries, and observability
• Implement data governance, lineage, access controls, data quality validation, monitoring, and privacy standards across enterprise data platforms
• Optimize Databricks and Spark workloads through partitioning strategies, query tuning, caching, file optimization, join optimization, and efficient compute utilization
• Collaborate closely with analytics teams, product owners, architects, and business stakeholders to transform requirements into high-quality, trusted datasets
• Conduct code reviews, mentor junior engineers, and drive adoption of engineering best practices and architectural standards
• Build and enhance CI/CD pipelines to automate testing, deployment, monitoring, and operational readiness of data platforms

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.