Data Engineer - ATA

Posted on: 13-10-2026

Experience: 8-12 years

India-Bengaluru

Job ID #1002281

Role Requirements

• Skills: Python, OOP, Multiprocessing, Asyncio, Decorators, Generators, Pytest, ETL/ELT, Databricks, Delta Lake, Structured Streaming, Auto Loader, Unity Catalog, Apache Spark, T-SQL, SQL, Dimensional Data Modeling, Star Schema, Snowflake Schema, SCD Types 1/2/3, Data Vault, Kimball, Inmon, Databricks Workflows, Airflow, Azure Data Factory, Azure, AWS, CI/CD
• Design, build, and own production-grade Python codebases for the data platform, including ETL/ELT pipelines, reusable internal packages, schema validation, and testing frameworks.
• Write clean, well-tested, performant Python using OOP, concurrency, decorators, and generators to solve pipeline-scale problems.
• Profile and optimize slow Python pipelines through algorithmic and code-level improvements, migrating heavy workloads to Databricks/Spark when Python alone will not scale.
• Set and enforce Python engineering standards across the team, including code reviews, packaging/versioning, structured logging, and exception-handling patterns.Design and deliver Bronze/Silver/Gold medallion pipelines on Databricks using Delta Lake, Structured Streaming/Auto Loader, and Unity Catalog governance.
• Own dimensional data modeling decisions, including star/snowflake schema design and SCD strategy, for core platform data models.
• Write and review advanced T-SQL for platform workloads, including window functions, MERGE/upsert logic, and execution-plan-based performance tuning.
• Partner with Data Architects and stakeholders on ETL versus ELT, orchestration, and cost/performance trade-offs across the platform.
• Mentor Data Engineers and Senior Data Engineers through design and code reviews.
• Demonstrate 8–12 years of data engineering experience, including at least 2 years in technical leadership or mentoring.
• Have strong hands-on Python experience, including OOP, generators, decorators, multiprocessing/asyncio, exception handling, and unit testing with pytest.
• Have experience building production ETL/ELT pipelines, schema validation, and reusable internal Python packages/frameworks.
• Demonstrate the ability to profile, debug, and optimize Python code for data-pipeline performance.
• Have deep, hands-on Databricks experience with Delta Lake, Structured Streaming, Auto Loader, and Unity Catalog.
• Experience tuning Spark jobs at scale, including data skew, cluster sizing/autoscaling, and cost optimization.
• Demonstrate strong dimensional modeling knowledge, including fact/dimension grain and conformed dimensions, with working knowledge of Data Vault and Kimball versus Inmon trade-offs.
• Have advanced T-SQL skills, including window functions, MERGE/upsert patterns, indexing strategy, and execution-plan-based query tuning.
• Have experience designing medallion architectures and evaluating ETL versus ELT and batch versus streaming trade-offs.
• Have experience with orchestration using Databricks Workflows, Airflow, or ADF on Azure or AWS.
• PCEP/PCAP or equivalent Python certification and Databricks Certified Data Engineer Professional certification are good to have.
• Experience with data governance/compliance, GDPR/HIPAA, FinOps, and formally leading or mentoring a team of 3+ engineers is preferred.
• A Bachelor’s or Master’s degree in Computer Science, Engineering, a related field, or equivalent practical experience.
• Demonstrate strong stakeholder communication and the ability to explain technical trade-offs to engineering and business audiences while owning ambiguous, enterprise-scale design decisions.