Senior Databricks data Engineer

Responsibilities:

• Design, develop, and optimize Apache Spark and PySpark workloads within Databricks for large-scale batch and streaming data processing.
• Build and maintain robust ETL/ELT pipelines supporting Bronze, Silver, and Gold layer data movement and transformation.
• Implement Medallion Architecture patterns, ensuring proper separation of raw, conformed, and certified data layers.
• Develop and support Delta Lake solutions utilizing ACID transactions, schema enforcement, incremental processing, and SCD Type 1 and Type 2 methodologies.
• Translate conceptual and logical data models into scalable physical Databricks tables, views, and analytical structures.
• Design metadata-driven and schema-aware frameworks that support reusable, parameterized pipeline execution.
• Implement orchestration strategies using Databricks Workflows, Airflow, Azure Data Factory, or similar tools.
• Embed automated data quality, validation, reconciliation, and monitoring controls into engineering solutions.
• Develop and execute unit, integration, and regression testing for pipelines and data products.
• Implement logging, observability, alerting, and operational monitoring frameworks.
• Support production deployments, stabilization activities, warranty periods, issue remediation, and BAU transition.
• Collaborate with architects, modelers, and business SMEs to validate data definitions, canonical keys, and entity-level acceptance criteria.
• Ensure compliance with governance, security, access control, and enterprise data management standards.
• Contribute to agile delivery activities including sprint planning, backlog refinement, code reviews, and continuous improvement initiatives.

Must-Have Skills:

• Advanced Databricks engineering experience.
• Strong expertise with Apache Spark, PySpark, Spark SQL, and DataFrame APIs.
• Advanced SQL development skills supporting transformations, joins, SCD processing, and analytical workloads.
• Strong Python development experience for reusable framework and pipeline development.
• Proven implementation experience with Medallion Architecture (Bronze, Silver, Gold).
• Deep understanding of Delta Lake including ACID transactions, schema enforcement, time travel, incremental loading, SCD Type 1 and Type 2 processing.
• Experience designing scalable ETL/ELT pipelines across diverse source systems.
• Experience with orchestration platforms such as Databricks Workflows, Airflow, or Azure Data Factory.
• Strong data quality, reconciliation, and automated testing experience.
• Experience implementing monitoring, logging, and observability solutions.
• Experience working on Azure, AWS, or GCP cloud platforms.
• Knowledge of Unity Catalog governance and enterprise access controls.
• Experience with role-based security, data classification, masking, and governance.
• Experience working in Agile delivery environments.