Data Engineer with Data Bricks

About the Role:

We are seeking a Senior Databricks Data Engineer to design and develop scalable data pipelines, optimize Spark workloads, and build enterprise-grade data engineering solutions using Azure Databricks. The engineer will play a critical role in migrating Finance data from on-premises SAP systems into a modern cloud Lakehouse architecture.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines.
• Develop PySpark and Spark SQL transformations.
• Build Bronze, Silver, and Gold layer data pipelines.
• Integrate SAP and enterprise source systems.
• Implement Delta Lake best practices.
• Optimize Spark jobs for performance and cost.
• Build reusable data engineering frameworks.
• Implement data quality and validation rules.
• Develop incremental and CDC pipelines.
• Collaborate with BI developers for optimized reporting datasets.
• Participate in code reviews and technical documentation.
• Support production deployments and troubleshooting.