Databrick Engineer

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build and optimize ETL/ELT processes for ingesting data from multiple source systems.
• Develop data transformation logic and implement business rules using Spark frameworks.
• Work closely with business stakeholders, data analysts, and architects to understand data requirements.
• Implement and manage Delta Lake architecture for reliable and high-performance data processing.
• Optimize Databricks jobs, notebooks, clusters, and Spark code for performance and cost efficiency.
• Develop and support data ingestion frameworks using Azure Data Factory (ADF) and other integration tools.
• Monitor production workloads, troubleshoot failures, and ensure data availability.
• Implement data quality checks, validations, and reconciliation processes.
• Ensure adherence to data governance, security, and compliance standards.
• Participate in code reviews, technical design discussions, and best practice implementation.
• Support production deployments and provide L2/L3 support for critical data pipelines.