Databricks Lead / Senior Developer

Role & responsibilities

• Design and architect scalable data processing solutions using Databricks and PySpark (Spark Streaming not DLT).
• Develop and maintain data pipelines, ensuring data quality and reliability.
• Implement data orchestration workflows using Azure Event Hubs and Azure Event Grid.
• Collaborate with data engineers, data scientists, and other stakeholders to understand data requirements and deliver solutions.
• Optimize and tune Spark queries for performance and efficiency.
• Ensure data security and compliance with industry standards and best practices.
• Provide technical guidance and mentorship to junior team members.
• Stay updated with the latest trends and advancements in Databricks, PySpark,and Azure services.

Preferred candidate profile