Databricks Engineer

Role Overview

We are looking for a Databricks Engineer with strong experience in Azure data engineering, Databricks and data lake technologies , along with an understanding of IT Security/Cybersecurity use cases . The ideal candidate is a generalist who can work across security data engineering, analytics, Data Science and emerging GenAI use cases.

Immediate joiners preferred.

Key Responsibilities

• Design, develop and maintain scalable data pipelines for IT Security and Cybersecurity use cases .
• Build and manage data solutions using Azure Data Lake Storage (ADLS/ADLS Gen2), Azure Databricks and Azure data services .
• Ingest, transform and process security data from multiple sources such as SIEM, endpoint, identity, network, cloud and application security platforms .
• Develop Databricks/PySpark solutions for processing and analysing large volumes of security telemetry.
• Work with security logs, alerts, vulnerability data, threat intelligence, authentication data and other security datasets.
• Work with Azure Security services and understand security-related data generated across the Azure/Microsoft security ecosystem.
• Develop data models and curated datasets to support Security Operations, Threat Detection, Risk Analytics and reporting .
• Collaborate with Data Scientists to prepare datasets for machine learning, anomaly detection, behavioural analytics and predictive use cases .
• Support GenAI/LLM/RAG use cases involving security data, knowledge retrieval and intelligent security analytics.
• Develop automation scripts using Python
• Build and maintain Power BI dashboards and reports for security metrics, trends, risks and operational KPIs.
• Implement data quality, governance, access controls and monitoring across data pipelines.
• Troubleshoot data ingestion, transformation and pipeline issues and optimize performance and cost.
• Collaborate with Cybersecurity, SOC, Data Engineering, Data Science and Cloud Engineering teams.

Required Skills & Experience

• 8+ years of experience in Data Engineering, Data Analytics, Security Data Engineering or a related field.
• Strong hands-on experience with Microsoft Azure data services.
• Strong experience with Azure Data Lake Storage (ADLS/ADLS Gen2) .
• Hands-on experience with Azure Databricks and PySpark .
• Strong SQL and Python skills.
• Experience working with IT Security/Cybersecurity data or security-related use cases .
• Good understanding of Azure Security / Microsoft security ecosystem .
• Experience in data ingestion, transformation, data modelling and pipeline development.
• Experience with Power BI dashboards and reporting .

Data Science / AI Exposure

• Good understanding of Data Science concepts , including statistics, machine learning, anomaly detection and feature engineering.
• Ability to work with Data Scientists and understand ML/data requirements.
• Exposure to GenAI, LLM, RAG or AI-powered analytics is preferred.
• Ability to explore and analyze large security datasets and identify meaningful patterns and trends.

Preferred Certifications:

• Databricks Certified Data Engineer Associate / Professional or equivalent Databricks certification.
• Microsoft security certifications such as SC-200 – Security Operations Analyst Associate , SC-500 – Cloud and AI Security Engineer Associate
• Other relevant certifications such as Security+, CEH, CISSP or equivalent are an advantage.