Azure Databricks Engineer

Key Responsibilities:

Data Architecture & Strategy

• Design and implement enterprise data architecture leveraging Databricks Lakehouse, Delta Lake, and Azure cloud-native services

• Create scalable and secure data pipelines that handle structured and unstructured data from internal and external sources.

• Author and maintain technological roadmap with mappings to business capabilities

• Experiment and establish adoption blueprint for emerging capabilities within Databrick and in data technology

• Secure the data with appropriate NIST cyber controls

Data Engineering & Analytics Enablement

• Lead the design of ETL/ELT pipelines using PySpark, SQL, Delta Live Tables, Lakeflow connect, Autoloader, Lake flow declarative pipelines, DBT, and Databricks Workflows.

• Partner with actuarial, underwriting, and BI teams to design semantic layers and analytics-ready datasets.

• Optimize performance and cost of Databricks clusters and workflows. • Enable machine learning and predictive analytics by providing clean, governed, and feature-rich data sets. Governance & Quality

• Review solution developed by the divisional teams and vendors to ensure the solution uses modern technology, built to scale, has resilience, and is cost effective to operate.

• Implement data quality, lineage, and metadata management frameworks using tools like Unity Catalog, Collibra, or Alation.

• Establish and enforce data security and compliance policies aligned with insurance regulations (e.g., NAIC, Privacy, HIPAA, GDPR, CFIUS).

• Ensure consistency of master data across operational and analytical systems. Collaboration & Leadership

• Collaborate with business leaders, data engineers, divisional data teams, data visualization experts, and platform head to align architecture with organizational goals. • Mentor teams on best practices for data modeling, Databricks optimization, and cloud data architecture. • Evaluate new technologies to continuously improve data strategy and capabilities.

Required Qualifications:

• Bachelors or masters degree in computer science, Data Engineering, Information Systems, or related field.
• 7+ years of experience in data architecture or data engineering, with at least 4 years in Databricks.
• Strong experience with insurance data models, including policy, claims, premium, fees, agency, underwriting, accounting, and insured domains.
• Expertise in cloud platforms (AWS, Azure, or GCP) and modern data lakehouse architecture.
• Proficiency in SQL, PySpark, Delta Lake, and Databricks SQL.
• Experience integrating with BI tools (Power BI, Tableau, Looker) and data governance tools.
• Excellent communication and stakeholder management skills.

Preferred Skills:

• Experience with streaming data frameworks (Kafka, Delta Live Tables). • Familiarity with AI/ML pipelines in Databricks.

• Certification(s):

• Databricks Certified Data Engineer Professional

• Azure/AWS Certified Data Architect

• Insurance Data Management Association (IDMA) certifications