Azure Databricks Engineer

Azure Databricks Engineer (Azure AI Foundry)

Fractal Analytics is a strategic AI partner to Fortune 500 companies with a vision to power every human decision in the enterprise. Fractal is building a world where individual choices, freedom, and diversity are the greatest assets. An ecosystem where human imagination is at the heart of every decision. Where no possibility is written off, only challenged to get better. We believe that a true Fractalite empowers imagination with intelligence. And that it will be such Fractalites that will continue to build the company for the next 100 years .

Please visit Fractal | Intelligence for Imagination for more information about Fractal.

Location: Remote

Note: This Position Is Not Eligible for Immigration Sponsorship at This Time.

Role Overview

We are seeking a highly skilled Azure Databricks Engineer with expertise in building scalable data platforms on Microsoft Azure and experience working with Azure AI Foundry to support advanced analytics, machine learning, and Generative AI solutions. The ideal candidate will have strong data engineering fundamentals, hands-on experience with Azure cloud services, and the ability to design and implement end-to-end data pipelines that enable AI-driven business solutions.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Azure Databricks .
• Build and maintain data ingestion frameworks from structured and unstructured data sources.
• Develop ELT/ETL processes using PySpark, Spark SQL, and Delta Lake .
• Implement modern data architectures including Data Lakehouse solutions.
• Integrate enterprise data platforms with Azure AI Foundry to enable AI and Generative AI use cases.
• Support AI/ML model development by preparing, transforming, and managing training datasets.
• Develop and manage data workflows using Azure Data Factory (ADF) .
• Build data quality, validation, monitoring, and governance frameworks.
• Collaborate with Data Scientists, AI Engineers, Architects, and Business Stakeholders to deliver scalable AI solutions.
• Implement CI/CD pipelines and DevOps best practices for data engineering workflows.
• Ensure compliance with data security, privacy, and governance standards.

Required Skills

Azure Data Engineering

• Azure Databricks
• PySpark
• Spark SQL
• Delta Lake
• Azure Data Lake Storage (ADLS Gen2)
• Azure Data Factory (ADF)
• Azure Synapse Analytics
• Azure SQL Database
• Data Modeling

Programming

• Python
• SQL
• Scala (Preferred)

Azure Platform

• Azure Storage Services
• Azure Key Vault
• Azure Monitor
• Azure Functions
• Azure Logic Apps

DevOps & Deployment

• Azure DevOps
• Git
• CI/CD Pipelines
• Infrastructure as Code (Terraform preferred)

Azure AI Foundry / AI Experience

• Experience working with Azure AI Foundry and Azure AI Services.
• Understanding of LLMs, RAG (Retrieval Augmented Generation), Vector Databases, Embeddings, and Prompt Engineering .
• Experience building data pipelines that feed AI and GenAI applications.
• Knowledge of Azure OpenAI integration patterns.
• Exposure to AI model lifecycle management and monitoring.
• Experience integrating enterprise knowledge repositories with AI applications.
• Understanding of AI governance, security, and responsible AI principles.

Preferred Qualifications

• Experience with Microsoft Fabric .
• Experience with Vector Databases such as Azure AI Search, Pinecone, or Chroma.
• Exposure to MLflow and MLOps frameworks.
• Familiarity with Kubernetes and containerization.
• Knowledge of real-time streaming using Spark Structured Streaming or Kafka.
• Experience supporting conversational AI or GenAI use cases.

Educational Qualifications

• Bachelor's or master's degree in computer science, Engineering, Data Science, Information Technology, or related field.

Nice-to-Have Certifications

• Microsoft Certified: Azure Data Engineer Associate (DP-203)
• Microsoft Certified: Azure AI Engineer Associate (AI-102)
• Databricks Certified Data Engineer Associate/Professional
• Microsoft Azure Solutions Architect Expert

Pay:

The wage range for this role takes into account the wide range of factors that are considered in making compensation decisions including but not limited to skill sets; experience and training; licensure and certifications; and other business and organizational needs. The disclosed range estimate has not been adjusted for the applicable geographic differential associated with the location at which the position may be filled. At Fractal, it is not typical for an individual to be hired at or near the top of the range for their role and compensation decisions are dependent on the facts and circumstances of each case. A reasonable estimate of the current range is: $120,000 - $145,000. In addition, you may be eligible for a discretionary bonus for the current performance period.

Benefits:

As a full-time employee of the company or as an hourly employee working more than 30 hours per week, you will be eligible to participate in the health, dental, vision, life insurance, and disability plans in accordance with the plan documents, which may be amended from time to time. You will be eligible for benefits on the first day of employment with the Company. In addition, you are eligible to participate in the Company 401(k) Plan after 30 days of employment, in accordance with the applicable plan terms. The Company provides for 11 paid holidays and 12 weeks of Parental Leave. We also follow a “free time” PTO policy, allowing you the flexibility to take time needed for either sick time or vacation.

Fractal provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state

or local laws.