Senior Data Engineer – Azure Databricks / PySpark

Role Summary

• Experience Required: 4-6 years
• Primary Role: Data Engineer / API Role with AI Builder
• Cloud Platform: Azure Cloud
• Role Type: Individual Contributor

Core Technical Skills

• Azure Databricks
• PySpark
• Scala
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Delta Lake
• Unity Catalog
• Python
• SQL
• GitHub
• CI/CD

Data Engineering Responsibilities

• Design, develop, and optimize scalable data pipelines.
• Build cloud-based data solutions using Azure.
• Develop ETL/ELT data processing workflows.
• Work with large-scale datasets in enterprise environments.
• Perform data modeling and performance tuning.
• Implement and maintain Delta Lake solutions.
• Work with Databricks, ADF, and PySpark for data engineering workloads.
• Develop end-to-end data solutions independently.
• Apply data governance and security best practices.

API Development Skills

• Strong hands-on experience building REST or GraphQL APIs.

• Understanding of:

• API design and contracts

• API security

• Error handling

• API performance optimization

• Experience deploying APIs using Kubernetes, preferably Azure Kubernetes Service (AKS).

AI / ML Skills

• Exposure to AI/LLM technologies.

• Experience or exposure to agentic frameworks.

• Familiarity with:

• Databricks AI/ML

• Codex

• Claude

• Other LLM/AI development frameworks

Cloud & DevOps

• Strong hands-on experience with Azure Cloud.
• Experience with CI/CD practices.
• GitHub experience.
• Knowledge of cloud-native data platforms.
• Understanding of Kubernetes/AKS deployment.

Data Governance & Security

• Strong understanding of data governance.
• Knowledge of data security and access control.
• Experience with Unity Catalog is preferred.
• Ability to implement secure and governed data solutions.

Essential Skills

• Databricks
• Azure Data Factory (ADF)
• PySpark
• Azure Cloud
• ADLS Gen2
• Delta Lake

Desirable Skills

• Scala
• REST APIs / GraphQL
• Kubernetes / AKS
• AI/LLM
• Agentic frameworks
• Databricks AI/ML
• CI/CD
• Unity Catalog

Key Responsibilities

• Own end-to-end solution delivery.
• Analyze and solve complex data engineering problems.
• Collaborate with cross-functional teams and business stakeholders.
• Build scalable and reliable data pipelines.
• Optimize data processing and API performance.
• Follow enterprise standards for governance, security, and deployment.