Senior Data Engineer

Responsibilities:

• Develop & Optimize Data Pipelines

• Build, test, and maintain ETL/ELT data pipelines using Azure Databricks & Apache Spark (PySpark).

• Optimize performance and cost-efficiency of Spark jobs.

• Ensure data quality through validation, monitoring, and alerting mechanisms.

• Understand cluster types, configuration, and use-case for serverless

• Implement Unity Catalog for Data Governance

• Design and enforce access control policies using Unity Catalog.

• Manage data lineage, auditing, and metadata governance.

• Enable secure data sharing across teams and external stakeholders.

• Integrate with Cloud Data Platforms

• Work with Azure Data Lake Storage / Azure Blob Storage/ Azure Event Hub to integrate Databricks with cloud-based data lakes, data warehouses, and event streams.

• Implement Delta Lake for scalable, ACID-compliant storage.

• Automate & Orchestrate Workflows

• Develop CI/CD pipelines for data workflows using Azure Databricks Workflows or Azure Data Factory.

• Monitor and troubleshoot failures in job execution and cluster performance.

• Collaborate with Stakeholders

• Work with Data Analysts, Scientists, and Business Teams to understand requirements.

• Translate business needs into scalable data engineering solutions.

• API expertise

• Ability to pull data from a wide variety of APIs using different strategies and methods

Required Skills & Experience:

• Azure Databricks & Apache Spark (PySpark) – Strong experience in building distributed data pipelines.
• Python – Proficiency in writing optimized and maintainable Python code for data engineering.
• Unity Catalog – Hands-on experience implementing data governance, access controls, and lineage tracking.
• SQL – Strong knowledge of SQL for data transformations and optimizations.
• Delta Lake – Understanding of time travel, schema evolution, and performance tuning.
• Workflow Orchestration – Experience with Azure Databricks Jobs or Azure Data Factory.
• CI/CD & Infrastructure as Code (IaC) – Familiarity with Databricks CLI, Databricks DABs, and DevOps principles.
• Security & Compliance – Knowledge of IAM, role-based access control (RBAC), and encryption.

Preferred Qualifications:

• Experience with MLflow for model tracking & deployment in Databricks.
• Familiarity with streaming technologies (Kafka, Delta Live Tables, Azure Event Hub, Azure Event Grid).
• Hands-on experience with dbt (Data Build Tool) for modular ETL development.
• Certification in Databricks, Azure is a plus.
• Experience with Azure Databricks Lakehouse connectors for SalesForce and SQL Server
• Experience with Azure Synapse Link for Dynamics, dataverse
• Familiarity with other data pipeline strategies, like Azure Functions, Fabric, ADF, etc

Soft Skills:

• Strong problem-solving and debugging skills.
• Ability to work independently and in teams.
• Excellent communication and documentation skills.