Senior Data Engineer

Desired Competencies (Technical/Behavioral Competency)
Must-Have**

1. Deep hands-on experience with Python,Pyspark,Spark batch,notebook in Azure
2. Experience in Cosmos DB including data modeling, indexing, partitioning, consistency levels, and performance tuning.
3. Strong understanding of Cosmos DB APIs (Core SQL API, MongoDB API, etc.) and integration patterns.
4. Proficiency in query optimization and troubleshooting Cosmos DB performance issues.
5. Experience in data processing using PySpark and Python.
6. Familiarity with Microsoft Fabric for data engineering and analytics workflows.
7. Proficient using source code management tools such as Git or GitHub
8. Experience with Test Driven Development and / or Behavior Driven Development.

Good-to-Have

1. Exposure to data governance tools like Azure Purview.
2. Familiar with various design patterns
3. Familiar with Azure: SQL Managed Instance, Cosmos DB, Storage Services, Azure Functions
4. Global project experience, and excellent communication skills, verbal and written, and soft skills in agile projects

SN Responsibility of / Expectations from the Role
1 Design and implement scalable and high-performance solutions using Azure Cosmos DB.
2 Develop data ingestion and transformation pipelines using PySpark and Azure Data Factory.
3 Optimize Cosmos DB performance through indexing, partitioning, and query tuning.
4 Collaborate with architects and data engineers to ensure best practices in data modeling and cloud architecture.
5 Implement and maintain CI/CD pipelines for automated deployments and testing.
6 Troubleshoot and resolve technical issues related to Cosmos DB and data pipelines.