Data Bricks Expert / Solutions Engineer

Roles and Responsibilities :

• Design, develop, test, deploy and maintain large-scale data pipelines using Azure Databricks, PySpark, Delta Lake and SQL.
• Collaborate with cross-functional teams to gather requirements for ETL processes and implement solutions that meet business needs.
• Develop complex data transformations using Python scripts and Spark APIs to integrate various datasets into a single platform.
• Troubleshoot issues related to data quality, performance optimization and scalability of the ETL process.

Job Requirements :

• 11-25 years of experience in Data Engineering with expertise in Azure Databricks, PySpark, Delta Lake and SQL.
• Strong understanding of big data technologies such as Hadoop ecosystem (HDFS) and NoSQL databases like Cosmos DB.
• Proficiency in writing efficient Python code using libraries like Pandas, NumPy, Matplotlib etc.