Data Bricks Engineer

Key Responsibilities

• Design, develop, and maintain scalable and reliable data engineering solutions using Databricks, Python, PySpark, SQL, and Apache Spark.

• Build and optimize Databricks notebooks, clusters, Jobs, Workflows, and SQL Warehouses.

• Develop efficient data pipelines and transformation frameworks for large-scale data processing.

• Work with Delta Lake, Lakehouse architecture, Medallion architecture (Bronze,Silver, Gold), and modern data-modeling practices.

• Optimize Spark jobs, queries, clusters, and data pipelines for performance,scalability, and cost efficiency.

• Develop and maintain data pipelines using Azure Data Factory, Airflow, and dbt.

• Work with Azure storage, identity, networking, and security services to build secure and scalable data solutions.

• Implement and manage Git-based version control and CI/CD pipelines for data engineering workflows.

• Work with Unity Catalog for data governance, access management, lineage, and discoverability.

• Ensure data quality, reliability, security, and compliance across data pipelines and platforms.

• Collaborate with business, analytics, technology, and cross-functional teams to understand requirements and deliver effective data solutions.

• Troubleshoot data pipeline issues and proactively identify opportunities for process and performance improvements.

• Contribute to data engineering standards, best practices, documentation, and continuous improvement initiatives.

Required Skills & Expertise

• Strong hands-on experience with Databricks and Apache Spark.

• Advanced proficiency in Python, PySpark, and SQL.

• Strong understanding of Delta Lake, Lakehouse architecture, Medallion architecture, and data modelling.

• Experience working with Databricks Notebooks, Clusters, Jobs, Workflows, and SQL Warehouses.

• Good understanding of Azure cloud data services, including storage, identity, networking, and security.

• Experience with Azure Data Factory, Airflow, dbt, Git, and CI/CD.

• Knowledge of Unity Catalog, data governance, lineage, data quality, and performance tuning.

• Strong analytical and problem-solving skills.

• Good communication and stakeholder-management skills.

• Ability to work effectively in a full-time work-from-office environment.

Experience & Qualification

Years of Experience: 57 Years

Relevant Experience: Minimum 35 years of hands-on data engineering experience,

including Databricks implementation and delivery.

Educational Qualification:

• BE / B. Tech. / MCA in Computer Science, Information Technology, Engineering, or

a related discipline.

Additional Qualification:

• Databricks certification is preferred.

• Experience or knowledge of pharmaceutical processes and business functions will

be an added advantage.