ADF/ Databricks Engineer

Pune

About Us

We empower enterprises globally through intelligent, creative, and insightful services for data integration, data analytics and data visualization.

Hoonartek is a leader in enterprise transformation, data engineering and an acknowledged world-class Ab Initio delivery partner.

Using centuries of cumulative experience, research and leadership, we help our clients eliminate the complexities & risk of legacy modernization and safely deliver big data hubs, operational data integration, business intelligence, risk & compliance solutions and traditional data warehouses & marts.

At Hoonartek, we work to ensure that our customers, partners and employees all benefit from our unstinting commitment to delivery, quality and value. Hoonartek is increasingly the choice for customers seeking a trusted partner of vision, value and integrity

How We Work?

Define, Design and Deliver (D3) is our in-house delivery philosophy. It’s culled from agile and rapid methodologies and focused on ‘just enough design’. We embrace this philosophy in everything we do, leading to numerous client success stories and indeed to our own success.

We embrace change, empowering and trusting our people and building long and valuable relationships with our employees, our customers and our partners. We work flexibly, even adopting traditional/waterfall methods where circumstances demand it. At Hoonartek, the focus is always on delivery and value.

Job Description

Job Title: Azure Databricks Lead

Job Overview:

As an Azure Databricks Lead/Specialist, you will play a critical role in designing, implementing, and optimizing data solutions using Azure Databricks. Your expertise will contribute to building robust data pipelines, ensuring data quality, and enhancing overall performance. You'll collaborate with cross-functional teams to deliver high-quality solutions aligned with business requirements.

Responsibilities:

1. Design and Develop Data Pipelines:

o Create scalable data processing pipelines using Azure Databricks and PySpark.

o Implement ETL (Extract, Transform, Load) processes to ingest, transform, and load data from various sources.

o Collaborate with data engineers and architects to ensure efficient data movement and transformation.

2. Data Quality Implementation:

o Establish data quality checks and validation rules within Azure Databricks.

o Monitor data quality metrics and address anomalies promptly.

o Work closely with data governance teams to maintain data accuracy and consistency.

3. Unity Catalog Integration:

o Leverage Azure Databricks Unity Catalog to manage metadata, tables, and views.

o Integrate Databricks assets seamlessly with other Azure services.

o Ensure proper documentation and organization of data assets.

4. Delta Lake Expertise:

o Understand and utilize Delta Lake, which provides ACID transactions and time travel capabilities on top of data lakes.

o Implement Delta Lake tables for reliable data storage and versioning.

o Optimize performance by leveraging Delta Lake features.

5. Performance Tuning and Query Optimization:

o Profile and analyze query performance.

o Optimize SQL queries, Spark jobs, and transformations for efficiency.

o Tune resource allocation to achieve optimal execution times.

6. Resource Optimization:

o Manage compute resources effectively within Azure Databricks clusters.

o Scale clusters dynamically based on workload requirements.

o Monitor resource utilization and cost efficiency.

7. Source System Integration:

o Integrate Azure Databricks with various source systems (e.g., databases, data lakes, APIs).

o Ensure seamless data ingestion and synchronization.

o Handle schema evolution and changes in source data.

8 Stored Procedure Conversion in Databricks:

• Convert existing stored procedures (e.g., from SQL Server) into Databricks-compatible code.

• Optimize and enhance stored procedures for better performance within Databricks

• SSRS conversion experience

Qualifications and Skills:

• Education: Bachelor's degree in Computer Science, Information Technology, or a related field.

• Experience:

o Minimum 5 years of experience architecting and building data platforms on Azure.

o Proficiency in Azure Databricks, PySpark, and SQL.

o Familiarity with Delta Lake concepts.

• Certifications (preferred):

o Microsoft Certified: Azure Data Engineer Associate or similar.

o Databricks Certified Associate Developer for Apache Spark.

• Soft Skills:

o Strong problem-solving abilities.

o Excellent communication and collaboration skills.

o Ability to work in a fast-paced, agile environment.

o

Additional Considerations:

• Forward-Thinking Approach:

o As an Azure Databricks Specialist, you'll contribute to innovative solutions that drive business insights and enhance data capabilities.

o Stay updated with industry trends and emerging technologies.