Data Engineer

Pune

About Us

We empower enterprises globally through intelligent, creative, and insightful services for data integration, data analytics and data visualization.

Hoonartek is a leader in enterprise transformation, data engineering and an acknowledged world-class Ab Initio delivery partner.

Using centuries of cumulative experience, research and leadership, we help our clients eliminate the complexities & risk of legacy modernization and safely deliver big data hubs, operational data integration, business intelligence, risk & compliance solutions and traditional data warehouses & marts.

At Hoonartek, we work to ensure that our customers, partners and employees all benefit from our unstinting commitment to delivery, quality and value. Hoonartek is increasingly the choice for customers seeking a trusted partner of vision, value and integrity

How We Work?

Define, Design and Deliver (D3) is our in-house delivery philosophy. It’s culled from agile and rapid methodologies and focused on ‘just enough design’. We embrace this philosophy in everything we do, leading to numerous client success stories and indeed to our own success.

We embrace change, empowering and trusting our people and building long and valuable relationships with our employees, our customers and our partners. We work flexibly, even adopting traditional/waterfall methods where circumstances demand it. At Hoonartek, the focus is always on delivery and value.

Job Description

Develop Scalable Data Pipelines:

o Utilize Databricks and PySpark to design, develop, and maintain data processing pipelines.

o Implement ETL (Extract, Transform, Load) processes, ensuring efficient data extraction, transformation, and loading.

o Deep knowledge and hands-on of Databricks features and services - Feature Engineering, Lakeflow, Lakebase, Spark

declarative pipeline

2. Data Quality Implementation:

o Establish data quality checks within Azure Databricks.

o Ensure data accuracy, consistency, and adherence to business rules.

o Monitor data quality metrics and address anomalies promptly.

3. Source System Integration:

o Integrate Azure Databricks with various source systems (databases, data lakes, APIs).

o Efficiently ingest data from diverse sources into Azure Databricks.

o Handle schema evolution and changes in source data.

4. Pyspark Coding and Optimization:

o Write efficient PySpark code for data transformations, aggregations, and analytics.

o Optimize Spark jobs for performance and resource utilization.

o Troubleshoot and debug PySpark scripts as needed.

5. Collaboration and Communication:

o Work closely with data scientists, engineers, and other stakeholders.

o Gather requirements, understand business needs, and deliver effective solutions.

o Participate in cross-functional teams to ensure successful project outcomes.

6 Delta Lake Expertise:

o Understand and utilize Delta Lake, which provides ACID transactions and time travel capabilities on top of data lakes.

o Implement Delta Lake tables for reliable data storage and versioning.

7 Stored Procedure Conversion in Databricks:

• Convert existing stored procedures (e.g., from SQL Server) into Databricks-compatible code.

• Optimize and enhance stored procedures for better performance within Databricks

Job Requirement

Education: Bachelor's degree in Computer Science, Information Technology, or related field.

• Experience:

o Minimum 3-4 years of experience working with Databricks and PySpark.

o Familiarity with big data technologies and distributed computing.

• Certifications (preferred):

o Microsoft Certified: Azure Data Engineer Associate or similar.

o Databricks Certified Associate Developer for Apache Spark.

• Soft Skills:

o Strong problem-solving abilities and quick troubleshooting skills.

o Excellent communication and collaboration skills.

o Adaptability to work in a dynamic, team-oriented environment.