Data Engineer

Key Responsibilities

• Design, build, and maintain scalable batch and streaming data pipelines using Apache Spark and cloudnative data technologies.
• Develop and optimize ETL/ELT workflows to ingest, transform, and curate data from diverse source systems into analyticsready datasets.
• Implement data modeling and transformation logic to support reporting, dashboards, and downstream analytical and machine learning workloads.
• Build and manage data processing workloads within modern Lakehouse platforms, including Microsoft Fabric / One Lake (preferred).
• Ensure data quality, reliability, and consistency by implementing validation checks, monitoring, and reconciliation processes.
• Optimize Spark jobs for performance, cost efficiency, and scalability across large and complex datasets.
• Manage and evolve data schemas while handling schema drift and upstream source changes.
• Develop reusable frameworks, libraries, and standardized patterns to improve data engineering productivity and consistency.
• Implement CI/CD pipelines for data workloads to enable automated testing, deployment, and rollback.
• Monitor data pipelines and jobs, troubleshoot failures, and resolve performance or data quality issues.
• Partner with analytics engineers, BI developers, and data scientists to understand data requirements and deliver curated datasets.
• Collaborate with platform, security, and governance teams to ensure data security, compliance, and proper access controls.
• Contribute to Agile delivery processes, including sprint planning, design reviews, and continuous improvement initiatives.

Required Qualifications

• Strong experience as a Data Engineer building and operating production data pipelines.
• Handson experience with Apache Spark for largescale data processing.
• Proficiency in Python, SQL, and data transformation best practices.
• Experience with cloudbased data platforms and storage (e.g., Data Lakes, Lakehouse architectures).
• Familiarity with Microsoft Fabric, One Lake, or similar analytics platforms (strong plus).
• Experience designing and optimizing data models for analytical workloads.
• Understanding of distributed data processing concepts, performance tuning, and fault tolerance.
• Experience with CI/CD, version control, and infrastructureascode concepts.
• Strong problemsolving skills and ability to troubleshoot complex data issues.
• Excellent communication skills and ability to collaborate across technical and nontechnical teams.
• 4+ years of experience in data engineering or equivalent role (preferred).

Preferred / NicetoHave Skills

• Experience with realtime or eventdriven data processing.
• Familiarity with data governance, metadata management, and data quality frameworks.
• Exposure to orchestration tools and workflow management systems.
• Experience supporting analytical, reporting, or machine learning use cases.Role & responsibilities