Data Engineer

Data Engineer

Job Summary

We are seeking a skilled Data Engineer to develop and maintain data ingestion and processing pipelines that consolidate data from multiple sources and formats into a centralized repository. The ideal candidate will have hands-on experience with data engineering, data processing frameworks, workflow orchestration, and infrastructure fundamentals to ensure reliable, scalable, and compliant data integration.

Key Responsibilities

• Design, develop, and maintain data ingestion pipelines to collect and integrate data from multiple sources into a centralized repository.
• Build and optimize data processing workflows to handle diverse data formats and large volumes of data.
• Ensure data ingestion and processing adhere to applicable data governance, quality, security, and compliance requirements.
• Develop and maintain ETL/ELT pipelines using Python, SQL, and Apache Airflow.
• Work with distributed data processing technologies such as Apache Spark and Apache Flink.
• Troubleshoot data pipeline failures, investigate data quality issues, and resolve processing bottlenecks.
• Collaborate with infrastructure, networking, and platform teams to support reliable data movement and integration.
• Monitor pipeline performance, data accuracy, and system reliability.
• Optimize data workflows for scalability, efficiency, and operational stability.
• Document data pipelines, processing logic, and technical procedures.

Required Skills and Qualifications

• Strong programming experience with Python and SQL .
• Hands-on experience building and maintaining data ingestion and processing pipelines.
• Experience with Apache Airflow for workflow orchestration and scheduling.
• Experience working with multiple data formats, such as JSON, CSV, Parquet, Avro, or similar.
• Understanding of ETL/ELT processes, data integration, and centralized data repositories.
• Knowledge of data engineering infrastructure, networking fundamentals, and data transfer mechanisms.
• Strong troubleshooting, debugging, and problem-solving skills.

Preferred / Plus Skills

• Experience with Apache Spark for distributed data processing.
• Experience with Apache Flink for stream processing.
• Experience with additional data querying, processing, and analytics tools.
• Familiarity with big data technologies and scalable data platforms.
• Experience with data governance, compliance, and data quality controls.

Ideal Candidate

The ideal candidate is a hands-on Data Engineer who can build reliable data pipelines, integrate data from heterogeneous sources, and deliver compliant, high-quality data into a centralized repository. They should be comfortable working across data processing, orchestration, infrastructure, and networking environments.

Core Technologies: Python, SQL, Apache Airflow

Additional Technologies: Apache Spark, Apache Flink, big data query tools