Senior Data Engineer

Job Title: Senior Open Source Data Engineer

Job Summary

We are looking for a highly skilled Senior Open Source Data Engineer to design, build, and scale a modern enterprise data platform using open-source technologies. The ideal candidate will have extensive experience in distributed data architectures, real-time data processing, data governance, and large-scale data platform engineering. This role will drive the design and implementation of secure, scalable, and high-performance data solutions supporting analytics, business intelligence, and operational systems across global environments.

Key Responsibilities

Data Platform Architecture & Engineering

• Design and implement scalable Data Lake architectures using Medallion Architecture (Bronze, Silver, Gold layers).
• Build and maintain end-to-end batch and real-time data ingestion pipelines.
• Architect scalable, distributed data platforms supporting enterprise-wide data requirements.
• Ensure seamless data flow from source systems to analytics, APIs, and downstream applications.
• Support multi-region and globally distributed data environments.

Open Source Technology Leadership

• Lead implementation and optimization of open-source data platforms including:

• Data Ingestion: Debezium, Apache Airflow

• Streaming & Processing: Apache Kafka (KRaft), Apache Flink

• Storage: Hadoop, Apache Iceberg

• Query & Data Access: Trino, Apache Gravitino

• Metadata & Governance: Apache Atlas

• Visualization: Apache Superset

• Drive containerized deployments using Docker and Kubernetes.

• Promote open standards and extensible architectures while minimizing dependence on proprietary technologies.

Data Modeling & Governance

• Design robust and scalable data models for complex business domains.
• Implement enterprise-wide data governance frameworks.
• Establish data quality, metadata management, lineage tracking, and access control mechanisms.
• Ensure compliance with organizational security and data protection standards.
• Maintain consistency, integrity, and quality of data across multiple systems and regions.

Data Processing & Integration

• Design and optimize real-time and near real-time processing pipelines.
• Integrate data platforms with APIs, B2B systems, and analytics platforms.
• Support high-volume transactional data processing across multiple geographies.
• Enable reliable and scalable data movement across enterprise platforms.

Analytics Enablement

• Build data foundations that support self-service analytics.
• Enable reporting and visualization through Apache Superset and other analytics platforms.
• Ensure data readiness, performance, governance, and secure access for analytics users.

Required Skills & Qualifications

• Bachelor's Degree (Any Discipline).

• 8+ years of experience in Data Engineering and Data Platform development.

• Proven expertise in designing and implementing large-scale distributed data platforms.

• Strong hands-on experience with:

• Apache Kafka

• Apache Flink

• Hadoop Ecosystem

• Apache Iceberg

• Apache Airflow

• Debezium

• Apache Atlas

• Apache Superset

• Strong understanding of:

• Data Warehousing Concepts

• Medallion Architecture

• Data Modeling

• Data Governance

• Data Quality Management

• Security & Access Control

• Experience with Change Data Capture (CDC) and Real-Time Streaming Architectures.

• Strong programming skills in Python, Java, or Scala.

• Experience working in globally distributed teams and environments.

Preferred Skills

• Experience in Logistics, Supply Chain, or Transportation domain.
• Knowledge of multi-region compliance and regulatory requirements.
• Exposure to AI/ML data pipelines and feature engineering.
• Experience integrating with platforms such as Snowflake, Databricks, or similar analytics ecosystems.
• Contributions to or experience working with open-source communities/projects.

Desired Competencies

• Strong leadership and mentoring capabilities.
• Excellent stakeholder management and communication skills.
• Strategic thinking with a hands-on execution mindset.
• Strong analytical and problem-solving abilities.
• High ownership and accountability.
• Focus on engineering excellence and business value delivery.

Employment Details

• Role: Senior Open Source Data Engineer
• Experience: 8+ Years
• Industry: IT Services / Product Development / Data Engineering
• Employment Type: Full-Time
• Education: Any Graduate