Big Data Engineer

Design, develop, maintain, and optimize data pipelines and data processing workflows.

• Develop and maintain solutions using Python, PySpark, Spark SQL, SQL, Hadoop/Hive, MongoDB and Apache Spark.
• Work with batch and real-time/streaming data processing.
• Support data modeling, data transformation, and data integration activities.
• Analyze large datasets to identify trends, anomalies, data quality issues, and opportunities for improvement.
• Address and resolve defects and performance issues in data pipelines.
• Support architectural changes and modernization initiatives.
• Identify opportunities for automation, performance optimization, and operational efficiency.

Platform & Production Operations

• Perform daily monitoring of METIS scheduled jobs, microservices, applications, data pipelines, and Power BI report refreshes.
• Monitor and manage workflows and schedules using Apache Airflow.
• Perform configurable changes, scheduling, restart/reprocessing activities, and operational troubleshooting.
• Perform manual/controlled data refreshes for various enterprise data sources.
• Troubleshoot failed jobs, data pipeline failures, application issues, and data quality problems.
• Perform root-cause analysis and implement sustainable solutions.
• Support production releases, deployments, and operational activities.
• Ensure platform availability, stability, and data freshness.

Incident, Problem & Change Management

• Handle Incident, Problem and Change tickets in accordance with ITIL processes.
• Perform root-cause analysis for recurring incidents and data issues.
• Coordinate with relevant development, infrastructure, application, and source-system teams for issue resolution.
• Maintain operational documentation and knowledge articles.
• Participate in production support and provide timely communication to stakeholders.

Data Quality & Testing

• Perform data quality checks during development and production.
• Identify, analyze, and resolve data inconsistencies and data quality issues.
• Conduct unit, integration, regression, and QA testing.
• Validate data pipelines, transformations, reports, and downstream data consumption.
• Establish and improve data quality and monitoring mechanisms.

Stakeholder & Requirement Management

• Act as a technical interface between business stakeholders, data engineering teams, application teams, architects, infrastructure teams, and data consumers.
• Understand business needs and translate them into clear technical/data requirements.
• Conduct requirement discussions, clarification sessions, and impact assessments.
• Identify new business use cases that can be enabled using existing METIS data.
• Prepare and maintain requirements, technical specifications, process documentation, and solution proposals.
• Communicate technical topics and issues clearly to both technical and non-technical stakeholders.
• Coordinate cross-functional activities and ensure timely delivery of agreed requirements.
• Proactively identify risks, dependencies, and potential improvements.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.