Data Engineer

Job responsibilities:

Develop API-driven systems to govern, manage, and monitor large-scale batch Big Data applications.

Build scalable backend services and data engineering solutions that support data processing and operational workflows.

Develop and maintain data transformation processes using Spark, SQL, Hive, Python, Scala, and related technologies.

Work with AWS cloud services such as S3, EC2, EMR, Lambda, DynamoDB, and API Gateway to build cloud-native data and backend solutions.

Build and enhance workflow orchestration using Apache Airflow, including advanced DAG design, dependency management, scheduling, monitoring, and failure handling.

Define technical scope, objectives, and implementation approaches by participating in requirements gathering, technical research, and process definition.

Participate in architecture reviews, code reviews, performance tuning, and operational readiness activities.

Contribute to test planning and validation for application integrations, functional areas, and project deliverables.

Partner with product owners, data engineers, backend engineers, QA, DevOps, and other cross-functional teams to deliver high-quality solutions in an Agile/Scrum environment.

Troubleshoot production issues, improve system reliability, and drive automation across data and backend workflows.

Required Qualification:

5+ years of hands-on experience developing enterprise-scale applications, data platforms, or distributed systems.

5+ years of experience developing, and operating Big Data platforms and cloud-based infrastructure services, preferably using AWS EMR and the Hadoop ecosystem.

Hands-on experience with AWS, Spark, Python and/or Scala, Airflow, SQL, Hive, and related data processing technologies.

Proficiency in either Python or Scala, with the ability to build production-grade data pipelines and backend services.

Experience with AWS cloud services, including S3, EC2, EMR, Lambda, DynamoDB, and API Gateway.

Strong experience with Apache Airflow orchestration, including complex workflows, scheduling, dependency handling, monitoring, and operational support.

Strong understanding of data engineering concepts, including data pipelines, data transformation, batch processing, real-time processing, data quality, and performance optimization.

Excellent analytical and problem-solving skills.

Strong communication and presentation skills, with the ability to explain technical concepts clearly to both technical and non-technical audiences.

Experience with version control and CI/CD tools, including Git and Jenkins, for source code management, build automation, testing, and deployment workflows.

Ability to work effectively in cross-functional Agile/Scrum teams.

Preferred Qualification:

Bachelor s degree in Computer Science, Engineering, Mathematics, Information Systems, or a related technical field, or equivalent practical experience.

Preferred Qualifications

Experience with LLMs, Generative AI, Agentic AI, or AI-assisted engineering workflows.

Experience with API design, microservices, event-driven architecture, or serverless backend development.

Experience with CI/CD pipelines, infrastructure as code, automated testing, and production deployment practices.

Experience with data governance, metadata management, lineage, data observability, or operational controls for enterprise data platforms.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.