Data Engineer Scala

Data Engineer with scala

Job Summary:

We are seeking a highly skilled and motivated Data Engineer with 6+ years of experience in designing, developing, and maintaining scalable data platforms and pipelines. The ideal candidate will possess strong expertise in Apache Spark, Scala, Python, SQL, and Cloud Technologies (Azure/AWS), with the ability to independently own and deliver end-to-end data engineering solutions. The role requires working closely with business stakeholders, data scientists, and engineering teams to build reliable, high-performance, and scalable data ecosystems that support analytics, reporting, and advanced data-driven initiatives.

Key Responsibilities:

Design, develop, and maintain large-scale batch and real-time data pipelines using Apache Spark, Scala, and Python.

Build scalable and reliable data processing frameworks for ingesting, transforming, and integrating data from multiple sources.

Develop and optimize complex SQL queries, stored procedures, and data models to support reporting and analytics requirements.

Design and implement cloud-based data solutions using Azure and/or AWS services.

Create and manage ETL/ELT workflows using cloud-native tools such as Azure Data Factory, Azure Data Lake, AWS Glue, and Amazon S3.

Collaborate with business stakeholders, data analysts, and data scientists to understand requirements and deliver data solutions.

Monitor, troubleshoot, and optimize data pipelines to ensure performance, reliability, and data quality.

Implement best practices for data governance, security, scalability, and operational excellence.

Participate in code reviews, architecture discussions, and technical design sessions.

Support CI/CD implementation and automation of data engineering workflows.

Work with distributed data processing systems and contribute to platform modernization initiatives.

Mentor junior team members and contribute to knowledge-sharing activities within the team.

Experience Required:

6+ years of hands-on experience in Data Engineering, Data Warehousing, and Big Data technologies.

Strong experience developing scalable data pipelines using Apache Spark, Scala, and Python in enterprise environments.

Proven experience working with cloud platforms such as Microsoft Azure and/or AWS, including data storage, processing, and integration services.

Advanced knowledge of SQL, including complex query development, performance tuning, data modeling, and query optimization.

Experience designing and implementing end-to-end ETL/ELT workflows for large-scale data processing and analytics.

Demonstrated ability to independently own and deliver data engineering solutions from requirements gathering through deployment and production support.

Strong troubleshooting and problem-solving skills with experience resolving complex data quality, performance, and scalability challenges.

Experience working within Agile/Scrum teams and collaborating effectively with cross-functional stakeholders to deliver business-critical data solutions.

Preferred to Have Skills

Experience with Azure Data Factory (ADF), Azure Synapse Analytics, or AWS Glue.

Hands-on experience with Apache Kafka or other real-time streaming platforms.

Knowledge of Delta Lake, Databricks, or Lakehouse architectures.

Experience with CI/CD pipelines and DevOps practices for Data Engineering.

Familiarity with Data Governance, Data Quality, and Metadata Management frameworks.

Exposure to Generative AI, Machine Learning data pipelines, or Analytics platforms is an added advantage.