PySpark/Scala Developer

About the Role

We are looking for skilled PySpark/Scala Developers to join a global banking risk analytics team. The role involves designing and developing large-scale data pipelines, implementing machine learning models, and building robust data processing solutions on Big Data platforms.

The ideal candidate will have expertise in PySpark, Scala, Hadoop ecosystem technologies, and experience within Credit Risk, Regulatory Risk, Banking, or Financial Services domains.

Key Responsibilities

• Design, develop, and optimize scalable ETL/ELT pipelines using PySpark and Scala.
• Build ingestion frameworks for structured, semi-structured, and unstructured data sources.
• Develop and enhance machine learning model pipelines in partnership with data scientists.
• Implement feature engineering, data transformation, model scoring, and retraining processes.
• Build reusable Spark-based frameworks for batch and near real-time data processing.
• Optimize Big Data workloads running on Spark, Hadoop, EMR, Databricks, or similar platforms.
• Ensure data quality, lineage, governance, and documentation standards.
• Collaborate with Risk Analytics, Data Science, and Business teams to deliver data solutions.
• Support deployment, version control, CI/CD practices, and production support activities.

Mandatory Skills

• 5+ years of experience in Data Engineering, Big Data Development, or Software Development.

• Strong hands-on experience in:

• PySpark

• Scala

• Python

• Spark

• Hive

• Hadoop Ecosystem

• MapReduce

• Unix Shell Scripting

• SQL

• Experience building enterprise-grade data pipelines and ingestion frameworks.

• Experience with distributed computing environments such as Hadoop, Spark, Databricks, or AWS EMR.

• Strong understanding of Data Lake and Data Warehouse concepts.

• Experience working with Banking, Financial Services, Credit Risk, or Regulatory Risk projects.

• Knowledge of relational databases and platforms such as PostgreSQL, Redshift, or equivalent.

Preferred Skills

• Spark Structured Streaming
• Kafka
• Delta Lake
• AWS Glue
• Machine Learning frameworks (Spark ML, Scikit-Learn, TensorFlow, PyTorch)
• Jenkins, Git, CI/CD pipelines
• Data Governance and Data Lineage tools

Domain Preference

Candidates with experience in:

• Credit Risk
• Regulatory Risk
• Risk Modeling
• Consumer Banking
• Wholesale Banking
• Financial Services

Education

Master's or Bachelor's Degree in:

• Engineering
• Statistics
• Mathematics
• Finance
• Computer Science