Senior AWS Data Engineer

Key Responsibilities

• Build and maintain scalable data pipelines across stages such as data download, ingestion, and analysis within a bespoke data platform.
• Develop high-performance data processing logic using Python and PySpark.
• Perform large-scale transaction data analysis using custom algorithms and in-house logic to detect financial crime patterns.
• Work with high-volume datasets and optimize pipeline performance.
• Design efficient data models and transformations for large-scale processing.
• Write optimized SQL queries on PostgreSQL (RDS), leveraging: Window functions / Partitioning /Query performance optimization
• Languages: Python, SQL
• AWS exposure (especially EMR and S3)
• Work with Delta tables and Parquet-based storage formats for efficient data processing.
• Build and maintain Spark batch and streaming jobs.
• Implement engineering best practices including unit testing, static code analysis, and CI/CD practices.
• Contribute to data platform architecture and system design decisions.

Required Skills

• Strong Python programming expertise
• Deep understanding of: Functional programming/ Object-Oriented Programming (OOP)/ Design patterns/ Python execution and invocation mechanisms
• Hands-on experience with Apache Spark / PySpark
• Strong SQL expertise including: Window functions/ Partitioning/ Query optimization
• Experience building end-to-end data pipelines
• Data Platform Engineering
• Experience designing scalable data processing architectures
• Strong understanding of distributed data processing
• Familiarity with standard data pipeline engineering practices
• AWS exposure (especially EMR and S3)
• Experience with Apache Airflow for job orchestration and scheduling