Senior AWS Data Engineer (Python & Spark)

Key Responsibilities

• Design, develop, and maintain scalable data platforms and ETL solutions on AWS.
• Build and optimize high-performance data pipelines using Python and Spark.
• Develop data ingestion, transformation, and processing frameworks leveraging AWS services.
• Implement and maintain Lakehouse architecture solutions using modern data engineering principles.
• Lead migration of existing data processing workloads to Lakehouse platforms utilizing Apache Iceberg capabilities.
• Collaborate with architects, product owners, and development teams to design and deliver enterprise data solutions.
• Break down solution designs into epics, stories, and technical deliverables.
• Ensure data quality, reliability, scalability, and performance across data platforms.
• Conduct code reviews, promote engineering best practices, and support quality assurance initiatives.
• Create and maintain technical documentation, architecture diagrams, and implementation standards.
• Support Agile delivery methodologies and contribute to sprint planning and technical estimations.

Required Skills

• Proven experience as a Senior Data Engineer.

• Strong expertise in Python and Apache Spark.

• Extensive experience building ETL pipelines and data processing frameworks.

• Hands-on experience with AWS services including:

• AWS EMR

• AWS Glue

• AWS Lambda

• AWS Step Functions

• API Gateway

• Amazon Athena

• Experience working with Data Lakehouse architectures.

• Strong understanding of data modelling and enterprise data platform design.

• Experience handling large-scale data processing workloads.

• Knowledge of data engineering best practices including testing, validation, and performance optimization.

• Strong communication and stakeholder management skills.

Preferred Skills

• Financial Services or Capital Markets experience.

• Experience working with market data and reference data platforms.

• Knowledge of Data Governance and Data Management frameworks.

• Experience with:

• Metadata Management

• Master Data Management (MDM)

• Data Quality frameworks

• AWS Certifications such as:

• AWS Certified Solutions Architect

• AWS Certified Data Analytics

Bonus Skills

• Knowledge of Indices and Index Construction methodologies.
• Asset Management domain experience.
• Exposure to modern open-table formats such as Apache Iceberg.

Desired Candidate Profile

• Strong technical leadership and mentoring capabilities.
• Excellent problem-solving and analytical skills.
• Ability to communicate complex technical concepts to technical and non-technical audiences.
• Experience working in Agile product teams.
• Strong ownership mindset with a focus on delivering scalable and maintainable solutions.
• Passion for modern cloud-based data engineering and platform modernization.