Mahmoud Mehdi

– Engineered financial data pipelines using Databricks, PySpark, and Delta Lake, ingesting and processing multi-source datasets (CFD Cash, CFD Positions, CFD Trades) from leading global investment banks including Morgan Stanley, Barclays, Goldman Sachs, and JP Morgan - ensuring data accuracy, consistency, and timely availability for downstream analytics. – Built scalable data pipelines on Palantir Foundry, leveraging the platform's transformation and governance capabilities to standardize financial data across the organization. – Orchestrated and monitored complex workflow scheduling using Stonebranch, ensuring reliable pipeline execution and SLA compliance.

– Contributed to the Delta Lake open-source project developed by the Apache community and Databricks. – Authored the "io.delta.implicits._" class, improving developer experience for Scala users. – Collaborated with Spark creators on a storage framework supporting Lakehouse architecture with engines including Spark, PrestoDB, Flink, Trino, and Hive.

– Built a real-time data collection system ingesting telemetry from power plants via AWS Kinesis, enabling real-time analytics for data scientists. – Enhanced data storage by migrating from CSV/Parquet to Delta Lake format using delta-rs, improving query performance and enabling ACID transactions. – Exposed Delta Lake tables for self-service querying through AWS Athena. – Implemented comprehensive unit testing with pytest and Pandas, executed automatically after each commit for continuous reliability. – Automated full infrastructure deployment using Terraform, ensuring reproducible and auditable environments.

– Led the technical team: defined project architectures, conducted FinOps studies, and estimated cloud budgets for each initiative. – Developed an Azure Function processing real-time event hub tickets to calculate client discounts on the fly. – Built Apache Spark jobs computing key client KPIs: coupon usage, purchase frequency, and campaign-based discount adoption. – Designed and deployed an API (Azure Functions + API Management + Delta Lake) used by all stores to determine product substitutes during order preparation. – Created a real-time API querying Azure Cosmos DB, enriched via event hub, to return client ticket history.

– Designed a comprehensive data architecture proposal for a client restructuring their data infrastructure. – Implemented a data mesh organizational model and served as solutions architect to define optimal cloud and data solutions. – Evangelized Delta Lake features and benefits within the client organization.

– Audited existing Apache Spark jobs and delivered optimization recommendations to enhance workflow performance. – Initiated and drove the migration strategy from Parquet to Delta Lake. – Implemented an SCD2 pattern framework with Pandas for the company's data science team. – Deployed Amazon Macie for PII data detection with automated CloudFormation-based deployment. – Configured AWS Glue DataBrew for automated PII handling through advanced transformations including replacement and encryption.

– Developed a big data royalty calculation system processing revenue from Deezer, Spotify, iTunes, and YouTube, computing producer payments based on contractual terms. – Applied Delta Lake expertise as an open-source contributor to enable ACID transactions on stored Parquet files. – Implemented SCD2 pattern using Delta Lake for full data lineage and change tracking. – Tuned Apache Spark jobs handling large-scale datasets for optimal performance. – Built data exposure APIs using AWS Lambda, API Gateway, and JOOQ.

– Managed complex data migrations from multiple legacy systems to RDS databases. – Wrote Apache Spark ETL jobs (Scala) processing automotive repair shop data on AWS EMR. – Developed a labor guide ETL estimating repair times for all vehicle parts, now a core product feature. – Optimized Spark job performance on EMR and migrated query-intensive data from RDS to Elasticsearch via AWS DMS.

– Contributed to a fraud detection solution that generated over €3 million in savings for AXA France and UK. – Designed the AWS data platform architecture for UK insurance claims data from multiple sources. – Developed Apache Spark jobs for data processing and used Spark GraphFrames for fraud detection and data insight exploration. – Automated infrastructure deployment with Terraform and exposed data via AWS Glue and Athena.

– Tackled diverse big data engineering use cases across data storage, processing, and exposure for one of Europe's largest retailers. – Worked extensively with Apache Spark/Scala, Kafka, Cassandra, and ElasticSearch for large-scale data management.

– Built a big data application predicting football match outcomes using Machine Learning algorithms with Spark/Scala. – Developed the full-stack application using Play Framework (Scala) and AngularJS.

– Built a DevOps platform establishing Agile/DevOps culture across development teams, providing Jira, Confluence, Bitbucket, SonarQube, and Jenkins. – Deployed the platform as code using Terraform, Ansible, Docker Swarm Mode, and Kitchen for testing.