Yassir Idhbi

- Modernized Python projects by integrating a Dependency Manager (Poetry), Linters, Formatters, etc., to standardize developers' tools and streamline their workflow. - Enhanced CI/CD Pipelines on Jenkins by adding code quality checks and unit tests before deployment. - Maintained Spark ingestion code and added new tables to the platform. - Upgraded the platform's data storage from CSV to Parquet and Iceberg formats. - Integrated Glue Crawlers into pipelines to automatically catalog tables in the Glue catalog. - Implemented table lineage tracking using OpenLineage and DataGalaxy on Spark jobs - Developed connectors for the Anaplan management tool to automatically extract and integrate files from/to AWS using the Anaplan API. Python · Apache Spark · AWS Lambda · AWS · AWS Glue · Amazon S3 · Apache Iceberg · Poetry · Jenkins · Terraform · Anaplan · AWS StepFunctions

- Built AWS infrastructure for a high-volume data streaming pipeline (EMR, Kinesis, S3, Lambda). - Created and maintained the project's CI/CD pipelines (Terraform, Spark with Scala, Python Lambdas). - Maintained the core application code in Spark Streaming. - Developed a Lambda function to monitor business aspects with alerts sent to Google Chat. - Set up dashboards in Grafana to monitor infrastructure load (EMR, Kinesis, S3, etc.). - Built AWS infrastructure for a high-volume data streaming pipeline (EMR, Kinesis, S3, Lambda). - Created and maintained the project's CI/CD pipelines (Terraform, Spark with Scala, Python Lambdas). - Maintained the core application code in Spark Streaming. - Developed a Lambda function to monitor business aspects with alerts sent to Google Chat. - Set up dashboards in Grafana to monitor infrastructure load (EMR, Kinesis, S3, etc.). GitLab · Python · Apache Spark · AWS · Terraform · Scala · Grafana · Amazon Kinesis

- Maintained and enhanced Spark (Scala) ingestion jobs on the platform. - Managed the orchestration of pipelines using Step Functions and Lambdas (Python). - Improved platform catalog readability by implementing a tool to easily add metadata to the Glue catalog from a configuration file. - Expanded unit tests and end-to-end (E2E) tests across the platform's codebase. - Maintained and improved the platform's CI/CD pipelines. - Managed AWS infrastructure as code using CloudFormation. - Maintained and enhanced Spark (Scala) ingestion jobs on the platform. - Managed the orchestration of pipelines using Step Functions and Lambdas (Python). - Improved platform catalog readability by implementing a tool to easily add metadata to the Glue catalog from a configuration file. - Expanded unit tests and end-to-end (E2E) tests across the platform's codebase. - Maintained and improved the platform's CI/CD pipelines. - Managed AWS infrastructure as code using CloudFormation. Python · Scala · Apache Spark · AWS Glue · AWS Lambda · AWS StepFunctions · GitLab · Apache Airflow · AWS CloudFormation

- Developed a serverless web application on AWS. - Implemented Lambdas to process data on S3 (storage, compression, decompression, etc.) and interact with DynamoDB. - Created the Rest API consumed by the application's frontend. - Maintained the authentication system using AWS Cognito. - Implemented a GraphQL API with AWS AppSync, secured by Cognito. - Deployed infrastructure as code with CloudFormation. Node.js · Appium · Amplify · AWS · React · GraphQL

Python, AWS (Kinesis, Athena, DocumentDB, Lambda, API Gateway, EMR), Spark, Angular, Terraform, Serverless Framework * Built a cloud architecture for a recommandation system on AWS using Terraform * Helped maintain the recommendation system Spark job and adapted it to the AWS infrastructure * Built a demo web app using Angular and Serverless to showcase the recommendation system's results