adel skhiri

- Led the implementation of streaming dataflow pipelines, resulting in a robust data platform that empowered data producers and consumers with greater autonomy and efficiency. - Significantly contributed to the development of an open-source data ETL framework based on PySpark, which has become a widely-used tool for data processing by our data engineering team. - Successfully migrated Airflow to Cloud Composer, a fully managed cloud-based solution, resulting in cost savings and improved scalability of the data platform. - Led the migration of the data processing system from Spark 2.4 to Spark 3, resulting in enhanced performance and efficiency of the data platform for faster processing and analysis. - Collaborated closely with cross-functional teams, including data scientists, data analysts, and developers, to ensure a seamless and efficient integration of the data platform with other systems and processes and to insure the use of data engineering best practices.

- Successfully moved our ETL system from on-premise to the cloud on AWS, using EMR, Athena, and Glue to build scalable and efficient data platform. - Industrialized and developed Spark applications on Scala to optimize data processing and enable easier access to data for different teams within the company using SQL queries on Athena.

- Participated in the capacity planning of an ELK stack, which enabled the processing of 1TB of data per day, and managed the run time of the stack. - Developed solutions for data aggregation and timeseries forecast preparation, utilizing Kafka and Spark, resulting in a faster and more accurate data processing system. - Worked collaboratively with the team to design and implement the solutions, utilizing best practices and ensuring scalability and maintainability of the system.

- Prepared data for analysis and built interactive dashboards using ELK (Elasticsearch, Logstash, Kibana) and Spark, allowing stakeholders to visualize and explore data in real-time and make informed decisions based on key metrics.

Led the development of custom captive portals for businesses that integrate with Facebook and Google to improve user engagement and data collection.