Yassine Hsouna

Skills: Apache Airflow · dbt · Selenium · Docker · Machine Learning · Microsoft Power BI · Chatbot Development · Large Language Models (LLM)

The pipeline begins by fetching user data from an API. This provides a steady stream of diverse, structured user data for processing. Orchestration - Apache Airflow Apache Airflow orchestrates the entire pipeline, automating tasks like data ingestion, processing, and storage. It ensures reliability and schedules jobs while maintaining metadata in a PostgreSQL database. Streaming - Apache Kafka & Zookeeper Apache Kafka serves as the core messaging and streaming platform, enabling real-time data transfer from the PostgreSQL database to the processing engine. Zookeeper is employed for managing Kafka’s distributed infrastructure. Monitoring - Control Center & Schema Registry Kafka's Control Center provides visibility into the streaming data flow, while the Schema Registry manages and enforces data schemas to maintain data integrity across topics. Data Processing - Apache Spark Data is processed using Apache Spark, leveraging its distributed processing capabilities. A Spark master node coordinates tasks executed across multiple worker nodes, ensuring scalability and speed. Storage - Cassandra Finally, the processed data is stored in Cassandra, a highly scalable and distributed NoSQL database designed to handle large volumes of data with high availability.

Developed an automated web scraping solution for gathering movie reviews and ratings. Developed and implemented advanced algorithms to analyze user reviews, resulting in actionable insights.

Analysis of a company’s energy data. Identify strategies to optimize consumption.