Yousra Maqsoud

• Designed and implemented a data quality framework across Lakehouse layers (Bronze, Silver, Gold) to ensure reliability and traceability of data pipelines. • Developed scalable PySpark validation pipelines on Databricks to enforce data quality rules and detect anomalies during ingestion and transformation. • Implemented cross-system data reconciliation between MongoDB/PostgreSQL source systems and Lakehouse datasets. • Built monitoring and data observability dashboards to track data quality KPIs, pipeline health, and anomaly detection. • Implemented automated rejection handling mechanisms with structured rejected-record tables to isolate invalid data and support remediation workflows. • Optimized PySpark processing and validation jobs to improve performance and scalability on large datasets. • Orchestrated validation and transformation pipelines using Databricks Workflows with automated scheduling and dependency management. • Implemented Delta Lake lifecycle management strategies, including automated purge and retention policies. • Partnered with data owners, producers, and governance teams to define data quality standards, validation rules, and remediation workflows across the platform.

• Designed and implemented Hadoop-based data pipelines using Spark, Hive, and ODI to process and transform large datasets. • Developed Spark applications reading from HDFS for data transformation, preparation, and enrichment workflows. • Built data processing workflows using Oozie for orchestration and scheduling of batch jobs. • Implemented data archiving and purging mechanisms to manage data lifecycle within the Hadoop ecosystem. • Developed Spark pipelines to index processed data into Elasticsearch and enable indicator visualization through Kibana dashboards. • Performed job monitoring, debugging, and configuration tuning to ensure reliable execution of Hadoop workloads. • Conducted validation and automated functional testing using Cucumber to ensure pipeline reliability.

• Designed and developed a recommendation system for an ENGIE web platform using machine learning techniques. • Performed extensive data cleaning and preprocessing on raw datasets, including normalization of free-text fields, removal of special characters, and data standardization. • Identified and reported data quality issues in source systems (e.g., uncontrolled free-text inputs) and proposed improvements to improve data consistency. • Conducted feature engineering and exploratory analysis to prepare datasets for machine learning models. • Evaluated multiple machine learning approaches and implemented a K-Means clustering model to generate recommendation segments. • Built data preparation pipelines to transform and structure raw data for analytical and machine learning use.