Zakaria Difallah

- Designed and implemented layered data pipelines (Bronze / Silver / Gold – Raw / ODS / Curated) using dbt on BigQuery, structured to support analytical and reporting use cases. - Developed containerized Python connectors deployed on Google Cloud Run to ingest source data into the Bronze layer. - Optimized dbt models on BigQuery for cost and performance by systematically adopting incremental modeling, reducing slot consumption and minimizing the volume of data processed during each execution. - Implemented automated dbt data quality tests (uniqueness, non-null, referential integrity) to ensure the reliability of the Silver and Gold layers, and generated dbt documentation and data lineage to improve traceability and facilitate knowledge transfer to downstream teams. - Automated workflow orchestration using Google Cloud Workflows, enabling parallel task execution, dependency management, and scheduled pipeline runs. - Designed and maintained a microservices architecture based on FastAPI, PostgreSQL, and Redis, exposing REST APIs consumed by both internal and external clients while managing the full API lifecycle. - Built and maintained multi-environment CI/CD pipelines, ensuring reliable deployments across development, staging, and production environments. - Contributed to the evolution of a cloud architecture under active development, balancing short-term business requirements with long-term architectural goals around scalability, cost efficiency, and reliability. - Established an MLflow evaluation framework to assess the quality of responses generated by an integrated LLM, measuring relevance and output quality while enabling objective comparisons across prompt versions and model configurations. - Improved the platform's LLM capabilities by enriching the inference pipeline with new input features and iteratively refining prompts to enhance response relevance and overall performance.

- Development of High-Performance Big Data Solutions: Design, implementation, and deployment of robust pipelines leveraging advanced technologies such as NiFi, Kafka, Cloudera, HDFS, PySpark, and Hive to meet scalability and performance needs. - Data Ingestion and Transformation: Building optimized data flows to address analytical and operational requirements, ensuring actionable outcomes. - Workflow Optimization with Dataiku: Automating and simplifying complex processes using advanced Dataiku capabilities, ensuring reliability and efficiency. - Team Support: Upskilling teams on Dataiku, sharing best practices, and integrating collaborative workflows. - Interdisciplinary Collaboration: Working closely with stakeholders to design and implement tailored solutions.

- Python and SQL Notebook Audit: In-depth analysis of existing scripts and queries to identify optimization opportunities, ensuring enhanced performance. - Code Optimization: Refactoring and improving code blocks to strengthen efficiency, readability, and long-term maintainability. - Standardization of Development Practices: Implementation of best practices in Python and SQL, promoting consistent and robust workflows. - Proactive Collaboration: Regular engagement with teams to understand business needs and ensure deliverables align perfectly with expectations.

- Development of Data Engineering Tools: Creation and maintenance of internal tools for data importation and deployment, integrating Airflow and advanced Dataiku plugins. - Geospatial Data Analysis and Processing: Leveraging SQL, Python, Dataiku, and PostGIS to process, analyze, and visualize complex geospatial data. - Vision Inference Infrastructure Management: Maintaining a high-performance infrastructure based on Python, GCP, BigQuery, Redis, Docker, and Kubernetes, optimized for machine learning models. - PostgreSQL Database Migration: Conducting reliable migrations using dump/restore methods, ensuring operational continuity. - Automation of Data Workflows: Designing automated processes for data archiving and restoration between databases and GCP buckets, reducing costs and ensuring optimal preservation. - Pipeline Orchestration and Optimization: Maintaining and automating pipelines using Python and GitLab CI/CD, ensuring fast and reliable deployments. - API Integration for Data Enrichment: Automating metadata insertion by connecting the JIRA API to the database for efficient tracking. - Data Quality Assurance: Integrating Great Expectations in Python to ensure optimal data governance and quality. - Talent Development and Mentorship: Training and mentoring interns, fostering their skill growth and project success. - Project Management: Efficiently planning and coordinating tasks using JIRA and Trello to align deliverables with objectives.

- Student Training and Support: Conducting educational sessions to help students master the use of devices and online services, promoting their digital autonomy. - Technical Assistance and Mentorship: Providing personalized support to guide students in completing their programming projects, delivering practical solutions, and enhancing their technical skills.