Adib Khaffaji

* Designed and productionized advanced SQL data processing in BigQuery, with automated execution through an internal ETL framework. * Built Python utilities to dynamically generate ETL commands based on workflows, dependencies, parameters, and deployment environments. * Designed incremental data processing and optimized MERGE operations to minimize reprocessing and reduce scanned data volumes. * Defined partitioning and clustering strategies based on data volume, cardinality, access patterns, and query workloads. * Optimized partition pruning, predicate filtering, aggregations, and join strategies to improve performance and reduce BigQuery processing costs. * Analyzed query execution plans and runtime metrics to identify expensive joins, data redistribution, skew, and processing bottlenecks. * Reworked complex SQL pipelines to reduce repeated scans, unnecessary intermediate aggregations, materializations, and data movements. * Optimized large-scale joins through early filtering, pre-aggregation, operation reordering, and reduction of intermediate datasets. * Reused recurring transformations through intermediate tables and materialized results, reducing duplicated processing across pipelines. * Monitored bytes processed, execution time, slot consumption, and query behavior to identify and redesign inefficient workloads. * Developed and optimized distributed data transformations using Apache Spark / PySpark, with attention to partitioning, shuffles, joins, and execution plans. * Applied Spark performance optimization techniques such as filter pushdown, partition pruning, broadcast joins, repartitioning, and reduction of unnecessary shuffles. * Worked with Databricks and Delta Lake for scalable data processing, incremental workloads, and reliable table management. * Implemented incremental processing patterns using Delta Lake MERGE, supporting idempotent ingestion and update workflows. * Worked with Bronze / Silver / Gold data layers to separate raw ingestion, data cleansing and enrichment, and business-ready datasets. * Used Databricks Jobs and workflow orchestration to structure dependent processing steps and production-oriented data pipelines. * Applied data reliability principles including idempotency, data quality controls, failure recovery, and safe pipeline re-execution. * Investigated performance issues using Spark execution plans, partition distribution, shuffle behavior, and stage-level processing characteristics. Environment: Google Cloud Platform (GCP), BigQuery, Databricks, Apache Spark, Delta Lake, internal ETL framework. Technologies: Advanced SQL, Python, PySpark, BigQuery, Spark, Databricks, Delta Lake. Team: Collaboration with Data Engineers, business analysts, and technical leads. Working method: Project-based delivery, request tracking, technical analysis, implementation, testing, and validation with business teams.

▪ Developed Spark jobs with PySpark for large-scale data ingestion and processing. ▪ Implemented unit tests with PyTest to improve pipeline quality and reliability. ▪ Optimized Spark jobs to reduce processing costs and improve performance. ▪ Optimized data schemas based on storage formats (Parquet, Avro) and partitioning strategies. ▪ Implemented automatic reprocessing for corrupted files in Spark and error detection from HDFS logs. ▪ Benchmarked Parquet versus Avro to select the most suitable format based on compression and Dremio read performance. ▪ Wrote SQL queries in Dremio to validate processing results and verify business data consistency. ▪ Contributed to source-code management in GitLab, using feature branches and weekly team code reviews. Environment: Hadoop cluster, Dremio, GitLab, CI/CD. Team: 2 business analysts, 8 data engineers, a technical lead, and a data architect. Technologies: PySpark, Python, PyTest, Spark, Bash, SQL. Working method: Agile (Scrum), weekly code reviews, daily meetings.

▪ Developed PySpark and Scala scripts to load and transform large-scale datasets in HDFS. ▪ Optimized Spark jobs through parameter tuning, including partitioning and broadcast joins, supported by log analysis. ▪ Monitored and managed Hadoop cluster resources (YARN, HDFS) to maintain balanced task distribution. ▪ Orchestrated workflows with Airflow and configured DAGs to automate complex processing tasks. ▪ Implemented monitoring metrics from YARN and Airflow logs to detect long-running or failed jobs and automatically alert the team by email. ▪ Developed Bash scripts to automate cleanup of old HDFS files according to business retention rules (purge after 30 days). ▪ Extracted YARN metrics (memory, CPU, duration) from the web UI to analyze slow jobs and recommend memory and partition adjustments. ▪ Added automated schema validation to reject incoming files that did not match the expected column names, types, or order. ▪ Fed Power BI dashboards with processed data. Environment: Hadoop cluster, GitLab. Team: 4 data engineers with collaboration across the BI team. Technologies: Hadoop, PySpark, Scala, YARN, HDFS, Hive, Airflow, Power BI.

Designed and developed PySpark ETL pipelines to ingest, transform, and process large volumes of text data. ▪ Developed PySpark and Scala scripts for data preprocessing and transformation of datasets stored in HDFS. ▪ Built, managed, and monitored Apache Airflow workflows to orchestrate extraction, transformation, and loading tasks. ▪ Improved Spark performance by reducing stages, shuffles, and redundant actions, and by optimizing grouped aggregations and intermediate datasets. ▪ Implemented compatibility tests between GCP and the existing environment, including exports from HDFS to Cloud Storage and ingestion into BigQuery. ▪ Contributed to the progressive migration of data workflows to GCP by adapting Airflow scripts to Cloud Composer and storing intermediate files in Cloud Storage. ▪ Designed an event-driven GCP pipeline triggered by new files in Cloud Storage, using Pub/Sub and Cloud Functions to parse and load cleaned data into BigQuery. ▪ Redesigned and optimized BigQuery analytical tables using partitioning, clustering, and column pruning, reducing read costs by up to 40% for selected queries. ▪ Improved Looker Studio load times by filtering data at the source in BigQuery, using materialized views and nested-column structures. ▪ Built dashboards in Tableau Desktop and Looker Studio for data monitoring and analysis. ▪ Implemented monitoring dashboards to track BigQuery query latency and identify expensive queries using Audit Logs. Environment: Hadoop, HDFS, YARN, Spark, PySpark, Scala, Airflow, MySQL, GCP, Cloud Composer, Cloud Storage, Pub/Sub, Cloud Functions, BigQuery, Looker Studio, Tableau Desktop, GitLab. Team: 6 Data Engineers and 2 business analysts. Working method: Agile.

Skills: Hadoop & Spark · Scikit-learn · NumPy · Linux · Pandas