ANDRE KONGA

𝗗𝗮𝘁𝗮 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻, 𝗢𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝗮𝗹 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀 𝗗𝗲𝗹𝗶𝘃𝗲𝗿𝘆 𝟭. 🤖 𝗠𝗶𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗼𝗻 • Development of Python scripts for automatic splitting of large files • Conversion and standardization of encodings to UTF-8 to ensure compatibility • Implementation of a data ingestion pipeline into a MySQL database 𝟮. 🔒 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 • Definition of a robust architecture with referential integrity and validation constraints • Implementation of an access and permissions management system • Creation of dynamic views to limit exposure of sensitive data 𝟯. ⚙️ 𝗢𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗼𝗻 𝘄𝗶𝘁𝗵 𝗔𝗶𝗿𝗳𝗹𝗼𝘄 • Airflow pipeline for periodic ingestion of files from AWS S3 to MySQL via S3KeySensor and PythonOperator / BashOperator operators • Conditional DAG routing via BranchPythonOperator based on detected file type or quality, with inter-task data passing via XCom • Sensor optimization: choice between modes (poke, reschedule, and deferrable) 𝟰. 📊 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝗮𝗹 𝗗𝗮𝘁𝗮 𝗩𝗮𝗹𝗼𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻 (𝗕𝗶𝗴𝗤𝘂𝗲𝗿𝘆) • Preparation and exposure of data to an analytical Data Warehouse (BigQuery) for reporting and BI use cases • Writing of analytical SQL queries (views, aggregations) for business consumption of data • Analytical data modeling and BI-oriented structuring (tables, views, KPIs) 🔧 Tools: Agile, Shell Linux, Web2py, Apache Airflow, Python, SQL, MySQL, AWS S3, Web2Py, GCP, BigQuery

𝗦𝗰𝗮𝗹𝗮𝗯𝗹𝗲 𝗗𝗮𝘁𝗮𝗹𝗮𝗸𝗲 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲, 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴 & 𝗦𝗻𝗼𝘄𝗳𝗹𝗮𝗸𝗲 𝗗𝗮𝘁𝗮 𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗼𝗻 𝟭. 🔐 Datalake Access Management • Collected and analyzed datalake consumers' requirements. • Defined permissions, roles, and user mappings for each consumer. • Deployed AWS configurations via CloudFormation using a GitHub Actions CI/CD pipeline. 𝟮. 📦 Processing and Storage optimisation • Developed PySpark transformation pipelines and optimized Spark cluster performance. • Developed Lambda functions to move, archive, and delete files. • Set up AWS EventBridge schedulers to orchestrate Lambda executions. 𝟯. 🛠️ Daily Monitoring & Pipeline Debugging • Assessed raw data extraction rates through Splunk dashboards. • Detected and categorized Step Functions errors. • Performed advanced debugging in CloudWatch by analyzing Step Functions execution traces. 𝟰. ❄️ Incremental Ingestion and CDC S3 → Snowflake: • Implementation of Snowpipe (AUTO_INGEST=TRUE) with S3 external stage, IAM storage integration, and SQS queue for event-driven ingestion into Bronze tables • Definition of CDC Streams on Bronze tables and writing of MERGE CDC queries (insert/update/delete) to feed the Silver layer • Creation of optimized Snow Tasks (SYSTEM$STREAM_HAS_DATA) to avoid empty runs 5. 🎯Medallion Modeling and Snowflake Cost Optimization: • Medallion modeling: 3NF-normalized Silver, Kimball-model Gold (facts/dimensions) with surrogate keys and refresh via Dynamic Tables • Definition of CDC Streams on Bronze tables and writing of MERGE CDC queries (insert/update/delete) to feed the Silver layer • Storage optimization: TRANSIENT tables in Bronze, PERMANENT in Gold/Silver • Warehouse optimization: X-Small/Small for merge tasks, Medium for Gold analytical queries, with auto-suspend at 60 seconds 𝗧𝗼𝗼𝗹𝘀 𝗨𝘀𝗲𝗱: Python, PySpark, SQL, YAML, IAM, S3, KMS, Lambda, Glue, Athena, Splunk, Docker, Snowpipe, Snow Stream, Kimball