Baye Ndao

– Creating and maintaining technical documentation. – Designed and implemented a DBT unit testing framework to automate SQL validation within a GitHub Actions CI/CD pipeline, ensuring that every change to a dbt model is rigorously tested before being merged or deployed. – Implementing new DQC checks and data validation functions throughout the pipelines with DBT test functions, Great Expectations & DBT-expectations. – Built a client-friendly UI to onboard new clients, automating DQCs, daily & snapshot files validation with auto-feedbacks. – Building and maintaining efficient, reusable, and reliable Python code modules and data models to construct Airflow DAGS. – Leading the development of data pipelines, significantly improving data integration from diverse sources for our new clients. – Implemented a monitoring DAG for the data warehouse to measure the daily duration, resource usage, cost, and error rate of BigQuery jobs, as well as the rows and volumes of tables, and the metadata of GCS files. – Identifying bottlenecks and improving them to reduce the airflow DAGs' duration. – Maintaining code quality and updating deprecated components. – Significantly optimized the SQL queries to enhance performance and minimize costs. – Implemented a comprehensive alerting system for data pipeline issues, ensuring immediate response to avoid scoring delay and workflow reliability. – Onboarding and managing Junior Data Engineers and Data Analysts

– Implemented ETL processes, such as data transformation using tools including Python, Airflow, DBT, Beam, BigQuery, and SQL. – Explored available technologies and developed solutions to continuously improve the data quality, workflow reliability, and scalability. – Implemented an alerting system to notify the team on Slack immediately whenever a data pipeline task fails, or the data ingestion is delayed. – Developed Web Application using Python, Django, Flask, APIs, Postgres, Redis, JavaScript