BAILO BAH

En tant que Tech Lead, responsable de la gestion et de l’encadrement d’une équipe technique pour la mise en œuvre et l’évolution des flux d’intégration de données dans le Datalake. Expertise en architecture Big Data, suivi des run en production, et maintenance évolutive (MCO). Expérience en gestion des pipelines d’ingestion de données provenant de sources variées (streaming, batch, CSV, API), ainsi que dans la conception de connecteurs entre les systèmes d’information et le Datalake. Gestion des métadonnées en lien avec les équipes de conformité et gouvernance (GDD). Capacité à définir et automatiser le déploiement des versions, configurer la CICD, et gérer l’accès aux données. Compétent en suivi de production, support de niveau 3, et gestion des recettes internes.

• Work in a fast-paced agile development environment to quickly analyze, develop, and test potential use cases for the business • Designed and implemented big data lake with Cloudera Data Platform of electricity/Gas usage. • Design and build ETL pipelines to automate ingestion of structured and unstructured data from API/CSV/json/ftp/Database SQL server/kafka/ etc with scala language. • Designed and implemented a real time/ driven event data pipeline to process semi structured, structured data. . Move application to cloud AWS

Responsibilities: • Collaborated with PO in understanding the requirement and architect a data flow system. • Design and developing complete end to end Framwork data processing with spart scala. • Used Spark API 2.2.0 over Hortonwort to perform analytics on data in Hive. • Implemented extensive Hive queries and creating schemas external, internal tables and views for adhoc and business processing using performance techniques like partitioning and bucketing. • Developed Spark code using Scala and Spark-SQL/Streaming for faster processing of data. • Developed Scala scripts, UDF’s using both Data frames/SQL and RDD/MapReduce in Spark 2.0.0 for Data Aggregation, queries and writing data back into Hive. • Developed Oozie 4.1.0 workflow jobs to execute hive 2.0.0, Spark jobs and map-reduce actions. • Used TOM/Talend to transfer and receive the files from various upstream and downstream systems. • Involved in complete end to end code deployment process in Production. • Worked on upgrades Spark 1.6 to Spark 2.2.0 and did the regression testing and unit tests. • Involved in some product business and functional requirement through gathering team, update the user comments in JIRA and documentation in Confluence. Environments: HDP, SPARK 2.2.0, Hive 2.0.0, Map Reduce, Oozie , Agile Methodology, JIRA, Git

Responsibilities: • Collaborated with the Internal/Client BA’s in understanding the requirement and architect a data flow system. • Establish and develop a framework for delivering RESTful APIs to external customers for Data exposition. • Create data model for external RESTful API configuration including API key management. • Developed Spark code using Scala and Spark-SQL/Streaming for faster processing of data. • Developed Scala scripts, UDF’s using both Data frames/SQL and RDD • Developed Oozie workflow jobs to execute hive 2.0.0, Spark jobs . • Involved in complete end to end code deployment process in Production. • Worked on upgrades Spark 1.6 to Spark 2.2.0 and did the regression testing and unit tests. • Involved in some product business and functional requirement through gathering team, update the user comments in JIRA and documentation in Confluence. Technologies utilized: Cloudera, Spring Boot, Spring Security, Spring Data Rest, JPA, Agile Methodology, JIRA, Git

• Design and Build data cleaner for automatic cleaning of parallel data (NLP), Machine Learning with PySpark, Hive • Designing, developing and maintaining of a mediation and administration platform for the automation of training statistical models and their deployments in Prod, architectural design and development. • Urbanization and transformation of a bash data processing chain into a transactional service using the Spark / Kafka / ActiveMq message manager • Developing of a ‘Data hub’, data crapping robots (parallel source / target documents), transformation, cleanning and alignment (translation memory) • Designing, developing and maintaining new generation machine learning based big-data web page categorization, data/IP mining, and malicious site detection systems • Implementation of a log monitoring architecture with ELK on a pools of 5 production servers (Installation, configuration, development, securing and monitoring) • Development of a plugin to integrate our API into the Trados KIT, .NET Framework Environments: Spark, SparkStream, PySpark, KafKa, Cassandra, MysQL, Elasticsearch, logstash, kibana, filebeat ActiveMQ, .NET, JAVA, Python, Perl