Alex Christian Frank BILLANG

🔧 Maintenance and Improvement of a Data Ingestion System with Apache NiFi, Python & Spark 🌐 As part of a project aimed at centralizing and processing diverse urban data (geospatial, parking, bikes, green spaces, water quality of the Seine), I implemented a data ingestion system using Apache NiFi in standalone mode, then evolved to a NiFi cluster leveraging the distributed computing power of the Spark cluster. 🔍 Context: Data Sources: Integration of data from various sources (APIs, local files, S3, ELS. etc.). Storage: Data is centralized in MinIO, an S3-compatible storage solution for efficient and secure management. ⚙️ Processing: Python & Spark: Utilization of Pandas for data processing, and PySpark for large-scale analysis. Spark's distributed and parallel computing optimizes performance by efficiently processing large volumes of data. Infrastructure: Deployment on Linux machines with container management via Docker images to ensure optimal scalability and portability. Exposure: Processed data is accessible via Trino, facilitating exploration and use by stakeholders. 📈 Maintenance: Implementation of data flow monitoring and incident management to ensure a rapid response to anomalies. Continuous performance optimization to guarantee the reliability and scalability of the system.

As a Big Data Consultant, I was responsible for collecting, cleaning, and integrating data from diverse and heterogeneous sources within the public sector. This involved establishing automated data pipelines to ensure the availability and freshness of data. 🌟 Key Achievements: 🛠️ Developed data ingestion scripts for the data lake using HDFS and Hive, employing Scala and SparkSQL with Parquet format. 🔄 Transformed and normalized data using Scala and SparkSQL to ensure consistency and accuracy. ⚙️ Industrialized data processing by designing robust data pipelines with Scala. 🧹 Conducted comprehensive data cleansing, enhancing data quality through effective search and cleaning techniques. ✔️ Implemented unit tests to ensure script reliability and functionality. 🚀 Optimized the performance of distributed data processing scripts on Spark, improving efficiency and speed. 🔄 Managed continuous deployment of scripts using GitLab and Jenkins, streamlining the development workflow. 🐞 Resolved bugs and optimized scripts for better performance and maintainability. 📈 Automated workflows using Oozie, enhancing process efficiency. This experience honed my skills in big data technologies and solidified my commitment to delivering high-quality data solutions.

As a Big Data Consultant, I integrated into a team focused on the maintenance and enhancement of an information system operating on an Oracle data warehouse. This system faced challenges such as being full, slow, and error-prone, leading to lengthy recovery times. My key contributions included: 🛠️ System Optimization: Collaborated on optimizing the Oracle data warehouse, significantly reducing errors and improving overall performance. 🌐 Data Lake Architecture: Contributed to the implementation of a modern data lake architecture using technologies such as HDFS, NiFi (cluster mode), MapReduce, Hive, Sqoop, and YARN for efficient data processing and management. 📊 Monitoring Solutions: Established Ambari for comprehensive cluster monitoring and Grafana for real-time tracking of key performance indicators and incident management. ⚠️ Error and Incident Management: Developed processes for proactive error detection and incident resolution, ensuring minimal disruption to data operations. 🏆 Agile Methodologies: Worked in an Agile environment, applying Scrum methodologies to enhance team collaboration and project delivery. 📈 Data Visualization: Developed impactful dashboards in Power BI to visualize key metrics, providing stakeholders with actionable insights. 👩‍🏫 Team Leadership: Managed junior team members and mentored interns, fostering a collaborative and growth-oriented atmosphere. 🎓 Training for Young Professionals: Designed and delivered training sessions for young professionals, helping them acquire skills in Big Data and associated technologies. 🤝 Client Collaboration: Engaged with clients to assist in creating requirement specifications and charge sheets, ensuring project alignment with business needs.

Concept and implementation of web applications in C# based on ASP.NET. Design and realization of the DIT ERP (Workflow) based on BONITA Conception et réalisation du DIT ERP (Workflow) basé sur BONITA: Information system for personnel management.

Data analyst and development the mathematics models for the prediction for the case of the rainforest

Conception and realization of a web application for the supervision of the laying of the fiber optical and LAN, WAN

Design and complete realization of the website of the Polytechnic company Valor with the CMS SPIP and examination of the security of the web site. simulate different web attacks on the system to detect system vulnerabilities.