
Dat NGUYEN
Senior Data Engineer
1 recommendation
Salary / Daily rate
1 person recommend
92160 Antony, France
Freelance
Can come onsite
Skills
Languages
About me
Intro
Hi ! I'm a Data Engineer with more than 6 years of experience in modeling, implementing and maintaining batch and real time data pipelines with focus on resilience, low latency and high availability. I have a strong technical background combined with good communication skills. I'm result oriented with a good team spirit
Work experience
Data engineer
Urssaf Caisse nationaleFreelance
Apr 2024 - Dec 2024
9 months
Île-de-France, France · Remote
Sector : Fund recovery Context : Modernise existing data stack by building a Data Lakehouse on an OpenShift cluster and following Hop architecture Achievements : • Modeling, implementing and maintaining ingestion and transformation pipelines. Data Volume of many Tb. • Optimisation subjects regarding data ingestion using Nifi and Data Transformation using PySpark for a high volume of data (many Tb) • Schedule and deploy pipelines on an Openshift cluster using Airflow • Comparing 2 transformation tools (DBT-dremio and PySpark) to guide the final choice • Communication/clarification on the prototype use cases with the Product Owners and help design the solution Methodology : Agile Scrum, Sprint 2 weeks, rituals (retro, sprint planning, …) Technical environment : PySpark, Python, SQL, Airflow, Nifi, Bash, MinIO, Gitlab, Git, Openshift, Dremio, DBT, MinIO, Artifactory, Kibana, Grafana, Iceberg, Nessie
Senior Data Engineer
StellantisFreelance
Nov 2021 - Jan 2024
2 years 3 months
Paris et périphérie
Sector : Automobile et Finance Context : Build a centralised data lake on HDFS for financing data using a Hop architecture Achievements : • Modeling, implementing and maintaining ingestion, transformation and exposure data pipelines. Up to 1 Tb of data • Optimization of spark processes by DAG analysis, monitoring using Spark UI. Gain 30% in compute time • Ensure skills development and best practices of junior profiles on the processes and tools used • Setting up automatic deployments of the projects using TeamCity • Ensure that the entire PSA bank data lake is maintained in operational condition (monitoring, incident opening, incident resolving, client communication) • interface with the client, communicate on the progress of projects and potential blocking points Methodology: Agile Kanban and V cycle Technical environment : Java, Spark, Python, SQL, Hive, Bash, HDFS, TeamCity, Git, Maven, Artifactory, Kafka, AWS S3, Kibana
Data Engineer
Mappy
Aug 2021 - Oct 2021
3 months
Paris, France
Sector : GPS App Context : Meet the needs of marketing teams in terms of data restitution, build and maintainData and BI processing Achievements : • Building and automating ingestion pipelines of all app logs (~200 Tb of data daily) using scala spark and Airflow • Gather marketing team needs and translate them into kibana and tableau boards to allow them to analyse and take business decisions • Maintain in operational condition all processes of BI Team(monitoring, Process restart, Data cleanup, ...) Methodology: Agile Scrum, Sprint 2 semaines, rituels (retro, demo, …) Technical environment : Scala, Spark, Python, SQL, Hive, Airflow, Git, Kibana, Tableau, Jenkins
Software Engineer
STET
Dec 2018 - Aug 2021
2 years 9 months
La Défense, France
Sector : Payment Context : Build and maintain the Instant Payment solution Achievements : • Build and maintain real time programs for the processing and control of interbank payment flows • Time optimization of the unit test battery of the Instant Payment project with a gain of 50% using Docker and Jenkins • Provisioning and deploy packages on AWS for integration tests using EC2 instances and EFS storage • Using Kafka Windowing features to set up an alerting mechanism for Instant Payment service Methodology: Agile Scrum et Kanban, Sprint 2 à 3 semaines Technical environment : Scala, Spark, Bash, SQL, Cassandra, Kafka, kafka streams, Spark Streaming, AWS, Maven, Docker
Software Development Engineer
BNP Paribas
Nov 2017 - Dec 2018
1 year 2 months
Région de Montreuil, France
Sector : Credit risk Context : Build and maintain a risk credit application (DIRECT) part of ITG SIT fin9 branch Achievements : • Integrate a new risk flow in the existing application using spring boot • Maintain business flows in Oracle databases using PL/SQL • Training of a Third Party Application Maintenance (TMA) in the maintenance and potential evolution of the application Methodology: V cycle Technical environment : Java, Spring Boot, PL/SQL, Oracle, Jenkins, Git
Ingénieur de production, développeur logiciel
CAPITALDATA › People-based Marketing solution
Feb 2017 - Aug 2017
7 months
Région de Paris, France
Sector : Marketing digital Context : Maintain and build a bid application for web publicity spots and technical management of digital marketing campaign Achievements : • Development and optimisation of internal software for publicity campaigns (automated targeted mails, bid software, …) • Monitoring of production processes • Prepare and manage web marketing campaigns Methodology: Agile scrum Technical environment : Golang, MySQL, MongoDB, Redis, Git, JSON
Education
Pierre and Marie Curie University
Master 2 (M2)
2015 - 2018
3 years 2 months
Pierre and Marie Curie University
Licence
2011 - 2014
3 years 2 months
Licenses & certifications
Databricks Certified Data Engineer Associate
Issued: Dec 2024
Big Data Analysis with Scala and Spark
Issued: Dec 2020