Raphael Auv

Feb 2020 - Present - Contributed to the Apache Airflow project by developing features, fixing bugs, and providing community support through code reviews and documentation - Training ( formations ) on Apache Airflow / Apache Spark / Apache Kafka / Trino / Apache Iceberg …

Drove the transition to a Data Mesh architecture, facilitating the migration of engineering teams to a new platform and event bus system AWS - S3 / EKS / GLUE / ATHENA / MSK Apache Kafka, Apache Kafka connect, Debezium, Strimzi, Apache Airflow, DBT, Apache Flink, Apache Iceberg, FastAPI, MLflow, Ray, Pandas / Polars, Kubernetes, Terraform, Grafana

Assumed Lead Data Engineer responsibilities during the incumbent's leave , ensured operational continuity and supervised a team of 7 (5 DEs, 2 Data Ops) GCP - GCS - BIGQUERY - GKE DATABRICKS AWS - ATHENA - REDSHIFT - S3 - EKS Apache Spark, Apache Kafka , Apache Kafka Stream, Apache Kafka Connect ,Apache Airflow, Apache Iceberg, DBT, Kuberbetes, Datadog, Github Actions

Entirely design and put in place the first data-stack : datalake (iceberg) and data/ML-pipelines (airflow) on the on-premise data center of INA. ON PREMISE Apache Parquet, Apache Iceberg, Tabular_rest_catalog, Apache Arrow, Polars, Trino, Apache Airflow, Apache Spark, DBT, FastAPI, object storage(S3), Postgres, Redis-cluster, ElasticSearch, Docker, Docker-compose, K8S_KIND, Kubernetes, Helm, OpenTelemetry, Grafana

Technical owner of the data-platform and all data-jobs, helping dataops , data-eng , data-analyst , data-scientist to setup and deploy new query/jobs/pipelines/api AWS : EKS, REDSHIFT, GLUE, SQS, S3, RDS, ATHENA, MSK, IAM OIDC Apache Airflow, FastAPI, Apache Spark (pyspark), Docker, Docker compose, Kubernetes, HELM, Terraform, , github actions

Put in place DEVOPS methodology and software engineering best practices for ML pipelines -> MLOPS Kubernetes - Helm - KIND - Docker - Docker compose - MLflow - Apache Airflow - github actions AWS - SCALEWAY

My mission : I've fully built from scratch the new data platform of ankorstore : the local dev stack , the preprod and production environnements, with full CI/CD so every commit can be deployed automatically in less than 2 minutes. Where data-scientist & developer (data engineer , analytic engineer ) can deploy in full autonomy new jobs ( contenerized ) and batch pipelines ( airflow dags ) and stream pipelines ( Kafka connect - CDC - DEBEZIUM ) to production. Technical : local dev stack : Docker-Compose , Kubernetes-KIND ( airflow , redpanda ... ) prod stack : Helm , Kubernetes , Pulumi monitoring : OTEL ( metrics , trace , logs ) CI / CD : github actions deployed : Apache Airflow , fastAPI ( tracing with opentelemetry ) , pyspark ( spark ) , kstream (apache kafka stream) , confluent-schema-registry , aiven-rest-proxy ( karapace ), kafka-connect (debezium , bq-sink ), akhq , provectus-kafka-ui, ksqldb , lightdash , opentelemetry-collector , mlflow , rocksdb , dbt serialization formats : Apache Avro , Apache Parquet GCP : GCS , COMPOSER , BIGQUERY , GKE AIVEN : KAFKA , CLICKHOUSE

Design and architect the data-mesh of the company ( data exchange politics and patterns between services ) Built the event bus and the event data platform with Kafka Also support the internal data-eng team in design and best practices on batch pipelines (airflow) Airflow , Kafka , confluent schema-registry , Kafka-connect , Debezium , confluent-S3-sink , Docker , Docker-compose , Kubernetes , AWS , Datadog , Snowflake , Apache Avro , Apache Parquet java - python

Working on data pipeline and ML forecast predictions databricks Delta lake , apache Spark (pyspark) , Docker , Docker-Compose , Azure Azure Data Factory Azure Databricks Azure Data Lake Storage Gen2 Azure DevOps java - python

My role was : Data / Machine Learning / Backend & Ops engineer of the Recommender feature team (in charge of the Recommender system of Leboncoin) following DevOps principle --- What I did : Data, ML Pipelines and API : Training ,Serving, Monitoring of models and API's --- Stack : Airflow - Kubeflow - FastAPI - Gensim - Spark - Redis - Kafka - Docker - Kubernetes - Datadog - Flask - Gunicorn - Apache Avro - Apache Parquet