Data Engineer

Context

Domain overview

In the team, the focus is on understanding customers deeply. The mission is to turn massive amounts of data into smart, actionable insights that make every customer's experience better and more personal. The team works to ensure this personalization feels seamless across the entire customer journey, whether on the app or visiting a store. This is not just about analytics; it’s about fundamentally changing how relationships are built. The organization is equipped with tools to move beyond "one-size-fits-all" interactions, with the clear goal of boosting conversions and building real, sustainable customer value (CLV).

To make this vision a reality, the team needs a Data Engineer to build the engine that powers it all. The main job is to build the rock-solid data foundations that algorithms and insights rely on. This involves managing pipelines and guaranteeing top-notch data quality and freshness, which makes personalization models smart. The Data Engineer will also be responsible for making these valuable data assets easy to access within the datalake, so marketing and analytics teams can grab insights and turn them into great, individualized experiences for customers.

Team & tech overview

• Technical stack: PySpark, Airflow, AWS, Databricks, Datadog

• Collaboration tools: Google Suite, Confluence, Jira, Github, Slack

Role overview

The company is seeking a talented Data Engineer to drive the technical excellence of the data infrastructure by developing and maintaining high-efficiency PySpark pipelines. The team is directly responsible for ensuring data remains fresh, accurate, and reliable for all stakeholders. The Data Engineer will collaborate with Product Managers and Data Scientists to continuously refine the data model, ensuring it scales effectively to handle growth and new use cases.

Missions

• Design and implement scalable data pipelines: architect, develop, and maintain high-volume, high-performance ETL/ELT pipelines using PySpark and Airflow to process vast amounts of customer data, ensuring data quality, reliability, and freshness.

• Ensure data quality and governance: establish and enforce rigorous data quality checks, monitoring, and validation procedures. Implement robust governance and compliance measures, particularly concerning GDPR and data privacy standards.

• Collaborate on data modeling: work closely with Data Scientists and Product Managers to continuously refine the core customer data model, optimizing it for both analytical insights and production-level algorithms.

• Manage and optimize cloud infrastructure: leverage expertise in Amazon AWS (e.g., EMR, S3) and Databricks to manage, monitor, and optimize the underlying data infrastructure for cost-efficiency and performance.

• Mentor and lead: act as a technical expert within the team, showing great autonomy and providing guidance on best practices in data engineering, code quality, and infrastructure maintenance.

Tools & Environment

• PySpark

• Apache Airflow

• Python

• Amazon AWS (EMR, S3)

• Databricks

• Datadog

• Great Expectations

• Delta Lake

• Collaboration tools: Google Suite, Confluence, Jira, Github, Slack

Working Conditions

• Location: Lille

• Expected starting date: June 29th

• Daily rate (TJM): 600€

1. Minimum 4 years of experience as a Data Engineer
2. Expertise in creating and implementing highly reliable and scalable data systems
3. Proficient in implementing data quality, governance, and compliance policies (GDPR) within data ecosystems
4. Mastered Airflow for workflow orchestration, ensuring reliable scheduling and monitoring
5. Utilize Python for developing robust and efficient ETL/ELT pipelines
6. Proficient in Spark (PySpark) for large-scale data processing, optimizations, and analytics
7. Proven experience with Amazon AWS, particularly services like EMR
8. Comfortable speaking and writing in English for daily collaboration with stakeholders
9. Experience implementing data quality tests with Great Expectations
10. Experience leveraging Datadog or similar monitoring tools for proactive pipeline health checks and performance optimization
11. Experience managing ACID transactions with Delta files