Data Engineer

Job Responsibilities

1. Data Warehouse Assessment and Planning
   Work closely with business teams to assess existing data assets, usage scenarios, and pain points. Prioritize and drive phased data warehouse initiatives to improve data availability and business data usage efficiency.
2. Data Warehouse Development and Optimization
   Participate in the overall architecture design and continuous optimization of the company’s data warehouse. Build layered data models, including ODS, DWD, DWS, and ADS, to support business analysis, decision-making, and data product needs.
3. Data Ingestion Development and Configuration
   Use tools such as Airbyte, Python, and Playwright to onboard, configure, monitor, and maintain stable ingestion pipelines for multi-source business data, including orders, users, products, and marketing data.
4. ETL Development and Data Modeling
   Develop data transformation logic using DBT and Python. Build reusable data models and data assets, and implement business rules for data cleansing, joins, aggregation, and metric definitions.
5. Data Synchronization and Workflow Orchestration
   Design and orchestrate data synchronization and processing workflows using Airflow, including DAG design, task dependencies, scheduling strategies, and exception handling, to ensure reliability and timeliness.
6. Data Quality and Governance
   Use metadata management tools such as OpenMetadata to establish data lineage, data quality monitoring, and alerting mechanisms. Quickly identify and resolve data anomalies to improve data trustworthiness.

Qualifications

Required

• Education: Bachelor’s degree or above in Computer Science, Mathematics, Statistics, Information Engineering, or a related field.
• Experience: 1+ years of experience in data development, data warehousing, or a related role.
• Programming: Proficient in Python and SQL, with the ability to independently develop data processing scripts and write and optimize complex SQL queries.
• Databases: Familiar with PostgreSQL or other mainstream relational databases, with an understanding of indexing, partitioning, and query performance optimization.
• ETL Tools: Hands-on experience with DBT, including model layering, incremental processing, testing, and documentation.
• Scheduling Tools: Experience with Airflow or similar workflow orchestration tools, including DAG design, task dependencies, and error handling.
• Data Integration: Familiar with the configuration, operation, and troubleshooting of Airbyte or similar data integration tools.
• Data Governance: Understanding of Open Meta data or similar metadata management tools, including data lineage and data quality monitoring.
• AI-Assisted Development: Proficient in using AI tools for development, such as AI coding assistants, code generation, and debugging tools, and able to apply them effectively to improve development efficiency.

Preferred

• Experience building a data warehouse from scratch.
• Familiarity with e-commerce or cross-border e-commerce data, including orders, products, users, and marketing.
• Practical experience with advanced DBT features, such as macros, packages, and snapshots.
• Hands-on experience implementing data governance projects.
• Deep experience with AI Agents or AI programming tools, such as using AI to develop complex ETL workflows or automate data processing pipelines.
• Practical experience using AI tools to improve the overall development efficiency of a data team.