Data Engineer – Treasure Data & Python Specialist

Experience: 7+ Years

Role: Data Engineer

Specialization: Treasure Data, Python/PySpark & Advanced SQL

Role Overview

We are looking for an experienced Data Engineer with 7+ years of experience in building robust, scalable, and optimized ETL pipelines, managing Customer Data Platform (CDP) infrastructure, and driving automated data workflows and processing.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines for large-scale data processing.
• Work extensively with Treasure Data CDP, including Treasure Data workflows and data processing environments.
• Develop distributed data processing solutions using Python and PySpark for large-scale data manipulation and batch transformations.
• Write and optimize complex SQL queries, including window functions, joins, aggregations, and data transformation logic.
• Perform data cleansing, schema design, query optimization, and performance tuning across complex database environments.
• Build and maintain automated data workflows to improve reliability, scalability, and processing efficiency.
• Work with Trino/Hive and SQL within the Treasure Data ecosystem.
• Collaborate with downstream business analytics teams to understand requirements and bridge technical/data infrastructure gaps.
• Participate actively in Agile/Scrum ceremonies, including stand-ups, sprint planning, retrospectives, and continuous delivery activities.
• Track development activities, requirements, and issues using tools such as Jira and Confluence.
• Adapt quickly to changing sprint priorities, technical constraints, and evolving data governance requirements.
• Maintain clear and comprehensive documentation covering ETL logic, data flows, pipeline architecture, and technical processes.
• Ensure adequate documentation and knowledge sharing to minimize pipeline knowledge gaps and dependency on individual team members.

Mandatory Technical Skills

• Treasure Data / CDP – 7+ years preferred
• Python / PySpark – Strong hands-on experience
• Trino / Hive
• Advanced SQL
• SQL query optimization and performance tuning
• Window functions and complex query structuring
• Data cleansing and schema architecture
• ETL/ELT pipeline development
• Large-scale batch data processing
• Data workflow automation

Agile & Collaboration

• Strong experience working in Agile/Scrum environments
• Hands-on experience with Jira and/or Confluence
• Strong cross-functional collaboration with analytics and business teams
• Ability to adapt to changing priorities and technical requirements
• Strong technical documentation and knowledge-sharing discipline

Ideal Candidate

A hands-on Data Engineer with strong Treasure Data, Python/PySpark, Trino/Hive, and advanced SQL expertise, capable of building scalable data pipelines while effectively collaborating with analytics and business teams

Skills: python,etl,data,processing,cdp,tresure data,sql,data processing