Data Engineer

Responsibilities:

• Set up, operate, and maintain a scalable in-house data platform (Lakehouse/Data Mesh) that supports both high-volume batch processing and real-time streaming.
• Identify and deploy appropriate open-source tools to enhance data management and analytics capabilities.
• Develop robust data pipelines to harmonize data from multiple disparate systems, performing complex transformations, deduplication, and entity resolution to create a "single source of truth."
• Collaborate with the AI team to set up data pipelines specifically for AI applications and LLMOps.
• Ensure data is structured to support generative AI use cases.
• Implement and maintain data governance frameworks to ensure compliance with relevant regulatory requirements.

Requirements:

• 3+ years of hands-on experience developing and deploying solutions with modern data technologies, including but not limited to Data Warehousing (Snowflake, BigQuery, Redshift), NoSQL (DynamoDB, MongoDB, Elasticsearch), Data Lakes (Databricks, Delta Lake), or Customer Data Platform
• Proficiency in Python, SQL, and Java
• Experience with cloud data services