Data Engineer

About the role

Design and maintain the data infrastructure that supports AI agents across multiple domains, with a particular focus on high-quality, trustworthy data for semiconductor and internet scenarios. Build and optimize data pipelines for ingestion, cleaning, transformation, and feature preparation from structured and unstructured sources such as design documents, manufacturing logs, system telemetry, and application data.

Key responsibilities

• Design and maintain robust, secure, and scalable data platforms that serve AI/ML workloads across complex domains such as semiconductor design/manufacturing and internet-scale applications
• Build and optimize batch and streaming pipelines for ingesting, cleaning, transforming, and preparing features from diverse sources including manufacturing logs, system telemetry, application metrics, design documents, PDFs, engineering notes, and chat logs
• Implement data models, cataloging, lineage tracking, and access control to ensure data is discoverable, reliable, and compliant
• Implement data quality validation rules, anomaly detection, and monitoring to ensure fitness-for-use in AI contexts
• Design semantic data models and implement metadata management, data lineage tracking, and access control policies (RBAC/ABAC)
• Curate and index datasets for Retrieval-Augmented Generation (RAG) including chunking strategies, embedding pipelines, and vector database integration
• Work closely with AI Agent Engineers and MLOps Engineers to provide well-curated datasets and retrieval indices for RAG systems, support experimentation, and enable scalable analytics
• Translate AI requirements into data specifications and quickly understand domain-specific data semantics

About you

• Proven experience designing and maintaining robust, secure, and scalable data platforms that serve AI/ML workloads across complex domains such as semiconductor design/manufacturing and internet-scale applications
• Expertise in building and optimizing batch and streaming pipelines for ingesting, cleaning, transforming, and preparing features from diverse sources
• Proficiency with modern data stack tools such as Apache Spark, Airflow, Kafka, dbt, Fivetran, Snowflake, BigQuery, and Delta Lake
• Strong focus on data quality, consistency, and reliability with experience implementing data quality frameworks such as Great Expectations, Soda, and Monte Carlo
• Experience with data catalog tools such as Amundsen, Atlas, and DataHub to enable discoverability and compliance
• Experience curating and indexing datasets for Retrieval-Augmented Generation (RAG) with knowledge of vector database integration such as Pinecone, Weaviate, and Milvus
• Strong coding skills in Python and/or SQL with experience in version control (Git), testing, CI/CD, and infrastructure-as-code such as Terraform
• Ability to design semantic data models and implement metadata management, data lineage tracking, and access control policies