Data Pipeline Engineer

A globally leading consumer device company headquartered in Sunnyvale, CA is looking for a Data Pipeline Engineer to join their dynamic team.

Years of Experience

• 8 years of software development in data modeling, database design & optimization, scalable data pipelines for image/video (multimedia) data, and full-stack development using Python and SQL
• 5 years of development on data warehouse and analytics solutions (e.g., Airflow, Trino, Spark, Snowflake)
• 5 years of development on cloud-based data processing pipelines
• Demonstrated software development skills using Python
• Preferred: Agentic AI development experience

Responsibilities

• Unified Metadata & Schema Design: Design and build standardized schema that unify file indexing, metadata, and enriched metadata across heterogeneous storage platforms (e.g., a document-store-based file platform, a relational per-protocol file platform, and other data collection platforms) and multiple dataset with differing file/metadata structures.
• Data Pipeline Engineering for Multimedia Data: Build and maintain efficient, scalable, monitored data pipelines that process image and video data at scale within a unified data platform, and support downstream consumer dataset generation.
• Database Design & Optimization: Create and maintain database schemas across SQL, vector, and unstructured data stores; apply normalization and denormalization appropriately; tune queries and storage for performance, reliability, and query efficiency at scale, addressing the known challenge of indexing/querying file- and frame-level metadata beyond classic DB solutions.
• Dashboards & Data Quality Monitoring: Develop dashboards to track data ingestion progress, surface metadata gaps, detect invalid/inconsistent metadata, and fix data issues early.
• Federated Ecosystem Enablement: Support the broader goal of a federated data ecosystem where independently contributed tables are connected, discoverable, and queryable through one consistent data schema interface.
• Full-Stack Development: Build front-end and back-end components of data-driven applications/dashboards, develop APIs, and integrate data from multiple sources/technologies into a single consumer-facing interface.
• Data Security & Compliance: Ensure data handling meets security and privacy compliance requirements across platforms and studies.
• Agentic AI Prototyping (stretch/preferred): Prototype and develop agentic AI capabilities to assist with data analysis and processing workflows on top of the unified data interface.

Preferred Skills

• Proficiency with multimedia processing tools (FFmpeg, GStreamer, OpenCV) and familiarity with AI/ML frameworks (TensorFlow, PyTorch) for video analysis
• Experience with Spark, Airflow, Trino, Snowflake, Iceberg-based lakehouse/data platforms
• Experience designing schema layers that separate interface from implementation to allow backward-compatible schema evolution
• Experience with config-driven/declarative pipeline and schema design
• Agentic AI development experience

Education

Bachelor's or Master's degree (or above) in Computer Science, Software Engineering, or a related field

Position Details :

Type: Contract (W2)

Duration: 06 months with extension

Work Location: Sunnyvale, CA (onsite)

Pay range: $85.00 - $100/hr (DOE)