Senior Data Engineer

Job Title: Senior Data Engineer

Location: Mountain View, CA (Hybrid)

Experience: 8+ years

Need local to CA

Core skillset – Python, Lang graph, Lang chain, Lang fuse, Databricks, Semantic Data & Agentic AI

Role Overview

We are looking for a strong Data Engineer to build reliable data pipelines and business-aware data foundations for AI agents using Databricks.

The ideal candidate combines hands-on data engineering with knowledge of semantic modeling, knowledge graphs, contextual graphs, and retrieval-augmented generation (RAG). Experience with LangChain, LangGraph, and Langfuse will help connect enterprise data to agent workflows and monitor their performance.

The role focuses on using existing LLMs and enterprise AI platforms.

Key Responsibilities

• Build and maintain batch and streaming pipelines using Python, SQL, PySpark, and Delta Lake.
• Use Databricks Lakeflow for ingestion, transformation, and workflow orchestration.
• Create semantic models defining common business entities, relationships, terminology, and metrics.
• Build knowledge graphs connecting information across enterprise systems.
• Develop contextual graphs capturing events, history, and workflow state to provide relevant context to AI agents.
• Prepare documents and structured data for RAG, semantic search, and graph-based retrieval.
• Use LangChain to connect retrieval pipelines, enterprise data sources, and tools to LLM applications.
• Use LangGraph to support stateful, multi-step agent workflows, including tool use and human review.
• Use Langfuse for tracing, prompt management, evaluation, and monitoring agent interactions.
• Apply Unity Catalog for data access, metadata, lineage, and governance.
• Monitor data quality, freshness, retrieval accuracy, latency, and cost.

Required Skills

• Strong production data engineering experience with Python, SQL, Spark/PySpark, and Databricks.
• Hands-on experience with Delta Lake, ETL/ELT pipelines, incremental processing, and data modeling.
• Practical experience with semantic modeling and knowledge graphs, including entity resolution and relationship mapping.
• Understanding of contextual graphs and how business history and workflow state support agent decisions.
• Experience with RAG, embeddings, vector search, and integrating enterprise data with AI applications.
• Hands-on experience with LangChain and LangGraph for retrieval, tool integration, and agent workflows.
• Familiarity with Langfuse for tracing, evaluation, and observability.
• Familiarity with graph technologies such as Neo4j, Amazon Neptune, or RDF-based platforms.
• Strong understanding of data quality, security, governance, and production support.

Relevant Technology Stack

• Data engineering: Python, SQL, PySpark, Delta Lake, and Databricks Lakeflow.
• Governance and semantics: Unity Catalog and Metric Views, where applicable.
• Retrieval: Databricks AI Search / Vector Search, embeddings, and graph databases.
• Agent development: LangChain, LangGraph, and Databricks Agent Bricks / Knowledge Assistant.
• Evaluation and observability: Langfuse and MLflow.

Strong Databricks data engineering skills are essential. Candidates should have practical agent integration experience, with depth in the tools relevant to the project.

Preferred Experience

• Combining graph retrieval and vector search.
• Integrating Databricks with graph databases.
• Building reusable data APIs and tools for AI agents.
• Supporting enterprise agent platforms with evaluation, monitoring, and human review.