Senior Software Engineer (AI Infrastructure)

• The AI Infrastructure team within Snorkel owns the platform layer that powers everything at Snorkel — and that layer is evolving
• Beyond the data platform (pipelines, access layers, event systems, governance, compute), we are now building Snorkel’s AI infrastructure: the foundational agentic stack that will let every team at Snorkel build, run, and govern AI agents, and the LLM efficiency layer that keeps AI costs under control as usage scales
• We are a small team with a large surface area, in the middle of two foundational shifts: moving from a single-database data path to a multi-source, event-driven platform (Postgres/RDS, Snowflake, S3, metrics platform), and moving from bespoke, one-off agent implementations to a shared, governed, agentic-first platform
• The decisions being made now will define how data and agents operate at Snorkel for years
• You will be making them
• You’ll also shape our AI-native development workflow, contribute to modernizing CI/CD (Buildkite, GitHub Actions), and integrate AI SRE tooling
• Your work will directly accelerate developer velocity, reliability, and product quality across the company
• Build the Agentic Factory Foundation
• Design and build the opinionated agentic stack that FDEs, delivery, and product engineering teams will use to scaffold agent workflows: a common orchestration layer for defining and running agents, a memory layer (short-term working memory and long-term persistence), a context graph / knowledge layer grounding agents in project, spec, and platform state, an MCP gateway providing secured, governed, auditable tool access, an evaluation layer testing agents against trace-level and outcome-level criteria, and an observability layer for traces, feedback, metrics, and cost
• Start pragmatic — leverage existing building blocks to ship real use cases (self-healing agents, spec-to-eval pipelines, debugging agents) before going deep on every layer
• Build the LLM cost and efficiency platform
• LLM token spend is growing with the business, and controlling it is a first-class engineering problem
• Build the queuing and throttling layer that governs synchronous LLM requests, async and batched call paths for workloads that don’t need real-time responses, token optimization (prompt compression, caching, model routing), token usage metering and attribution so teams can see what they spend and why, and world-model approaches that let agents reuse knowledge instead of re-querying models
• Build the foundational data access layer and SDKs
• Design the shared access library that Platform, Packaging, and Dataset API teams use to read from and write to multiple data sources (Snowflake, S3, RDS) — abstracting entity data from the specific infrastructure underneath so we can scale and improve infrastructure without every product team absorbing the change
• Interfaces provide built-in auth, RBAC enforcement, pagination, and query governance
• Design and implement event-driven data flows using event brokers, CDC connectors, schema registry, event routing, and dead letter queues
• Make sure events flow reliably and failures are visible and recoverable
• Build governance, lineage, and audit infrastructure — for data and for agents
• Track how data moves through the platform, enforce who (and which agent) can access what, and log what happened
• This includes PII handling, retention policy enforcement, and audit infrastructure for enterprise and federal compliance, extended to agent actions and tool calls through the MCP gateway
• Own reliability and cost
• Instrument the platform with OpenTelemetry, define and monitor SLOs for query latency, pipeline success rates, and agent workflow health, and build alerting that catches issues before they become incidents
• Contribute to cost visibility and optimization across both infrastructure (query cost estimation, workload right-sizing, storage tiering) and AI spend (token cost attribution, model routing)
• You will be on-call for the systems you build

Benefits

• Health: Snorkelers and their dependents are covered by comprehensive medical, dental, and vision plans.
• Wellness: All Snorkelers get a yearly wellness stipend to use on anything related to health and well-being.
• PTO and rest days: We provide unlimited personal time off so you can find your work-life balance. Plus, two company-wide rest days per quarter in addition to company holidays.
• One global team: From our headquarters in Redwood City, CA, to regional offices in San Francisco and New York, we work together as one global team.
• Events: Get to know your fellow Snorkelers outside of work! We host regular events and learning opportunities for our teams across our offices and virtually to connect, have fun, and grow.
• Family leave: Snorkel offers generous parental leave for both birthing and non-birthing parents based on eligibility.- Hands-on experience building with LLMs in production — working with LLM APIs, and reasoning about tokens, context windows, rate limits, batching, and caching. You understand why an async batched call costs less than a sync one and can design systems around that
• Experience with AWS — S3, RDS, EKS, EventBridge, IAM. Comfortable working in a Terraform-managed environment
• Fluency with AI-assisted development tools (Claude Code, Cursor, or similar). This is a hard requirement — the team uses these tools daily and we expect engineers to leverage them for code generation, debugging, and investigation
• Strong proficiency in Python. Our stack is Python-heavy across Prefect, FastAPI, dbt, and the SDK layer
• Hands-on experience with SQL and at least two of: Snowflake, Redshift, Postgres. You understand the performance characteristics of each and can write queries that don’t bring down production
• 4+ years building platform infrastructure, data infrastructure, or backend systems with significant data components. You have built and operated pipelines, data access layers, or production services other teams depend on
• Familiarity with data orchestration tools (Prefect, Airflow, or Dagster) and transformation frameworks (dbt)
• Understanding of data governance concepts — RBAC, PII handling, audit logging, data lineage — and interest in extending them to agent and tool-call governance
• Experience with Kubernetes. Our workloads run on EKS and you will deploy, debug, and scale services on K8s
• Experience building agentic systems or infrastructure — agent orchestration frameworks, MCP servers/gateways, agent memory or knowledge layers, or agent evaluation harnesses
• Experience with LLM serving/gateway infrastructure — request queuing, rate limiting, model routing, semantic caching, or token cost optimization at scale
• Experience building shared libraries or SDKs consumed by multiple teams — versioning, backwards compatibility, migration support
• Experience with OpenTelemetry, ClickHouse, or similar observability infrastructure, including LLM/agent trace observability
• Experience with event-driven architectures — CDC, event buses, schema registries, at-least-once delivery semantics
• Prior work in regulated environments (SOC 2, FedRAMP, HIPAA) where compliance requirements shaped system design
• Experience with Ray for distributed compute workloads