Senior Site Reliability Engineer

Who you are

• We're looking for engineers who want to solve problems at scale, own critical systems end to end, and help shape the future of infrastructure for AI and beyond
• 5+ years of experience operating Linux systems in production or HPC environments, with hands-on storage experience at scale on scale-out or software-defined platforms (e.g., CEPH, Lustre, GPFS, or similar)
• Hands-on experience operating Software-Defined Storage (SDS) platforms at scale, including integrating with their management and data-plane APIs
• Strong incident-response instincts: comfortable owning a production storage incident end to end, from first alert through root cause to postmortem
• Working experience with monitoring and logging platforms such as Prometheus, Grafana, Alertmanager, Datadog, or SumoLogic — including building dashboards and alert/pager routing for multiple audiences
• Working experience with Kubernetes (GitOps tooling such as ArgoCD, Helm/Kustomize) and hands-on troubleshooting
• Working experience with CI/CD tooling (GitHub Actions, Jenkins, BuildKite), containerization (Docker/Podman), and systems programming in Python or Go
• Working experience with Infrastructure as Code (Terraform, Ansible)
• Solid understanding of core storage protocols across file (NFS, SMB), object (S3), block (NVMe-oF/TCP), and structured (vector DB, SQL) storage
• Experience with cutting-edge software-defined storage solutions such as VAST or Weka
• Enterprise storage expertise in solutions such as NetApp, Dell PowerScale, GPFS, or Lustre
• Experience writing or operating Kubernetes CSI drivers
• Experience with SR-IOV and virtualization (KVM/QEMU)
• Experience with GPUDirect Storage, RDMA, InfiniBand, or RoCE networking
• Familiarity with NIC-level diagnostics (ethtool, mlxlink) and fleet-wide operations tooling (clush or similar)
• Contributions to open-source storage projects

What the job involves

• Lambda's Storage Engineering team is the backbone behind our world-class storage offerings, operating at a scale seldom seen in today's markets
• We own the full spectrum of Lambda's data platform services—from low-level storage systems to the APIs and tooling our customers build on every day
• Our work directly powers some of the most demanding compute workloads in the industry, which means reliability and performance aren't just goals—they're the baseline
• Own the reliability, performance, and capacity health of Lambda's production storage fleet across all data centers, operating behind Lambda's own software-defined data plane
• Build and maintain monitoring, dashboards, and alerting for storage performance, capacity, and hardware failures
• Investigate and resolve storage-related incidents using deep telemetry, logs, and performance profiling — from a single flapping NIC to a cluster-wide rebuild
• Automate ticketing, escalation, and incident-response workflows so the team spends less time on repetitive triage and more time on root cause
• Design and maintain self-healing automation for common failure modes: drive replacement, node swaps, rebuild monitoring, and capacity rebalancing
• Implement CI/CD pipelines for storage automation and tooling
• Partner with Storage Engineers, Fleet Orchestration, and Release Engineering to automate the deployment and configuration of software-defined storage across new and existing sites using tools such as Ansible, Jenkins etc
• Work with hardware and networking teams to diagnose low-level I/O and network issues — NIC errors, RDMA/RoCE/InfiniBand fabric health, path multipathing — that surface as storage-layer symptoms
• Participate in an on-call rotation supporting Lambda's storage fleet, with a focus on driving down MTTR and building the automation that keeps you from getting paged for the same thing twice

Benefits

• Health, dental vision
• 401k match
• 5 sick days and 12 paid holidays
• Flexible PTO
• Paid parental, medical, and caregiver leave