C
CoreWeave
Senior Software Engineer (AI Infrastructure Performance Insights & Observability)
Skills
LookerData EngineeringPythonGolangPyTorchGrafanaMachine LearningAIPrometheusKubernetes
2 months ago
Share this opportunity
Share this opportunity to other talents of your network:
✓ Offer them a visibility boost with clients.
✓ Help your contacts find their next job.
Important information
Contract type:
Permanent
Salary:
$182,000 – $242,000 / year
Location:
Sunnyvale, CA, USA
Work mode:
Onsite
Published on:
21 August 2026
What they need
- We’re looking for a Senior Engineer to be a driving force on CoreWeave’s Benchmarking & Performance team, with a focus on building the performance insights and observability systems that make our AI infrastructure legible at every layer, from individual GPUs and NVLink/InfiniBand fabrics up through distributed training and inference workloads
- You will own how we detect, diagnose, and surface performance signals across every data center in our global infrastructure, turning billions of raw telemetry events into real-time insight that engineers, product teams, and executives can act on with confidence
- This is not a straightforward data engineering or BI role. The role will focus on building the observability and insight tooling that lets us answer, in near real time, whether a GPU fleet, a fabric, or a training run is performing the way it should, and why it isn’t when it’s not
- If you’re energized by building the systems that turn raw infrastructure telemetry into trusted, actionable performance intelligence, and you want that work to sit closer to the hardware and the workload than to a dashboard, this role was built for you
- Performance Insights & Observability - Design and build the systems that continuously assess AI infrastructure health and performance: GPU utilization and efficiency, interconnect (NVLink, InfiniBand, RoCE) fabric behavior, distributed training and inference throughput, and hardware degradation signals. Build the detection and diagnosis logic that surfaces anomalies and regressions before they become incidents, not just dashboards that report on them after the fact
- Time-Series & Metrics Infrastructure - Own and extend our time-series database (TSDB) layer as the backbone of real-time observability. Write and optimize PromQL/MetricsQL queries that power alerting, anomaly detection, and trend analysis across thousands of GPUs and hundreds of benchmark runs. Bridge streaming metrics and batch-analytical workloads so engineers get sub-second answers during live incidents and analysts get complete historical context for root cause work
- Fabric & GPU Telemetry - Build and validate the pipelines and metrics that make network fabric and GPU-level behavior observable and comparable across racks, clusters, and hardware generations, including gray failure detection, congestion and error-rate signals, and health scoring that holds up under audit
- Data Lake Architecture (in support of insight work) - Design and build the performance data lake that underpins the above: table formats (Apache Iceberg, Parquet, Avro), hot/cold tiering, and schema evolution for latency distributions, throughput metrics, GPU utilization, cost-per-token, and hardware health signals. This is foundational infrastructure, not the end product
- Query Optimization & Performance - Profile and tune query engines against columnar and time-series stores so that the observability layer meets its own strict P99 latency and freshness SLAs. Benchmark the benchmarking infrastructure itself
- BI & Reporting (secondary) - Where needed, build self-service views (Grafana, Looker, or similar) for engineers, product managers, and executives, but as a downstream output of the insight and observability work above, not the primary deliverable- Strong coding in Python or Go (C++ a plus) and deep familiarity with networked systems, GPU infrastructure, and performance analysis
- Strong communicator comfortable collaborating with cross-functional teams and external partners
- Familiarity with data lake architectures and modern table formats (Iceberg, Parquet, Avro) sufficient to support an insights platform, though this is not the primary skill this role is hiring for
- 5+ years of experience building distributed systems, observability platforms, or performance engineering tooling, ideally for infrastructure or ML systems rather than general-purpose BI
- Hands-on experience with Kubernetes at production scale, CI/CD, and observability stacks (Prometheus, Grafana, OpenTelemetry) used to monitor and diagnose infrastructure, not just report on it
- Comfortable working close to hardware and workload behavior: GPU utilization patterns, interconnect fabric health, distributed training/inference performance characteristics
- Working knowledge of time-series databases and fluency in PromQL or MetricsQL for building real-time alerting and anomaly detection, not only historical dashboards
- Experience with time-series databases, LSM-based storage engines, or custom telemetry pipelines
- Experience running MLPerf submissions or similar large-scale audited benchmarks
- Contributions to OSS projects such as Apache Iceberg, Apache Spark, Trino, llm-d, vLLM, or PyTorch
- Direct experience benchmarking or monitoring large GPU fleets or multi-region clusters
- Experience with CUDA kernels, NCCL/SHARP, RDMA/NUMA, or GPU interconnect topologies
- Familiarity with data cataloging, lineage tools, or data governance frameworks
Other offers great for you!
These companies are also looking for great profiles
N
NGC x OPTIMA
Ingénieur Logiciel embarqué
Freelance
Cergy, France
Hybrid, Onsite
Skills
AUTOSARBSW
1 month ago
Quick apply
S
Salutech
Ingénieur Logiciel IA
Freelance
Paris, France
Onsite, Hybrid
Skills
DockerPythonTesting & QAKubernetesSoftware refactoringCI/CDObservabilityAsynchronous programming
10 days ago
Quick apply
T
Tenth Revolution Group
Ingénieur Logiciel Fullstack Senior (évolution Squad Lead) – Systèmes distribués
Freelance
Paris, France
Hybrid
Skills
TypescriptGolangJavaDevOpsTesting & QAAIDistributed systemsDistributed ArchitectureReplicated databasesalgorithmique
3 days ago
Quick apply
No company information available.