Observability Pipeline Engineer (Fluent Bit / OpenTelemetry)

Observability Pipeline Engineer (Fluent Bit / OpenTelemetry)

Location: Remote (US) | Type: Contract (3+ months) | Experience: 5+ years | Pay: Up to $70/hr

About the Role

We're hiring an observability engineer to own the data highway between a large fleet of servers, applications, and clusters and the OpenSearch environment everything lands in. You'll build and run the log and metrics ingestion pipelines with Fluent Bit and the OpenTelemetry Collector, tune the aggregation tier so it holds up when volume spikes, and keep data flowing into OpenSearch on AWS without drops.

The client is a managed cloud services firm that designs and operates AWS environments for public sector organizations, where uptime, auditability, and access control aren't negotiable. The telemetry volume is real, the environments are locked down, and the pipeline is treated as production infrastructure rather than a side system.

This is not a generic DevOps or SRE seat. Plenty of engineers have used Datadog, Grafana, and Prometheus. Far fewer have actually engineered the collection and ingestion layer at scale, sized buffers, chased backpressure at 2am, and figured out where the missing 4% of logs went. That second group is who this role is for.

What You'll Do

• Design, deploy, and operate log and metrics ingestion pipelines built on Fluent Bit and the OpenTelemetry Collector, from source config through parsing, filtering, enrichment, and routing
• Build and scale the aggregation tier so it adds capacity when traffic jumps 10x and scales back down when it doesn't
• Get high volumes of telemetry into OpenSearch efficiently, using OpenSearch Ingestion (OSIS) and Data Prepper where they fit
• Tune buffering, queueing, retry, and backpressure behavior so a downstream slowdown doesn't turn into data loss
• Troubleshoot the ugly production cases: collectors running out of memory, indexing latency climbing, records silently dropping, pipelines backing up under load
• Wire up authentication and transport security across the pipeline with IAM, SigV4, TLS/mTLS, and fine-grained access control in OpenSearch
• Run the pipeline components on Kubernetes/EKS, including deployment patterns, resource sizing, and autoscaling
• Document the pipeline architecture and hand off working runbooks so the platform team can operate it without you in the room

Qualifications

Core Pipeline Engineering

• 5+ years in platform, observability, or infrastructure engineering, with meaningful time spent on telemetry pipelines specifically
• Production experience configuring Fluent Bit (inputs, parsers, filters, outputs, buffering) at scale, not just proof of concept work
• Hands-on experience with the OpenTelemetry Collector, including receivers, processors, exporters, and running a gateway or aggregation deployment
• Demonstrated experience shipping high-volume logs and metrics into OpenSearch or Elasticsearch, including index and ingest tuning
• Track record diagnosing data loss, backpressure, buffer saturation, and throughput bottlenecks in live systems

AWS and Platform

• Strong AWS background across the services telemetry pipelines actually touch
• Experience with Kubernetes and EKS, including autoscaling the collector layer
• Working knowledge of IAM, SigV4 request signing, TLS/mTLS, and fine-grained access control
• Infrastructure as code and CI/CD habits for pipeline configuration changes

Nice to Have

• OpenSearch Ingestion (OSIS) and Data Prepper in production
• Public sector or otherwise compliance-heavy environments
• Experience standing up observability for teams migrating off a commercial APM vendor

Why Join Us

You'll be working with a cloud team that treats observability as engineered infrastructure with owners, budgets, and SLAs, not as a checkbox someone turned on. The problems are concrete, the feedback loop is fast, and there's no ambiguity about whether the work is landing, because you can watch the throughput.

If you like the part of the job where you find out exactly why the records stopped arriving, this one will keep you interested.