Senior Data Engineer

Senior Data Engineer

Location: Remote (US) | Type: Contract | Experience: 7+ years | Pay: Up to $80/hr

About the Role

This is a streaming specialist role, not a general ETL role. You'll build and run the real-time data architecture at a cloud services and data consultancy: Kafka and AWS MSK at the center, Kinesis alongside, feeding analysts, data scientists, and operational systems that can't wait for a nightly batch.

You'll own the clusters, not just the consumers. Topic and partition design, schema evolution, throughput and latency tuning, retention, and the failure paths when a broker drops or a consumer group falls behind.

If you've been writing producers against a cluster somebody else runs and you want the platform to be yours, that's the shift this role represents.

What You'll Do

• Design and maintain real-time pipelines on Kafka and MSK: topic design, partitioning strategy, schema management, producers and consumers
• Tune clusters for throughput, latency, and fault tolerance, and manage broker scaling, replication, and retention
• Configure Kafka security (SSL/TLS, SASL, ACLs) and run Schema Registry for serialization and schema evolution
• Build Kafka Connect pipelines for streaming integration
• Implement Kinesis Data Streams and Firehose, configure Kinesis Analytics, and manage shard scaling
• Write stream processing applications in Spark Streaming, Kafka Streams, or Lambda, including windowing, real-time aggregation, and stateful processing
• Wire streaming into the batch and analytics side (Spark, Redshift, S3) so real-time and historical reporting agree with each other
• Implement lineage, automated data quality checks, and validation on data in motion
• Own the streaming platform in Terraform or CloudFormation, automate deploys through CI/CD, and instrument health, metrics, and alerting
• Mentor junior data engineers and explain streaming behavior to people who don't think in partitions

Qualifications

Required Experience

• 7+ years in data engineering with real streaming specialization
• Bachelor's degree in Computer Science, Engineering, or a related STEM field
• Production Apache Kafka: cluster management, performance tuning, ecosystem tools
• AWS MSK and Amazon Kinesis in production
• AWS Data Engineer Associate certification

Streaming and Processing

• Topic and partition design, Kafka Connect, schema registry (Confluent or AWS Glue), consumer group management
• Event-driven architectures in addition to traditional batch ETL
• Stream processing with Spark Streaming, Flink, or Lambda
• Avro, Protobuf, and JSON serialization, plus schema evolution

Cloud and Infrastructure

• AWS across MSK, Kinesis, Lambda, EMR, and Glue
• Terraform or CloudFormation, Docker and Kubernetes
• Observability through CloudWatch, Prometheus, Grafana, or Confluent Control Center
• Python, Java, and Scala for streaming applications

Nice to Have

• Flink for advanced processing, or ksqlDB for SQL-based stream processing
• Event sourcing and CQRS patterns
• Financial services, IoT, or another real-time data intensive domain
• Confluent Certified Administrator or Developer, AWS Solutions Architect Professional

Why Join Us

The data team here supports client work across a range of industries, so the streaming problems change shape often. You'll have the latitude to design the architecture rather than inherit somebody's decisions, and the accountability that comes with that.