Staff SE - Apache Kafka

About the Role

We are building an in-house team to maintain and contribute to our internal Apache Kafka distribution.

This is not an operations role — you will work on the Kafka codebase itself: the broker, the KRaft

controller, the protocol, the clients, and the storage layer. You will be our technical anchor for all things

Kafka internals, driving upstream contributions, backporting critical fixes, and shaping our internal

distribution to meet the needs of a large-scale production fleet.

We already do this successfully with GlusterFS, where we maintain an in-house distribution backed by

deep upstream expertise. We want to replicate that model for Kafka.

Key Responsibilities

• Own our internal Kafka distribution: branch management, patch curation, release cutting, upgrade

planning

• Contribute upstream to Apache Kafka — bug fixes, KIPs, code reviews, design discussions
• Diagnose and fix deep broker/controller issues: replication, partition reassignment, KRaft

consensus, log storage, ISR management, protocol handling

• Build custom patches for production needs that aren't suitable for upstream (performance tuning

for our workload profile, observability hooks, operational safety valves)

• Establish engineering practices for Kafka development within the team: how we test, how we

benchmark, how we validate patches before rolling them out

• Mentor a supporting engineer and grow the team's capability over time
• Act as the internal authority on Kafka internals — advise platform teams on configuration,

capacity planning, and architectural decisions that require knowledge of broker behavior

• Act as the primary interface for audit and compliance teams, ensuring PhonePe Kafka

distribution meets various compliance and regulatory standards

What are we looking for?

• Deep familiarity with the Kafka codebase — you have read, modified, and debugged the broker,

controller, or client internals. Not just operated Kafka, but worked on it.

• Track record of upstream contribution — committed patches, participated in KIP discussions,

reviewed PRs, or equivalent engagement with the Apache Kafka community. Active

committer/PMC status is strongly preferred but not strictly required if your contribution history

demonstrates equivalent depth.

• Strong distributed systems fundamentals — consensus protocols, replication, exactly-once

semantics, log-structured storage, distributed state machines. You understand these not as

buzzwords but as engineering tradeoffs you've navigated.

• Systems-level programming comfort — JVM internals (GC tuning, memory layout, JIT behavior),

OS-level I/O (page cache, fsync semantics, zero-copy), networking (TCP tuning, socket buffers).

Kafka performance work lives at this layer.

• Experience maintaining a fork or distribution — you've managed the tension between carrying

local patches and staying mergeable with upstream. You know what it costs to diverge and how to

minimize that cost.

• 8+ years of experience in systems/infrastructure software engineering, with a meaningful portion

spent on Kafka or comparable distributed systems (Pulsar, Bookkeeper, Zookeeper, etcd,

CockroachDB, etc.)

Good to have

• Apache Kafka Committer or PMC member
• Experience with KRaft (the ZooKeeper-free controller)
• Authored or co-authored a KIP
• Experience with Kafka Streams or Kafka Connect internals (not just usage)
• Familiarity with Kafka's test infrastructure (system tests, Trogdor, ducktape)

What this is NOT

• This is not a Kafka SRE/operations role. We have separate teams that run Kafka. You build and

fix the thing they run.

• This is not a "use Kafka to build applications" role. You work on Kafka itself.
• This is not a consulting role. You will write code, review code, and ship patches.
• We commit to protecting your upstream contribution time and distribution engineering focus. In

return, you commit to building the team's capability so that knowledge doesn't bottleneck through

you.

What we Offer

• Dedicated upstream time. This isn't a "contribute in your spare time" situation. Upstream work is

the job.

• Real production scale. Your patches get tested against a massive fleet. You see how Kafka

behaves at a scale that most contributors don't have access to.

• We've done this before with GlusterFS, it works, leadership understands what it takes.
• You get to shape the team. Participate in hiring the supporting engineer, define practices, set the

roadmap. This is a build-from-scratch opportunity.

Required Skills ['apache kafka']

Additional Information