Flume C++ Developer

Role Overview

We are looking for a Software Engineer to lead the migration of our mission-critical Mapfacts Improvement Program (MIP) organizational relations pipeline. The current legacy pipeline is written in FlumeC++ and runs as a daily BorgCron job, compiling candidate occupant pairs, evaluating map heuristics, and exporting tasks to the ALF (Atlas Labelling Framework) operator queue.

Due to the scaling demands of our new Machine Learning-based relationship models (which are authored in Python), we are migrating this entire backend data pipeline to FlumePython (Apache Beam SDK). The successful candidate will refactor the legacy C++ processing logic, integrate our new ML inference scoring workflow, and establish a robust, non-overrunning automated upload queue to CNS and ALF.

Key Responsibilities

• Pipeline Refactoring: Deconstruct the existing C++ Flume pipeline and translate its business logic, spatial joins, and geometry processing into a clean, modular FlumePython pipeline using the Apache Beam Python SDK.
• ML Model Integration: Embed our trained AutoTFX/TensorFlow multi-class classification model into the Python Beam pipeline's ParDo steps to run inline inference on millions of candidate pairs.
• Performance Optimization: Implement geographic partitioning and S2 cell-based spatial filtering (ST_DWithin equivalent) to prevent data skew and hash-join memory issues on Borg workers.
• Reliability & Automation: Debug and resolve the legacy pipeline's current "overrun" and stuck states by designing a staged, throttled upload mechanism that writes stratified task batches to CNS/ALF.
• Testing and Validation: Write comprehensive Python unit tests and integration tests (beam_test_pipeline) in the google3 environment to guarantee 100% parity with legacy C++ calculations.

Required Skillset & Experience

1. Core Google Infrastructure (Must-Have)

• Flume / Apache Beam: At least 1 year of experience writing, debugging, and optimizing Flume pipelines (FlumeC++, FlumeJava, or FlumePython). Deep understanding of PCollections, PTransforms, ParDo operations, and custom DoFns.
• google3 Development Environment: Complete familiarity with Blaze, Piper, CitC, and Google's Python style guidelines.
• Borg & CNS: Experience deploying, configuring, and monitoring Borg jobs, BorgCron scheduling, and managing large-scale dataset storage in CNS (Colossus).
• Protocol Buffers: Strong experience reading, writing, and parsing nested Protocol Buffer structures (Protos) and schemas (.pb / .textpb).

2. Software Engineering & Math

• Python 3 & C++: Proficient in both Python (for the target pipeline) and C++ (to read and dissect the legacy codebase).
• F1 / GoogleSQL: Advanced SQL capabilities to construct high-performance queries and export dataset metrics.
• Basic GIS/Geospatial Math: Experience with spatial data types, coordinates, bounding boxes, or S2 Geometry.