Software Engineer

Roles and Responsibilities :

• Pipeline Refactoring: Deconstruct the existing C++ Flume pipeline and translate its business logic, spatial joins, and geometry processing into a clean, modular FlumePython pipeline using the Apache Beam Python SDK.
• ML Model Integration: Embed our trained AutoTFX/TensorFlow multi-class classification model into the Python Beam pipeline's ParDo steps to run inline inference on millions of candidate pairs.
• Performance Optimization: Implement geographic partitioning and S2 cell-based spatial filtering (ST_DWithin equivalent) to prevent data skew and hash-join memory issues on Borg workers.
• Reliability & Automation: Debug and resolve the legacy pipeline's current "overrun" and stuck states by designing a staged, throttled upload mechanism that writes stratified task batches to CNS/ALF.
• Testing and Validation: Write comprehensive Python unit tests and integration tests (beam test pipeline) in the google3 environment to guarantee 100% parity with legacy C++ calculations.

Required Skillset & Experience

1. Core Google Infrastructure (Must-Have)

• Flume / Apache Beam: At least 1 year of experience writing, debugging, and optimizing Flume pipelines (FlumeC++, FlumeJava, or FlumePython). Deep understanding of PCollections, PTransforms, ParDo operations, and custom DoFns.
• google3 Development Environment: Complete familiarity with Blaze, Piper, CitC, and Google's Python style guidelines.
• Borg & CNS: Experience deploying, configuring, and monitoring Borg jobs, BorgCron scheduling, and managing large-scale dataset storage in CNS (Colossus).
• Protocol Buffers: Strong experience reading, writing, and parsing nested Protocol Buffer structures (Protos) and schemas (.pb / .textpb).

2. Software Engineering & Math

• Python 3 & C++: Proficient in both Python (for the target pipeline) and C++ (to read and dissect the legacy codebase).
• F1 / GoogleSQL: Advanced SQL capabilities to construct high-performance queries and export dataset metrics.
• Basic GIS/Geospatial Math: Experience with spatial data types, coordinates, bounding boxes, or S2 Geometry.