Roles and Responsibilities :
- Pipeline Refactoring: Deconstruct the existing C++ Flume pipeline and translate its business logic, spatial joins, and geometry processing into a clean, modular FlumePython pipeline using the Apache Beam Python SDK.
- ML Model Integration: Embed our trained AutoTFX/TensorFlow multi-class classification model into the Python Beam pipeline's ParDo steps to run inline inference on millions of candidate pairs.
- Performance Optimization: Implement geographic partitioning and S2 cell-based spatial filtering (ST_DWithin equivalent) to prevent data skew and hash-join memory issues on Borg workers.
- Reliability & Automation: Debug and resolve the legacy pipeline's current "overrun" and stuck states by designing a staged, throttled upload mechanism that writes stratified task batches to CNS/ALF.
- Testing and Validation: Write comprehensive Python unit tests and integration tests (beam test pipeline) in the google3 environment to guarantee 100% parity with legacy C++ calculations.
Required Skillset & Experience
1. Core Google Infrastructure (Must-Have)
- Flume / Apache Beam: At least 1 year of experience writing, debugging, and optimizing Flume pipelines (FlumeC++, FlumeJava, or FlumePython). Deep understanding of PCollections, PTransforms, ParDo operations, and custom DoFns.
- google3 Development Environment: Complete familiarity with Blaze, Piper, CitC, and Google's Python style guidelines.
- Borg & CNS: Experience deploying, configuring, and monitoring Borg jobs, BorgCron scheduling, and managing large-scale dataset storage in CNS (Colossus).
- Protocol Buffers: Strong experience reading, writing, and parsing nested Protocol Buffer structures (Protos) and schemas (.pb / .textpb).
2. Software Engineering & Math
- Python 3 & C++: Proficient in both Python (for the target pipeline) and C++ (to read and dissect the legacy codebase).
- F1 / GoogleSQL: Advanced SQL capabilities to construct high-performance queries and export dataset metrics.
- Basic GIS/Geospatial Math: Experience with spatial data types, coordinates, bounding boxes, or S2 Geometry.
