Data Engineer

[Hiring for a Client]

Position requires strong GCP / Big Query experience

As a Data Engineer, you will design and build the data infrastructure that powers our next-generation platform. You will be responsible for architecting high-throughput data pipelines, managing large-scale distributed data lakes, and ensuring the integrity and accessibility of data across our ecosystem. Your work will focus on transforming raw, high-velocity telecommunications events into actionable insights, enabling real-time analytics and supporting mission-critical decision-making for our platform.

Position Details

・Design and implement the data infrastructure that powers our B2B, B2C, and ISP business intelligence

・Build high-performance data pipelines that ingest, transform, and analyze massive volumes of telecommunications data, ensuring the data platform is robust, scalable, and ready for real-time decision-making

・Architect and optimize BigQuery datasets, including partitioning, clustering, materialized views, and complex SQL query tuning for large-scale analytical workloads

・Build and maintain production-grade ETL/ELT pipelines using orchestration frameworks such as Apache Airflow or Google Cloud Dataflow (Apache Beam)

・Develop custom data connectors, micro-transformers, and integration services using Go or Java

・Design efficient data models (star/snowflake schemas) for analytical reporting, including handling nested/repeated structures common in CDR logs

・Integrate real-time streaming telecommunications data using message brokers such as Kafka or Pub/Sub

・Ensure data consistency between operational and analytical layers, working with distributed databases such as YugabyteDB or similar PostgreSQL compatible systems

・Deploy and scale data processing workloads on Kubernetes using containerized services

・Collaborate with product managers, QA engineers, DevOps/SRE, and counterpart engineering teams across global offices to deliver end-to-end data

solutions

Work Environment

・Global distributed team across Japan, India, China, and Singapore

・Agile/Scrum development with two-week sprint cycles

・English as the primary working language for documentation, design discussions, and cross-site communication

・Modern data stack: BigQuery, Kafka/Pub/Sub, Airflow/Dataflow,

containerized on Kubernetes, deployed on cloud infrastructure

・Strong emphasis on engineering craftsmanship: code review culture, automated testing, and continuous delivery

Mandatory Qualifications:

・Minimum of 7 years of relevant working experience.

・BigQuery Mastery: Deep understanding of BigQuery architecture, including partitioning, clustering, materialized views, and optimizing complex SQL queries for massive datasets

・Data Pipeline Orchestration: Proven experience building and maintaining production-grade ETL/ELT pipelines using tools like Apache Airflow, Google Cloud Dataflow (Apache Beam), or similar orchestration frameworks

・Programming Proficiency: Strong coding skills in Go (Golang) or Java for building custom data connectors, micro-transformers, and integration services

・Data Modeling: Ability to design efficient schemas for analytical reporting, including star/snowflake schemas and handling nested/repeated data structures common in CDR logs

・Event-Driven Integration: Experience with message brokers (e.g., Kafka, Pub/Sub) for real-time ingestion of streaming telecommunications data

・SQL & Database Literacy: Expert-level SQL skills and experience working with distributed databases (e.g., YugabyteDB or similar PostgreSQL-compatible distributed systems) to ensure data consistency between operational and analytical layers

・Cloud-Native Foundation: Hands-on experience with Kubernetes (K8s) and containerization (Docker) to deploy and scale data processing workloads

・Business Level English (TOEIC 800 or above, or equivalent qualification)

Desired Qualifications:

・Infrastructure as Code (IaC): Proficiency in Terraform or Pulumi for provisioning data infrastructure and BigQuery resources in a reproducible manner

・Streaming Analytics: Experience with real-time stream processing frameworks like Apache Flink or Spark Streaming for sub-second CDR analysis

・Data Governance & Quality: Experience with data cataloging tools (e.g., Data Catalog, Collibra) and implementing automated data quality frameworks (Great Expectations, dbt tests)

・Advanced Telecom Knowledge: Domain expertise in telecommunications billing, charging systems (OCS/CCS), or network protocol analysis (Diameter, GTP, SIP)

・CI/CD for Data: Experience setting up CI/CD pipelines specifically for data projects (e.g., automated SQL linting, schema migration testing, and data regression testing)

・ML Integration: Basic familiarity with BigQuery ML or Vertex AI for predictive analytics on historical CDR patterns (e.g., churn prediction, anomaly detection)

・Experience in agile environments with globally distributed teams

Language: English

Visa sponsorship available