Lead Data Engineer

Role Description & Requirements:

• Lead technical architecture and solution design for a large FSI customer Transactions Ingestion Data Platform Real Time and Batch
• Interact with customer’s data engineering team to provide opinionated guidance on Transactions Ingestion Data platform architecture, best practices, schema designs (Spanner/BigQuery), and solution development
• Independently develop and unit test ingestion and processing pipelines (e.g., Kafka to Spanner/BigQuery via Dataflow, GCS to Spanner/BigQuery), and handover developed solutions to customer’s technical team
• Perform pair programming with customer’s data engineers
• Project manage sprint-based engagements, leading sprint planning, backlog management, scope definition, risks/issues, and timelines for successful MVP delivery
• Collaborate with cross-disciplinary project team, play the role of workstream or overall project tech lead depending on size and scale of project / program

Minimum Qualifications

Expertise Required

• In-depth knowledge and experience of GCP data and analytics technologies
• Designing Large scale Transaction Data Platform architecture for (a) migrating open source or public cloud data platforms to GCP cloud native services (b) designing greenfield data platforms on GCP
• Define solution architecture and detailed design, and perform hands-on implementation for
• data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on-prem files, on-prem databases, APIs, etc.
• real-time data ingestion and event-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery
• Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL
• BQ query performance optimisation
• CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform
• Orchestration setup and Cloud Composer DAG development for data pipeline workflows
• Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog) ○ Experience applying Generative AI technologies and integrations in enterprise environments.

Tools and languages experience Required

Must have:

• GCP Dataflow, Cloud Spanner, BigQuery, Pub/Sub, Cloud Composer, Cloud Workflows, Confluent Kafka, Cloud Run, Cloud Build, Terraform
• Must have: programming knowledge and willingness to be hands-on - Python, Java

Good to Have:

• Experience with dbt or Dataform, Terraform in context of data pipelines
• Experience with open source ecosystem and distributions such as Hadoop, Spark, Cloudera/Hortonworks and frameworks and tech such as SPARK, Oozie, Kafka, HBASE
• Understanding and experience with NoSQL databases such as HBASE, MongoDB
• Knowledge of cloud databases such as Spanner, BigTable, Cloud SQL, DB migrations Certification
• Good to have GCP Data engineering Please share any profiles matching these requirements.

LTM is proud to be an equal opportunity employer. We are committed to equal employment opportunity regardless of race, ethnicity, nationality, gender, gender-identity, gender expression, language, age, sexual orientation, religion, marital status, veteran status, socio-economic status, disability or any other characteristic protected by applicable law.

▶️ Watch the video: https://www.youtube.com/watch?v=5tTtIX92ipY