Senior BigData Engineer - Payments - Local only

Role : Senior BigData Engineer - BFSI Cloud & Payments Platform

Location : Charlotte, NC (Hybrid)

Role Summary

We are seeking an experienced Senior Data Engineer to design, develop, and optimize large-scale real-time data platforms supporting transaction tracing, reconciliation, observability, analytics, and graph-enabled investigations for Banking and Financial Services (BFSI) clients. The role requires expertise in streaming architectures, event-driven systems, distributed data processing, transaction lifecycle management, and modern cloud data platforms.

________________________________________

Key Responsibilities

Real-Time Data Engineering & Platform Development

• Design and develop real-time and batch data ingestion pipelines.

• Build scalable data processing frameworks using Kafka, Spark Streaming, and PySpark.

• Develop cloud-native and distributed data processing solutions.

• Create reusable data engineering components, frameworks, and accelerators.

• Design highly available, fault-tolerant, and performance-optimized data pipelines.

• Establish data engineering standards, patterns, and best practices.

BFSI Data Processing & Transaction Traceability

Develop and support data solutions across:

• Payments Platforms (ACH, RTP, FedNow, SWIFT, ISO 20022)

• ATM Transaction Processing

Key responsibilities include:

• Design transaction tracing and correlation frameworks.

• Create canonical transaction and event processing models.

• Correlate transactions using STAN, RRN, timestamps, terminal IDs, and account references.

• Build reconciliation and exception management workflows.

• Enable end-to-end transaction lifecycle visibility and auditability.

• Perform log analysis and transaction mapping.

Data Quality, Reliability & Observability

• Implement reconciliation, validation, and data quality frameworks.

• Establish end-to-end lineage tracking across all processing stages.

• Detect missing transactions, duplicate events, processing anomalies, and broken lineage chains.

• Build monitoring, alerting, and operational observability solutions.

• Define data quality metrics, SLAs, and operational dashboards.

• Support root-cause analysis and transaction recovery investigations.

Security, Governance & Compliance

• Ensure adherence to banking security and regulatory requirements.

• Implement secure data processing and access control mechanisms.

• Implement data encryption, masking, and secure data handling practices.

• Support audit, lineage, traceability, and regulatory reporting requirements.

• Ensure compliance with enterprise data governance standards.

Cloud Engineering & DevSecOps

• Build and deploy cloud-native data engineering solutions.

• Implement CI/CD pipelines for data platform deployments.

• Support containerized workloads and Kubernetes-based data platforms.

• Enable DevSecOps practices across data engineering solutions.

Data, Analytics & Knowledge Graph Engineering

• Design Operational Data platform Data pipelines.

• Develop real-time analytics and event-streaming pipelines.

• Design data preparation and transformation pipelines for Neo4j and graph databases.

• Implement entity extraction and relationship mapping frameworks.

• Build graph-ready datasets supporting transaction tracing and investigation.

• Support graph analytics, impact analysis, and root-cause investigations.

• Enable AI/ML and GenAI initiatives through trusted and governed data assets.

Leadership & Stakeholder Management

• Collaborate with solution architects, business teams, operations, payments, fraud, and technology stakeholders.

• Participate in architecture reviews, technical governance, and design discussions.

• Mentor junior and mid-level data engineers.

• Support estimation, planning, and technical solution discussions.

________________________________________

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or related field.

• 8+ years of experience in Data Engineering and distributed data processing.

• Strong hands-on expertise with:

o Apache Kafka

o Apache Spark/PySpark

o Python

o SQL

o NOSQL DB

o CDC Frameworks

o Event-Driven Architectures

o Graph Databases

• Experience building large-scale streaming and batch processing pipelines.

• Strong understanding of distributed systems and real-time data processing.

• Experience with Operational Data platforms

• Knowledge of transaction processing systems, event correlation, and log analytics.

• Strong troubleshooting, performance tuning, and optimization skills.

________________________________________

Preferred Qualifications

• Banking and Financial Services domain experience.

• Hands-on experience with Payments and Transaction Processing platforms.

• Knowledge of ISO 8583, ACH, RTP, SWIFT, and ISO 20022 standards.

• Experience with Neo4j and Graph Database technologies.

• Familiarity with Knowledge Graph and Graph Analytics solutions.

• Experience supporting observability and transaction monitoring platforms.

• Exposure to AI/ML and GenAI-enabled analytics solutions.

• Cloud certifications in Azure, AWS, or GCP preferred.

________________________________________

Success Measures

• High-performance real-time data processing pipelines.

• Accurate transaction correlation and traceability across systems.

• Reliable transaction lineage and audit tracking.

• High-quality graph-ready datasets supporting investigations and analytics.

• Improved observability and operational monitoring.

• Reduced transaction processing failures and reconciliation gaps.

• Secure, scalable, and compliant data engineering solutions.

• Successful delivery of enterprise-grade data platforms supporting banking and payments modernization.