Sr. Data Engineer

Description May sit hybrid onsite in Wilmington, DE, McLean, VA, Richmond, VA or NYC

Our client seeks a Data Engineer professional with strong ability to support a large-scale customer migration and data load initiative. The role will build and optimize AWS-based data pipelines using Java, Scala, and Spark, with PySpark and AWS Glue across multi-stage workflows. The engagement will emphasize event-driven patterns, data quality, and operational resilience.

Due to client requirements, applicants must be willing and able to work on a w2 basis. For our w2 consultants, we offer a great benefits package that includes Medical, Dental, and Vision benefits, 401k with company matching, and life insurance. Rate: $70.00 to $80.00/hr. w2

Responsibilities

• Design, build, and maintain AWS data pipelines using S3, Glue, Step Functions, and Lambda.
• Develop back-end data processing services in Java and Scala with Spark and PySpark.
• Implement event-based architectures leveraging EventBridge and S3 event notifications.
• Configure and scale AWS Glue jobs, addressing version and table format considerations.
• Establish data validation, security checks, schema drift handling, and backfill strategies.
• Integrate external API calls within data pipelines where required.
• Contribute to CI/CD pipelines and Spring Boot based repositories.
• Support production runbooks, reruns from event triggers, and downstream job synchronization.

Experience Requirements

• Hands-on Java with data engineering background. Python considered, Java preferred.
• Proficiency with Scala, Spark, and PySpark.
• Strong AWS experience: S3, Glue, Step Functions, Lambda, and EventBridge.
• Knowledge of state machine scheduling and multi-stage pipeline orchestration.
• Experience with data quality controls, schema evolution, and backfill management.
• CI/CD pipeline implementation and operations.
• Spring Boot repository familiarity.
• Plus: external API integration in pipelines and AI automation exposure.
• Prior experience supporting financial services data migrations preferred. Prior Capital One experience preferred.

Education Requirements