Data Engineer

About the Company:

At Syncredible Solutions, we believe in the transformative power of technology. we specialize in Digital transformation services, Enterprise solutions, Power BI Analytics solutions and staffing services.

Job Description

The Data Engineer will join the Datastore Migration Factory team, responsible for migrating data and workloads from the on-premises Data Lake to an AWS-hosted Lakehouse environment. This is a high-visibility, business-critical project at Goldman Sachs.

Key Responsibilities

1. Pipeline Migration

• Refactor and migrate extraction logic and job scheduling from legacy frameworks to the new Lakehouse environment.
• Execute data migration while ensuring data integrity and consistency.
• Collaborate with data owners and stakeholders to validate migrated assets and obtain sign-off.

2. Consumption Pattern Migration

• Convert and optimize legacy SQL and Spark-based workloads for compatibility with Snowflake and Apache Iceberg.
• Analyze data usage patterns to identify and deliver required data products.
• Coordinate with stakeholders to validate migrated workloads and meet business requirements.

3. Data Reconciliation & Quality

• Develop and execute data validation and reconciliation processes to ensure migrated data remains functionally equivalent to existing production datasets.
• Collaborate with data management platform teams and adapt to new workflows and technologies.

Technical Skills

1. Basic Qualifications

• Education: Bachelor’s or Master’s degree in Computer Science, Applied Mathematics, Engineering, or a related quantitative field.
• Experience: 3–5 years of hands-on coding experience in a collaborative environment, with strong SQL troubleshooting and basic scripting skills.
• Programming: Proficiency in Python or Java.
• Development Practices: Strong understanding of SDLC, CI/CD best practices, and Kubernetes deployments.

2. Core Data Engineering Competencies

• Temporal Data Modeling: Managing historical data and state changes, including SCD Type 2.
• Schema Management: Schema evolution and enforcement, particularly with Apache Iceberg.
• Performance Optimization: Data partitioning and clustering.
• Data Architecture: Normalization, denormalization, and the use of natural and surrogate keys.

3. Technical Stack

• Extraction & Processing: Kafka, ANSI SQL, FTP, Apache Spark
• Data Formats: JSON, Avro, Parquet
• Platforms: Hadoop (HDFS/Hive), Snowflake, Apache Iceberg, Sybase IQ

Core Competencies

• Demonstrates integrity, accountability, and sound ethical judgment.
• Collaborates effectively across teams, functions, and global time zones.
• Communicates clearly and manages stakeholder expectations proactively.
• Takes ownership of deliverables and drives tasks to completion.
• Maintains urgency while ensuring quality and professionalism.
• Demonstrates curiosity, identifies risks early, and continuously seeks feedback and improvement.

Equal Opportunity Statement:

Syncredible Solutions is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status.