Data Engineer - Tester

Job Summary

We are currently seeking a Data Engineer - Tester to join our team in Bengaluru, Karntaka (IN-KA), India (IN).

Required Technical Skills

• Data Testing Frameworks: Hands-on experience with DataTester to automate data quality checks.
• Big Data Processing: Strong proficiency in PySpark and Apache Spark for manipulating and querying large-scale datasets.
• Cloud Storage: Deep understanding of AWS S3 storage structures, partitioning strategies, and file formats (Parquet, Delta, Avro).
• Testing Methodology: Solid background in functional and non-functional data testing, including data migration and regression testing.
• Database & Querying: Advanced SQL skills for querying relational databases and verifying data warehousing solutions.
• Version Control: Experience with Git and CI/CD deployment pipelines for automated test execution

Key Responsibilities

• Pipeline Validation: Build automated test suites to validate complex ETL/ELT pipelines using PySpark and DataTester.
• Data Quality Assurance: Verify data accuracy, completeness, and consistency across source systems and AWS S3 data lakes.
• Test Automation: Design and maintain scalable, reusable data testing frameworks to perform schema validation, null checks, and data reconciliation.
• Component Testing: Conduct rigorous unit, integration, and regression testing on distributed data processing applications.
• Defect Tracking: Identify, document, and track data anomalies, performance bottlenecks, and pipeline failures.
• Collaboration: Partner closely with core Data Engineers and Platform teams to understand data models and implement test-driven development (TDD) practices.