DATA QUALITY ENGINEER

Job Title: Senior Data QA / Test Engineer

Location: Plano, TX 75024

Visa :- Independent Visa Only

Duration: Contractual / Fulltime Role (W2)

Note:

We are looking for a Senior Data QA / Test Engineer with strong experience in data/ETL testing, SQL Server, Databricks, Azure Data Factory, PySpark, and data migration testing . The ideal candidate will own the testing strategy for migrating a SQL Server data platform to Databricks on Azure, validate data parity between legacy and modern platforms, establish quality gates, and ensure end-to-end validation of ADF pipelines and Delta Lake workloads.

Key Skills (50%)

Strong experience with Data QA / ETL Testing / Data Validation

Strong SQL experience including T-SQL, Spark SQL, joins, window functions, aggregates, and data reconciliation

Hands-on experience with Databricks, Delta Lake, notebooks, Jobs/Workflows, and Unity Catalog

Hands-on experience with Azure Data Factory (ADF) including pipelines, triggers, datasets, monitoring, and debugging

Strong experience with PySpark for test automation and reconciliation frameworks

Experience with source-to-target reconciliation, row counts, checksums/hashes, aggregates, and row-level data comparisons

Experience testing data transformations, migrations, schema changes, incremental loads, and business logic

Experience integrating automated tests into CI/CD using Azure DevOps or GitHub Actions

Experience (30%)

5+ years of experience in Data / ETL Testing

Experience testing at least one data platform migration

Strong experience validating SQL Server data platforms and modern cloud data platforms

Experience converting SQL business logic, stored procedures, and views into test cases

Experience validating PySpark / Spark SQL outputs against legacy SQL results

Experience defining quality gates, entry/exit criteria, test metrics, and release readiness

Experience testing ADF pipelines, dependencies, parameters, triggers, retries, and incremental/watermark logic

Experience with Delta Lake MERGE/upsert, SCD Type 1/2, schema evolution, deduplication, and late-arriving data

Experience with UAT, regression testing, defect management, and production validation

Other Skills (20%)

Strong analytical, troubleshooting, and problem-solving skills

Experience reporting defects, reconciliation results, test metrics, and go/no-go readiness

Strong communication and collaboration with Data Engineers, Data Quality Engineers, Developers, and Project Leadership

Experience working in Agile/Scrum environments

Strong documentation and test strategy development skills

Experience with pytest or similar test automation frameworks

Knowledge of Terraform, Databricks Asset Bundles, and Azure DevOps

Knowledge of Databricks Labs DQX, Great Expectations, Soda, or Lakeflow/DLT expectations is an asset

Experience with Spark performance testing including partitioning, file sizing, and cluster sizing is an asset

Knowledge of SSIS, SSRS, or Power BI report validation is an asset

Must Have

Data / ETL Testing

SQL / T-SQL / Spark SQL

Databricks / Delta Lake

Azure Data Factory

PySpark

Data Migration Testing

Source-to-Target Reconciliation

Data Validation / Quality Testing

Test Automation / pytest

CI/CD / Azure DevOps or GitHub Actions

Quality Gates / Test Strategy

Responsibilities

Define and execute the migration test strategy and test plan covering schema, data, transformation logic, performance, and regression testing

Build automated source-to-target reconciliation for migrated tables

Validate row counts, column-level checksums/hashes, aggregates, and row-level differences

Convert legacy SQL business logic, stored procedures, and views into test cases

Validate equivalent PySpark/Spark SQL outputs against legacy SQL results

Define and enforce quality gates across code merge, Bronze/Silver/Gold promotion, UAT sign-off, and production cutover

Validate Azure Data Factory pipelines including parameters, triggers, dependencies, mappings, failure/retry paths, incremental loads, and watermark logic

Test Delta Lake behavior including MERGE/upsert, SCD Type 1/2, schema evolution, deduplication, and late-arriving data

Integrate automated tests into CI/CD pipelines using Azure DevOps, GitHub Actions, or equivalent

Partner with the Data Quality Engineer to embed DQX/data-quality checks into pipelines

Report defects, reconciliation results, test metrics, and go/no-go readiness to project leadership

Support UAT, regression testing, production validation, defect resolution, and migration cutover activities

Ensure zero critical data defects are introduced into production following each migration wave

If interested, please share your resume at:

📧 [email protected]