Data Quality Engineer (Senior)

Who are we?

CFI Financial Group is an award-winning trading provider, possessing more than 25 years of experience with multiple offices around the world including London, Larnaca, Beirut, Amman, Dubai, Kuwait, Port Louis, and others.

Check out more about CFI here.

CFI is hiring! Make your mark in the online trading industry.

Are you looking to pursue a career in finance? Do you want to work with a dynamic and growing team in the exciting world of online trading and investing? If you answered yes, then we have some amazing opportunities for you!

Description

Join our Data & AI Engineering team as a Senior Data Engineer specializing in Data Quality and QA. This role is ideal for engineers who are passionate about building trusted, reliable, and production-ready data products.

You'll work within a modern Databricks Lakehouse environment, spending approximately 60% of your time driving data quality, testing, observability, and governance initiatives, and 40% building scalable data pipelines with quality embedded from day one.

Responsibilities

• Design and execute test strategies for Databricks pipelines, APIs, and data products across bronze, silver, and gold layers.
• Design and build automated testing frameworks using PySpark, Python, and pytest for data quality validation and end-to-end pipeline testing.
• Validate data accuracy, completeness, consistency, timeliness, and lineage across datasets governed within Unity Catalog.
• Test ingestion pipelines, including incremental/watermark loads and CDC processes, covering updates, deletes, and late-arriving data.
• Run regression testing on Databricks Asset Bundle deployments across development, staging, and production environments.
• Implement automated data observability, data quality monitoring, anomaly detection, alerting, and SLA tracking.
• Validate pipeline performance, scalability, reliability, and resilience under production workloads.
• Verify PII masking, access controls (column masking, row-level filtering), and regulatory compliance requirements including GDPR and financial services standards.
• Track, analyze, and report on data quality KPIs, incidents, and remediation activities.
• Build and maintain production-grade PySpark and SQL pipelines on Databricks (Delta Lake, Lakeflow, Workflows).
• Embed quality at the source through expectations, schema validation, reconciliation checks, and quarantine/rejection handling.
• Develop reusable data quality frameworks, validation libraries, and testing templates adopted across engineering teams.
• Contribute to ingestion design including incremental loading, CDC, idempotency, reconciliation, and recovery mechanisms.
• Deploy and maintain solutions using Databricks Asset Bundles and GitHub Actions CI/CD, including linting, unit testing, integration testing, schema validation, and automated quality gates.
• Collaborate on data observability and monitoring initiatives to improve platform reliability and trustworthiness.
• Partner closely with Data & AI Engineers, Analytics Engineers, and the Platform team, participating in design reviews and pull request reviews.
• Evaluate and recommend data quality tooling such as Great Expectations, Soda Core, dbt tests, Lakeflow Expectations, and similar solutions.
• Mentor team members on testing strategies, data quality practices, observability, and engineering standards.

Requirements

• Bachelor’s degree in computer science, Engineering, Data Science, or a related technical field; master’s degree is a plus.
• Minimum 3 years of experience in data engineering, data quality engineering, ETL testing, or test automation for data platforms.
• Proven experience building, validating, and supporting production data pipelines in cloud environments.
• Strong Python, PySpark, and SQL skills.
• Hands-on experience with Databricks, including Unity Catalog, Delta Lake, Asset Bundles, Workflows, and Lakeflow/DLT .
• Experience implementing automated testing frameworks and data quality controls.
• Hands-on experience with data quality tools such as Great Expectations, Soda Core, dbt tests , or equivalent platforms.
• Experience with data observability and monitoring solutions.
• Experience with performance testing, scalability testing, and pipeline reliability validation.
• CI/CD and Git-based development workflows (GitHub Actions, GitLab CI, Jenkins, or similar).
• Working knowledge of data governance, GDPR, and security testing practices including masking and access controls.

Nice to Have

• Experience designing and testing watermark-based incremental ingestion and CDC pipelines.
• Exposure to dbt for data transformation, testing, and documentation.
• AWS experience (S3, IAM, networking) and orchestration tools such as Databricks Workflows or Airflow.
• API testing experience (Postman, REST Assured, OpenAPI).
• Data catalog, governance, and lineage platforms (Unity Catalog, OpenMetadata, Collibra, etc.).
• Power BI experience.
• Experience working within financial services, fintech, or other highly regulated environments.

Why Join CFI?

• Join a fast-growing, multinational organization with a strong global presence.
• Competitive compensation and benefits package.
• Work alongside highly experienced industry professionals.
• Collaborative, supportive, and innovative work environment.
• Continuous learning opportunities and clear career growth pathways.
• Be part of a team shaping the future of data and AI within a leading financial services organization.