Data Engineer

Job Summary

Job title: Data Engineer. Experience: 6+ years. Location: Chennai. Shift: US Eastern Time (5:00 PM - 2:00 AM).

We are seeking a hands-on Data Engineer to develop, optimize, and maintain automated data pipelines supporting data governance and analytics initiatives. This role will focus on building production-ready workflows for ingestion, transformation, quality checks, lineage capture, access auditing, cost usage analysis, retention tracking, and metadata integration, primarily using Azure Databricks, Azure Data Lake, and Microsoft Purview.

Responsibilities

• Pipeline Development - Design, build, and deploy robust ETL/ELT pipelines in Databricks (PySpark, SQL, Delta Lake) to ingest, transform, and curate governance and operational metadata from multiple sources landed in Databricks.
• Granular Data Quality Capture - Implement profiling logic to capture issue-level metadata (source table, column, timestamp, severity, rule type) to support drill-down from dashboards into specific records and enable targeted remediation.
• Governance Metrics Automation - Develop data pipelines to generate metrics for dashboards covering data quality, lineage, job monitoring, access permissions, query cost, usage consumption, retention lifecycle, policy enforcement, sensitive data mapping, and governance KPIs.
• Microsoft Purview Integration - Automate asset onboarding, metadata enrichment, classification tagging, and lineage extraction for integration into governance reporting.
• Data Retention Policy Enforcement - Implement logic for retention tracking and policy compliance monitoring (masking, RLS, exceptions).
• Job Query Monitoring - Build pipelines to track job performance, SLA adherence, and query costs for cost and performance optimization.
• Metadata Storage Optimization - Maintain curated Delta tables for governance metrics, structured for efficient dashboard consumption.
• Testing Troubleshooting - Monitor pipeline execution, optimize performance, and resolve issues quickly.
• Collaboration - Work closely with the lead engineer, QA, and reporting teams to validate metrics and resolve data quality issues.
• Security Compliance - Ensure all pipelines meet organizational governance, privacy, and security standards.

Qualifications

• Experience: 6+ years in data engineering, with strong Azure and Databricks experience.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.