Senior Data Platform / Data Product Engineering Lead

Senior Data Platform / Data Product Engineering Lead

Must Have Technical/Functional Skills

Job Description: Senior Data Platform / Data Product Engineering Lead

Role Overview

We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development.

The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams.

________________________________________

Required Skills & Experience

• 10+ years of experience in Data Engineering / Data Platform roles
• Strong hands-on expertise in:
• Databricks (Delta Lake, workflows, DAG)
• Apache Airflow / Astronomer
• Python, SQL, DBT ,AWS
• Proven experience implementing CI/CD frameworks (Harness, GitHub Actions, Azure DevOps)
• Deep understanding of:
• Data governance (catalogs, lineage, contracts, metadata)
• Data quality and masking techniques
• Enterprise data platforms and marketplace ecosystems
• Experience with API-based integrations (e.g., entitlement systems like AccessCentral)
• Monitoring/observability tools (e.g., Datadog)

Job Description: Senior Data Platform / Data Product Engineering Lead

Role Overview

We are looking for a Senior Data Platform / Data Product Engineering Lead to drive enterprise-scale data product lifecycle enablement across modern data platforms. This role will lead the design, standardization, and adoption of a paved path for data creators, enabling self-service, governed, and scalable data product development.

The role requires deep expertise in Databricks, Airflow (Astronomer), CI/CD automation, data governance, and data marketplace constructs, along with the ability to lead platform transformation initiatives and mentor engineering teams.

________________________________________

Core Responsibilities

• Platform Strategy & Self-Service Enablement
• Define and implement a self-service data platform strategy to reduce onboarding friction.
• Lead automated provisioning of:
• Databricks workspaces (via DevHub)
• Airflow/Astronomer environments
• Access and entitlements (AccessCentral APIs)
• Establish isolated, stable development environments for federated teams.
• Drive platform observability by integrating metrics into tools lik e Datadog.

________________________________________

• Data Discovery, Access & Governance
• Architect and implement enterprise-wide data discovery and marketplace enablement.
• Drive adoption of:
• Data contracts
• Metadata standards
• Domain-aligned catalogs (Unity Catalog)
• Enable secure access to curated, masked datasets in dev and production environments.
• Implement tagging, access patterns, and entitlement automation.
• Partner with risk/compliance teams to enforce regulatory and governance controls (BFSI-aligned).

________________________________________

• Data Engineering, Curation & Orchestration
• Lead design of scalable data ingestion, curation, and transformation frameworks.
• Build and standardize modular, reusable frameworks:
• LaunchLake templates
• Airflow DAG libraries
• DBT-based transformation models
• Ensure:
• Data quality and consistency
• Embedded governance and compliance policies
• Enable concurrent development using standardized patterns and environments.

________________________________________

• CI/CD, Automation & Deployment
• Define and enforce standard CI/CD pipelines across data products:
• Harness (or equivalent)
• Databricks Asset Bundles (DAB)
• Automate:
• DAG deployments (Airflow/Astronomer)
• DBT pipeline releases
• Reduce manual interventions and ensure consistent, repeatable deployments.
• Improve release reliability with feedback loops, notifications, and monitoring.

________________________________________

• Data Product Publishing & Marketplace Enablement
• Drive publishing of data products to:
• Unity Catalog
• Enterprise Data Marketplace
• Define and enforce:
• Documentation standards
• Data ownership models
• Versioning and contract management
• Enable cross-domain data sharing with embedded governance and access controls.

________________________________________

• Operations, Observability & Reliability
• Establish a scalable operating model for data product support.
• Implement:
• Monitoring dashboards (Datadog)
• Data quality frameworks
• Usage and performance metrics tracking
• Improve visibility into:
• Pipeline health
• Data lineage
• Access and consumption patterns
• Lead incident management, root cause analysis, and escalation processes.

________________________________________

• Transformation, Roadmap & Innovation
• Drive execution of platform priorities such as:
• Data contract activation strategy
• Domain catalog integration
• Data masking in development environments
• Data quality frameworks
• DBT adoption and POCs
• Lead maturity uplift from:
• Manual, fragmented workflows → standardized, automated paved paths
• Champion continuous improvement and innovation in developer experience.

________________________________________

Salary Range- $100,000-$120,000 a year