Lead Data And AI Architect

JD of Lead Data and AI Architect

Exp - 8-12yrs , 40 lpa

Architect, build, and maintain robust, scalable data pipelines that ingest, transform, and serve data from multiple internal and external sources.

Own the end-to-end orchestration of data workflows using tools like Dagster, ensuring observability, reliability, and maintainability of pipelines.

Design and implement entity resolution workflows — including matching, merging, and survivorship logic — using tools such as Splink, to produce clean, deduplicated, golden records.

Build and maintain web scrapers to source data from external providers, ensuring resilience to source changes, rate limits, and data quality issues.

Integrate and reconcile data coming from multiple, often inconsistent, sources into unified, trustworthy datasets.

Design and maintain data models and schemas across transactional and analytical systems, ensuring consistency, scalability, and performance.

Leverage AI/LLM-based tools and techniques to enhance data pipeline capabilities — e.g., intelligent data extraction, automated data quality checks, or AI-assisted entity matching.

Define and enforce best practices around pipeline design, testing, monitoring, and documentation.

Collaborate closely with data engineers, product managers, and other stakeholders to translate business requirements into scalable data architecture.

Provide technical leadership and mentorship to the data engineering team.

Required Skills & Experience

Strong hands-on experience building and maintaining production-grade data pipelines at scale.

Practical experience with Dagster (or similar orchestration tools like Airflow/Prefect) for pipeline orchestration.

Experience with Splink or similar probabilistic/deterministic record linkage tools for entity matching, merging, and survivorship.

Strong proficiency in Python, including experience writing and maintaining web scrapers.

Proven experience integrating and maintaining data pipelines that pull from multiple, heterogeneous data sources.

Experience applying AI/ML tools within data engineering workflows (e.g., LLM-assisted data cleaning, extraction, or matching).

Hands-on experience with relational and distributed databases such as PostgreSQL and Google Cloud Spanner.

Strong understanding of data modeling principles (normalization, dimensional modeling, schema design) across OLTP and OLAP systems.

Experience with cloud data warehousing platforms such as BigQuery, Redshift, and cloud platforms (GCP/AWS/Azure).

Strong communication skills and experience working cross-functionally with engineering and product teams.

Experience with distributed data processing frameworks (e.g., Spark, Dask).

Familiarity with data governance, lineage, and cataloging tools.

Prior experience in a lead or architect-level role guiding a data engineering team.

What We're Looking For

A technically strong, hands-on leader who can balance architectural thinking with the practical grit of debugging a flaky scraper or tuning a matching algorithm — someone who's comfortable owning both the big picture and the messy details of real-world data.

Pay: ₹3,500,000.00 - ₹4,000,000.00 per year

Work Location: In person