Lead Data Engineer

Role : Lead /Senior Data Engineer (Full Time/FTE)

Location : New York City / Los Angeles, CA

Experience : 8+ Years

Work Model: Hybrid (3 days in a week)

Primary Skill: PySpark, and AWS, Databricks,Delta Lake,Notebooks, and SQL

Secondary Skill: media measurement datasets (Nielsen)

About the Role:

We are looking for an experienced Lead Data Engineer with deep expertise in Databricks, PySpark, and AWS. The ideal candidate will have a strong understanding of media measurement datasets (Nielsen) and a proven ability to lead multiple concurrent projects across onsite and offshore teams. This role requires a senior technical leader to define architecture, manage delivery, and mentor teams within the fast-paced New York media industry.

Key Responsibilities:

1. Technical Leadership: Architect and deliver enterprise-scale data platforms using Databricks, PySpark, Delta Lake, and AWS. Lead the development of media analytics pipelines, including the ingestion and modeling of Nielsen data.
2. Project & Delivery Management: Manage scope, timelines, and risks for multiple simultaneous projects. Collaborate with New York-based stakeholders and drive sprint delivery.
3. Onsite–Offshore Coordination: Act as the primary technical lead onsite, coordinating with offshore teams to ensure alignment, quality, and continuity.
4. Team Mentorship: Mentor junior and mid-level engineers through code reviews and architectural guidance.

Skills:

1. Databricks: Delta Lake, Workflows, Unity Catalog, Notebooks, and SQL.
2. Data Processing: Strong PySpark and distributed computing experience.
3. AWS: S3, EMR, Glue, Lambda, Step Functions, Kinesis, and IAM.
4. Domain Expertise: Proven experience handling Nielsen datasets (National/Local TV ratings, Digital Ad ratings, Panels).
5. Leadership: Experience leading large data engineering programs and managing senior stakeholders.

Good to Have:

1. Knowledge of Snowflake, Airflow, or Dagster.
2. Infrastructure as Code (Terraform / CloudFormation).
3. Exposure to ad-tech ecosystems (Comscore, Kantar, IAS, Moat) or ML feature pipelines.

Soft Skills:

1. Strong decision-making and ownership mindset.
2. Exceptional communication skills for interacting with senior business stakeholders.
3. Strategic, problem-solving, and delivery-focused approach.