Data Platform Lead

About the job

Upstart AEC is hiring a Data Platform Lead for a full-time, fully remote role.

This is a hands-on leadership position for someone who can help shape and deliver an Azure-based data platform from technical design through production. The role combines architecture, engineering, platform integration, security, and operational ownership.

The ideal candidate is comfortable leading delivery while still working directly with Azure, Python, SQL/T-SQL, Spark, PostgreSQL, infrastructure tooling, and production systems .

Key Responsibilities

• Help define and implement the architecture for an Azure-based data platform, including security, networking, identity, and data-serving patterns.
• Lead hands-on delivery across ingestion, transformation, storage, serving, testing, and production readiness.
• Build and oversee SQL Server Change Tracking pipelines, including incremental loads, replay, backfill, reconciliation, and schema-change handling.
• Develop data workflows using Python, SQL/T-SQL, ADLS Gen2, Parquet/Delta, and PySpark/Spark .
• Design and support PostgreSQL serving marts for analytics and application use cases.
• Implement secure multi-tenant data access, including tenant isolation, identity-based routing, regional data controls, and failure testing.
• Coordinate platform integration across AKS, Terraform, GitHub Actions/ArgoCD, Entra managed identity, Key Vault, private endpoints, and Azure Monitor .
• Partner with infrastructure and platform teams on CI/CD, GitOps, deployment, networking, and infrastructure-as-code.
• Drive production readiness through monitoring, automated testing, troubleshooting, documentation, runbooks, and knowledge transfer.

Required Skills

• Strong experience delivering production Azure data platforms .
• Hands-on experience with AKS, ADLS Gen2, private networking, managed identity, and Key Vault .
• Advanced Python and SQL/T-SQL skills.
• Experience with SQL Server Change Tracking and incremental ingestion.
• Strong understanding of backfills, replay, reconciliation, schema drift, and recovery processes.
• Experience with Parquet/Delta, PySpark/Spark, and PostgreSQL .
• Working knowledge of Terraform, CI/CD, and GitOps .
• Experience designing secure, multi-tenant data platforms.
• Ability to lead technical delivery while remaining involved in implementation, testing, and troubleshooting.

Nice to Have

• SQLMesh
• dlt
• Cube or similar semantic-layer platforms
• Dagster or comparable orchestration tools
• Experience with EAV or custom-field data models
• Retail analytics experience