Senior Platform Data Engineer

Senior Platform Data Engineer

We are seeking a Senior Platform Data Engineer. This is a production engineering role focused on building and operating a centralized, enterprise-grade data platform that institutionalizes proprietary knowledge and delivers data as a product — not a side effect.

You will design, build, and operate models, pipelines, integrations, and infrastructure across the full medallion architecture (Bronze → Silver → Gold). This role requires true end-to-end ownership: design, build, test, deploy, monitor, and continuously improve.

What You’ll Do

• Build and maintain ingestion pipelines from raw Bronze landing through curated Gold domain models
• Develop transformation models using dbt with testing, Slim CI, and exposure tracking
• Author and manage production-grade Airflow DAGs with SLA monitoring and backfill strategies
• Design and implement streaming and event-driven pipelines (Kafka, Glue Streaming, Delta Live Tables)
• Collaborate with data scientists, platform engineers, and product teams to ship reliable data products
• Implement data quality frameworks and enforce governance standards
• Containerize workloads and deploy to Kubernetes (EKS)
• Contribute to infrastructure as code (Terraform provisioning for S3, IAM, Glue, etc.)
• Document lineage, metadata, ownership, and sensitivity classification

This is not a “pipeline-only” role — it is platform ownership at scale.

Core Technical Skills

Python

• Production-level standards

SQL

• Advanced analytical queries, window functions, optimization
• Experience with Redshift, Snowflake, or Databricks SQL

dbt

• Gold-layer modeling and testing
• Slim CI, exposures, meta configurations

Orchestration

• Apache Airflow (DAG authoring, task dependencies, SLAs, backfills)
• Prefect or Dagster is a plus

Streaming & Event Pipelines

• Kafka topic consumption
• Bronze landing jobs

Cloud Data Platforms (hands-on with at least one)

• Databricks (Delta Lake, Unity Catalog, Auto Loader, DLT)
• AWS (S3, Glue, Lake Formation, Redshift)

Data Quality

• Great Expectations, dbt tests, or equivalent

Docker & Kubernetes

• Containerized jobs
• Kubernetes Jobs and CronJobs
• EKS deployments

Infrastructure as Code

• Terraform familiarity

Data Modeling

• Dimensional modeling
• Slowly Changing Dimensions
• Medallion architecture design

Data Platform-Specific Experience

• SaaS ingestion via Fivetran, Airbyte, or custom connectors (REST, SFTP, JDBC)
• Change Data Capture (Debezium or connector-based CDC modes)
• Handling semi-structured and unstructured data (PDF, Excel, JSON)
• Designing Bronze schemas for fidelity and downstream flexibility
• Vector embedding pipelines (chunking strategies, embeddings, pgvector/OpenSearch)
• Data lineage tooling (OpenLineage, DataHub, Unity Catalog)
• Governance standards: PII handling, sensitivity classification, column masking, partition-level access

Ways of Working

• Agile sprint participation (planning, standups, retros)
• Treat pipelines as production software (tests, runbooks, SLAs, alerts)
• Migration-safe schema changes (expand/contract pattern)
• Proactive data quality ownership
• Rigorous peer code reviews
• Close collaboration with data science and platform engineering teams

Nice to Have

• Migration of legacy data assets (Excel, Access, shared drives)
• ML feature engineering and point-in-time correct pipelines
• Open-source data tooling contributions

Qualifications

• 5–8+ years of experience developing data products and platforms
• Strong background in Agile software development
• Demonstrated success in prior engineering roles
• Strong analytical and problem-solving capabilities

What We’re Looking For

• Clear communicator who can document complex data flows
• Results-driven and impact-focused
• Proactive and self-directed
• Comfortable interfacing across technical and business stakeholders
• Entrepreneurial mindset with a willingness to innovate

If you’re a production-minded data engineer who thrives on building scalable platforms and owning pipelines end-to-end, we’d love to connect.