Data Platform Engineer

Data Platform Engineer | Location: Remote | Direct Hire | Hourly Pay Range: $125,000-$150,000 The Data Platform Engineer role focuses on managing and optimizing enterprise data infrastructure built on Databricks and Azure. This senior-level position requires deep technical expertise in platform operations, governance, and supporting AI-enabled data workflows within a cloud environment.

Key Responsibilities

• Administer and optimize Databricks workspace, including upgrades, cluster policies, and Delta table lifecycle management.
• Manage Unity Catalog access controls, governance policies, and data lineage.
• Tune Spark jobs for performance, reliability, and cost efficiency by profiling bottlenecks and optimizing partitioning and Z-ordering.
• Architect and enforce platform standards such as naming conventions, schema evolution policies, and medallion layer contracts.
• Leverage Databricks Mosaic AI and Genie to develop AI-native DataOps capabilities, including pipeline monitoring and anomaly detection.
• Operate the full Azure data services stack: ADLS Gen2, Azure Data Factory, Azure Monitor, Log Analytics, Key Vault, and Event Hub.
• Collaborate with Azure infrastructure teams on networking, security, and resource provisioning.
• Drive cost governance through Azure Cost Management, Databricks DBU optimization, and storage lifecycle policies.
• Support multi-platform data systems, including ADF, Synapse, Cosmos DB, and Azure SQL Server, ensuring access management, storage optimization, and compliance.
• Own CI/CD pipelines for data workflows, promoting changes from development to production with automation, testing, and rollback procedures.
• Manage identity and access via Azure AD, RBAC, and Unity Catalog permissions, partnering with data governance teams.
• Maintain environment parity across Dev, Staging, and Production Databricks environments, including configuration management and infrastructure-as-code scripts.
• Produce technical documentation such as runbooks, deployment playbooks, and incident reports.
• Support on-call rotation and SLA commitments for critical data domains.
• Build and deploy AI-powered platform observability tools, integrating ML models for anomaly detection and self-healing capabilities.
• Define and enforce Delta Lake standards, including schema evolution, vacuum, optimize, and data lifecycle policies.
• Design disaster recovery strategies, including Delta table backups, geo-redundancy, and workspace recovery procedures.

Required Qualifications

• 5+ years of platform or infrastructure engineering experience in a production environment.
• Expert-level Databricks administration: Unity Catalog, cluster policies, workspace management, Delta Lake internals.
• Strong command of Azure data services: ADLS Gen2, ADF, Azure Monitor, Log Analytics, Key Vault, and Event Hub.
• Experience with identity and access management: Azure AD, service principals, RBAC, Unity Catalog governance.
• Hands-on experience with ADF pipeline design and orchestration at scale.
• Spark performance tuning: partitioning, Z-ordering, compute spend optimization, bottleneck profiling.
• Proven CI/CD experience using GitHub Enterprise, including environment promotion, rollback, and release management.
• Proficiency in Python and SQL for automation, validation, and tooling.
• Knowledge of medallion architecture, schema evolution, and multi-environment deployment discipline.
• Infrastructure-as-Code (IaC) tools: Terraform or similar for provisioning Databricks and Azure resources.
• Azure networking: VNet configuration, private endpoints, NSG rules, DNS resolution.
• Web monitoring and analytics: Web server logs, Omniture, Google Analytics.
• Delta Lake internals: VACUUM, OPTIMIZE, ZORDER, time travel, transaction logs.

Preferred Qualifications

• Experience with Synapse Analytics, Cosmos DB, or Azure SQL Server in production.
• Familiarity with Databricks Mosaic AI, Genie, or AI-native observability tools.
• Data lineage and metadata management experience.
• Background in legacy BI migration or platform modernization.
• Experience with Linear for sprint planning and release tracking.

About Delphi-US Delphi-US is a national IT Services firm based in Newport, Rhode Island. Delphi attracts our nation's best and brightest Technical & Professional Talent and positions our nation's finest employers to further their competitive advantages. Delphi's core technical focus includes Data Intelligence Engineering & Analytics, AI/ML Operational Excellence, DevOps & Cloud Computing, Health IT & Life Sciences, and Energy Management/Power Systems; serving domains from sensitive national energy sectors to critical financial services, biomed, life sciences & pharma, major government & municipal institutions, and much of the USA's Fortune 500 corporations. Delphi Associates are deployed on critical IT Projects where their consultative business acumen, comprehensive technical expertise, and committed service mission drives our client's success. Our proprietary skill-based and cultural matching process results in higher qualified project teams, custom curated for our client's specific technical & functional goals. We are supremely experienced, friendly, professional, and ready to advocate on your behalf. Delphi has an indelible understanding of employer expectations, a track record of project success and we deliver lasting results.

6394