Databricks Engineer/Lead

• Develop scalable pipelines using Databricks, Apache Spark, PySpark, SQL and Delta Lake.
• Build Bronze, Silver and Gold pipelines following the approved architecture.
• Implement source ingestion, cleansing, transformation, enrichment and conformance.
• Develop reusable pipeline components for multiple sources.
• Implement incremental, historical and schema-evolution processing.
• Build portfolio accounting/custody data products covering holdings, tax lots, transactions, GL, cash, accruals and corporate actions.
• Process benchmark/index data and integrate Security Master and Account Master data.
• Implement enterprise identifier matching and cross-references.
• Implement data-quality validation, reconciliation, exception handling and certification.
• Support up to 10 years of historical backfill with reconciliation and auditability.
• Optimize Spark jobs, Delta tables and SQL workloads.
• Develop unit/integration tests and support SIT, UAT, production deployment and stabilization.
• Contribute to documentation and knowledge transfer.

Required Experience

• 712+ years in data engineering.
• 4+ years hands-on Databricks/Spark experience.
• Strong Databricks, Apache Spark, PySpark, SQL and Delta Lake skills.
• Enterprise-scale batch and/or streaming pipeline experience.
• Strong medallion architecture knowledge.
• Data transformation, cleansing, enrichment and conformance experience.
• Data-quality and reconciliation framework experience.
• Large historical dataset experience.
• Strong debugging and performance-tuning skills.

Financial Services Experience

Strong preference for investment management, asset management, portfolio accounting, custody, ABOR/IBOR, securities, benchmarks/indexes, Security Master or Account Master.

Preferred Databricks Skills

Databricks Jobs/Workflows, Lakeflow/declarative pipelines, Unity Catalog, Databricks SQL, Delta optimization, schema evolution, SCD, lineage, data-quality frameworks, metadata-driven ingestion and large-scale historical backfill.

Out of Scope

Azure infrastructure, VNet/Private Link/networking, Terraform, Azure landing zone, CI/CD platform engineering, Databricks workspace provisioning and infrastructure operations.

Education

Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, Data Engineering or related discipline.