Databricks Architect

Role: Senior Data-bricks Architect

Location: REMOTE

Domain- Retail / Grocery
Work Type: Contract
Experience: 10+ years overall IT experience, with 5+ years in Databricks / Cloud Data Engineering / Data Architecture

Role Overview

We are looking for an experienced Senior Databricks Architect to lead the architecture and technical execution of a large-scale migration for a leading Retail/Grocery client.

The architect will be responsible for designing the target Databricks Lakehouse architecture, defining migration patterns, guiding the data engineering team, and ensuring successful modernization of existing Talend-based ETL/ELT workloads onto Databricks.

The role requires strong hands-on expertise in Databricks, Apache Spark, SQL, cloud data platforms, data architecture, ETL modernization, and migration strategy, along with a good understanding of Retail/Grocery business processes and data domains.

Key Responsibilities

Architecture & Solution Design

· Lead the end-to-end architecture for migration of Talend ETL/ELT workloads to Databricks.

· Define the target-state Databricks Lakehouse architecture, including data ingestion, transformation, storage, orchestration, governance, and consumption layers.

· Design scalable, secure, highly available, and cost-optimized data solutions.

· Define architecture standards, design patterns, coding standards, and best practices for Databricks development.

· Establish migration frameworks and reusable patterns for converting Talend jobs into Databricks/Spark-based pipelines.

· Evaluate existing Talend jobs and determine appropriate migration strategies, including re-platforming, re-engineering, consolidation, or retirement.

· Provide technical leadership for complex data pipelines and integration scenarios.

Talend to Databricks Migration

· Analyze existing Talend workflows, jobs, mappings, dependencies, schedules, and data transformations.

· Develop migration strategies for batch and incremental data processing workloads.

· Translate Talend transformations and business rules into PySpark/SQL/Databricks implementations.

· Identify opportunities to simplify and optimize legacy ETL processes during migration.

· Define data reconciliation and validation strategies to ensure parity between Talend and Databricks outputs.

· Establish migration sequencing based on business criticality, dependencies, complexity, and risk.

· Support migration of high-volume and business-critical data pipelines.

Databricks & Data Engineering
· Design and implement solutions using Databricks, Apache Spark, Delta Lake, PySpark, and SQL.

· Design Medallion Architecture (Bronze, Silver, Gold) and appropriate data processing patterns.

· Implement incremental processing, CDC, SCD Type 1/Type 2, data quality, error handling, and audit frameworks.

· Optimize Spark workloads, Delta tables, SQL queries, partitioning, clustering, and job execution.

· Leverage Delta Live Tables / Lakeflow capabilities, Workflows, Unity Catalog, and other Databricks platform services where appropriate.

· Design data pipelines integrating structured and semi-structured data from databases, files, APIs, and enterprise applications.

· Establish monitoring, logging, alerting, operational support, and performance-management patterns.

Cloud & Integration

· Strong experience with at least one major cloud platform, preferably Azure or AWS.

· Design integration between Databricks and cloud storage, databases, messaging platforms, APIs, and enterprise applications.

· Experience with technologies such as ADLS/S3, Azure Data Factory, Kafka, REST APIs, relational databases, and cloud-native services is highly desirable.

· Define secure connectivity, networking, secrets management, and access-control patterns.

Retail / Grocery Domain

· Work closely with business and technology stakeholders to understand Retail/Grocery data requirements.

· Experience with retail data domains such as:

o Product / Item

o Store / Location

o Customer / Loyalty

o Sales / Transactions

o Pricing

o Promotions

o Inventory

o Supply Chain

o Vendors / Suppliers

o Orders

o Forecasting

o Merchandising

· Understand retail-specific challenges such as high-volume transaction processing, store-level data, product hierarchies, promotions, pricing, inventory movements, and customer analytics.

· Translate business requirements into scalable data architecture and technical solutions.

Technical Leadership

· Act as the senior technical authority for the Databricks migration program.

· Provide technical direction and mentoring to data engineers and developers.

· Conduct architecture/design reviews and code reviews.

· Troubleshoot complex production and performance issues.

· Collaborate with offshore/onshore teams, enterprise architects, infrastructure teams, security teams, and business stakeholders.

· Prepare architecture diagrams, technical design documents, migration plans, and implementation standards.

· Communicate technical risks, dependencies, and recommendations effectively to client leadership.

Required Technical Skills

Databricks

· Strong hands-on experience with Databricks.

· Apache Spark / PySpark.

· Delta Lake.

· Databricks Workflows/Jobs.

· Unity Catalog.

· Databricks SQL.

· Performance tuning and optimization.

· Medallion/Lakehouse architecture.

· Experience with modern Databricks data engineering capabilities.

Data Engineering

· Strong SQL and data modeling skills.

· ETL/ELT architecture.

· Batch and incremental data processing.

· CDC and SCD implementation.

· Data quality and reconciliation frameworks.

· Large-volume data processing.

Pay: $70.00 - $90.00 per hour

Work Location: Remote