Sr. Data Architect

Job highlights

• . MUST HAVE. Azure Cloud Architecture (Expert Level - Required)
• . Advanced PySpark . (Expert Level - Required)
• Expertise (Expert Level - Required)
• . PostgreSQL . (Required)
• Performance & Scalability (Expert Level - Required)
• . GOOD TO HAVE

DETAILED JOB DESCRIPTION

Roles & Responsibilities

• 17 to 22 years of experience in cloud-based systems and data platforms management with 8 to 10 years of experience as a Data Architect.

• Design, develop, and implement robust and scalable data architectures on Azure, considering performance, cost-efficiency, and security.

• Identify performance bottlenecks, analyse system behaviour under load, and develop strategies to improve efficiency and scalability.

• Provide guidance on system design and architecture to optimize performance, including database configurations, caching mechanisms, and code optimization.

• Develop and maintain scalable and efficient streaming frameworks.

• Integrate Salesforce solutions with other system components to meet business requirements.

• Utilize scripting languages such as Python for automation and data processing tasks.

• Collaborate with cross-functional teams, including development, operations, and security, to ensure effective cloud architecture and integration.

• Manage a team of 6 to 8 members with below skills.

Detailed Job Description

MUST HAVE

Azure Cloud Architecture (Expert Level - Required)

Advanced PySpark (Expert Level - Required)

• Mastery of DataFrame APIs and Spark SQL

• Performance tuning: broadcast joins, caching strategies, shuffle reduction

• Handling skewed data and optimizing wide transformations

• Writing scalable, testable PySpark code with modular design

Databricks Expertise (Expert Level - Required)

• Cluster sizing, autoscaling, and spot instance optimization

• Job orchestration using Workflows with retries, alerts, and dependencies

• Delta Lake:

• OPTIMIZE with Z-Ordering

• MERGE operations and schema evolution

• Time travel and versioning

• Use of Unity Catalog for data governance and access control

• Monitoring via Databricks metrics and audit logs

PostgreSQL (Required)

• Query profiling and execution plan analysis

• Indexing strategies for analytical workloads

• Table partitioning and parallel query execution

Performance & Scalability (Expert Level - Required)

• Designing horizontally scalable pipelines

• Spark memory management: executor sizing, GC tuning

• Fault tolerance and retry logic in distributed systems

• Monitoring pipeline health and latency metrics

GOOD TO HAVE

Kafka (Nice to have)

• Designing high-throughput Kafka consumers with Spark Structured Streaming

• Managing consumer groups and offset strategies

• Handling schema evolution with Avro/Protobuf and Schema Registry

• Backpressure handling and watermarking in Spark

Structured Streaming in Spark (Nice to have)

• Stateful aggregations and windowed operations

• Exactly-once semantics with checkpointing and write-ahead logs

• Integration with Delta Lake for streaming upserts

Machine Learning & MLOps (Nice to have)

• MLflow integration for model tracking and deployment

• Deploying Spark ML pipelines

• Feature store design and versioning

• Unity Catalog

Node.js - (Nice to have)

• Building REST APIs for data access and control

• Integrating with PostgreSQL

• Typescript Node and/or React experience