Aws Databricks Architect -Sr Staff

Job Summary

The Databricks Architect will be responsible for technical decision-making, code quality, and the successful delivery of complex data projects using Delta Lake, Unity Catalog, and Mosaic AI. The Databricks Architect/Lead owns the end-to-end strategy for data ingestion, storage, and analytics and serve as a liaison between business stakeholders and engineering teams, ensuring that the platform is secure, cost-optimized, and capable of supporting advanced AI/ML workloads.

Key Responsibilities

• Lakehouse Design: Architect end-to-end data solutions using the Medallion Architecture (Bronze/Silver) and Delta Lake.
• Platform Governance: Implement and manage Unity Catalog for centralized data governance, fine-grained access control, and lineage tracking.
• Technical Leadership: Lead and mentor cross-functional teams of data engineers; conduct code reviews and define engineering best practices in Python, PySpark, and SQL.
• Optimization Performance: Drive cost-efficiency by tuning Spark jobs, managing cluster policies, and leveraging serverless compute and auto-scaling.
• DevOps DataOps Integration: Integrate Databricks workflows into CI/CD pipelines using tools like Databricks Asset Bundles, Terraform, and Azure DevOps or GitHub.
• AI/ML Support: Provide architectural guidance for integrating machine learning models, including feature engineering and model deployment via MLflow.
• Deep hands-on technical understanding in building complex data models for supporting various analytics needs.
• Comfortable in working with cross-functional teams such as Data Engineering, Analytics, SMEs and Business stakeholders.
• Passion to understand complex existing footprint of data solutions and build out the strategy for migrating to Databricks medallion architecture.
• Accountability and Ownership of the delivery in driving the projects to completion.

Experience

• 14+ years of experience in data management, data engineering, or data science roles, with at least 7 years in building enterprise level data models as part of data modernization and DW migration efforts.
• 8+ years of work experience with Data Structures and algorithms
• 8+ years of work experience with programming (e.g., Java, Python)
• 8+ years of work experience with SQL or NoSQL Databases
• 3+ years of work experience in a role requiring interaction with senior leadership, leading modernization of Data platforms.

Education & Qualifications

• Bachelor's or Master's degree in Computer Science, Data Science, Information Technology, or a related field.
• Strong understanding and hands-on experience with lakehouse medallion data architecture and governance
• Proven implementation experience with migrating legacy DW data models to 3-layer medallion architecture.
• Strong passion in exploring GenAI based solutions for documenting the existing DW footprint and build out strategic data model for Databricks platform
• Experience with machine learning, artificial intelligence, and predictive analytics is a plus.
• Excellent leadership, communication, and collaboration skills in a global hybrid delivery model
• Strong problem-solving skills with strategic thought process and self-initiative and proctive soft skills

Minimum Qualifications

• 7+ years of IT-related work experience with a Bachelor's degree in Computer Engineering, Computer Science, Information Systems or a related field.
• OR 9+ years of IT-related work experience without a Bachelor's degree.
• 5+ years of work experience with programming (e.g., Java, Python).
• 3+ years of work experience with SQL or NoSQL Databases.
• 3+ years of work experience with Data Structures and algorithms.