Senior Data Engineer

Job Title: Lead / Principal Databricks Subject Matter Expert

Location: Dallas, TX

Work Arrangement: Onsite, 5 days per week

Experience Required: 10+ years in Data Engineering, Analytics, or Platform Architecture

Job Summary

BuzzClan is seeking a highly experienced Lead / Principal Databricks Subject Matter Expert to lead enterprise-level data engineering, cloud architecture, and data transformation initiatives. The ideal candidate will serve as a technical leader in designing, implementing, and optimizing scalable solutions using the Databricks Lakehouse Platform. This role requires expertise in data architecture, Apache Spark, Delta Lake, cloud technologies, machine learning, and MLOps, along with strong leadership, client advisory, and stakeholder management skills.

The successful candidate will collaborate with senior technology executives, data leaders, and engineering teams to define data strategies, develop scalable platform architectures, oversee production deployments, and establish best practices for data governance, quality, security, and performance.

Key Responsibilities

1. Solution Architecture and Design

• Lead the end-to-end design and implementation of Databricks Lakehouse architectures, incorporating Delta Lake, Unity Catalog, and medallion architecture models, including Bronze, Silver, and Gold layers.
• Define enterprise data architecture standards, reference architectures, technical frameworks, and reusable solution accelerators.
• Evaluate and recommend appropriate deployment strategies across Azure Databricks, AWS Databricks, and Google Cloud Platform (GCP).
• Align cloud architecture decisions with organizational infrastructure, security, scalability, performance, and data governance requirements.
• Develop architectural roadmaps and technical strategies to support complex enterprise data transformation initiatives.
• Establish best practices for platform design, data integration, workload management, and infrastructure optimization.

2. Data Engineering and Pipeline Development

• Architect, develop, and oversee large-scale ETL/ELT pipelines using Apache Spark, PySpark, Scala, Delta Live Tables, and Databricks Workflows.
• Design and implement batch and real-time data ingestion frameworks for structured, semi-structured, and unstructured data sources.
• Develop scalable data processing solutions that support enterprise analytics, reporting, and business intelligence requirements.
• Establish data quality, data lineage, monitoring, and observability standards using Unity Catalog, Delta Lake capabilities, and complementary tools.
• Optimize data pipelines for reliability, scalability, processing efficiency, and cost-effectiveness.
• Troubleshoot complex data engineering issues and implement solutions to improve pipeline performance and operational stability.
• Ensure data processing workflows follow established standards for security, governance, and data integrity.

3. Machine Learning and MLOps

• Guide the implementation of end-to-end machine learning workflows using MLflow, Feature Store, and Databricks Model Serving.
• Design and recommend MLOps frameworks supporting model development, deployment, versioning, monitoring, and lifecycle management.
• Implement and advise on A/B testing, automated model retraining, model performance monitoring, and deployment strategies.
• Collaborate with data scientists, machine learning engineers, and application development teams to operationalize machine learning models.
• Support the integration of machine learning outputs into enterprise applications and business processes.
• Establish best practices for scalable, reliable, and maintainable machine learning operations in production environments.

4. Data Governance, Security, and Platform Optimization

• Implement and oversee data governance frameworks using Unity Catalog, including access controls, permissions, and data lineage tracking.
• Establish standards for data quality, data discoverability, security, and compliance with organizational requirements.
• Recommend platform optimization strategies to improve query performance, resource utilization, and processing efficiency.
• Evaluate Databricks capabilities, including the Photon engine, to enhance workload performance.
• Support secure and scalable data platform deployments across cloud environments.
• Develop standards for platform monitoring, operational support, and continuous improvement.

5. Client Advisory and Engagement Leadership

• Serve as a trusted technical advisor to senior stakeholders, including CTOs, VPs of Data, and Engineering Leads, on data platform strategy and Databricks adoption.
• Lead technical discovery sessions, architecture workshops, platform assessments, and solution design reviews.
• Translate complex technical concepts into clear recommendations for executive and nontechnical audiences.
• Develop data modernization roadmaps, architectural recommendations, and implementation strategies aligned with business objectives.
• Manage multiple concurrent technical engagements while maintaining delivery quality, timelines, and stakeholder expectations.
• Identify technical risks, recommend mitigation strategies, and resolve complex architectural and implementation challenges.
• Support presales activities by preparing technical proposals, estimating project effort, defining solution approaches, and delivering technical demonstrations.

6. Technical Leadership and Team Mentoring

• Provide technical direction and architectural guidance to data engineers, architects, consultants, and delivery teams.
• Mentor team members on Databricks, Apache Spark, Delta Lake, cloud data engineering, and enterprise architecture best practices.
• Conduct technical reviews to ensure solution quality, consistency, scalability, and maintainability.
• Promote knowledge sharing, technical collaboration, and continuous professional development across data engineering teams.
• Help establish reusable development standards, implementation frameworks, and engineering methodologies.
• Guide teams through complex implementation challenges and support successful production deployments.

7. Thought Leadership and Practice Development

• Develop technical frameworks, reference materials, white papers, training curricula, and reusable intellectual property related to Databricks and modern data platforms.
• Contribute to the development and growth of data engineering and cloud data architecture capabilities.
• Promote best practices and emerging technologies within the data and analytics practice.
• Represent the organization at industry conferences, technical events, partner summits, and webinars.
• Stay informed about new Databricks features, cloud technologies, data engineering methodologies, and machine learning capabilities.
• Identify opportunities to improve service offerings, strengthen technical expertise, and deliver innovative data solutions.

Required Qualifications

• 10+ years of professional experience in data engineering, analytics, or data platform architecture.
• 5+ years of hands-on experience with Databricks, including enterprise-scale production implementations.
• Expert-level proficiency in Apache Spark, PySpark and/or Scala, Delta Lake, and SQL.
• Proven experience designing and implementing cloud-based Lakehouse architectures using Azure, AWS, or GCP.
• Strong experience with Unity Catalog for data governance, access control, and data lineage management.
• Demonstrated experience leading technical engagements and communicating complex architectural concepts to executive stakeholders.
• Experience working in consulting, professional services, or enterprise technology environments.
• Ability to manage multiple projects or engagements simultaneously.
• Bachelor's degree in Computer Science, Engineering, Data Science, or a related technical discipline, or equivalent practical experience.
• Strong analytical, problem-solving, communication, collaboration, and technical leadership skills.
• Ability to work onsite in Dallas, TX, five days per week.

Preferred Qualifications

• Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional certification.
• Hands-on experience with Delta Live Tables, Databricks Workflows, and Photon engine optimization.
• Familiarity with dbt and infrastructure-as-code tools such as Terraform or Pulumi.
• Experience implementing CI/CD pipelines for data platforms.
• Production experience with MLflow, Feature Store, and Databricks Model Serving.
• Previous experience supporting a Databricks-aligned consulting practice or technology partner ecosystem.
• Master's degree or Ph.D. in Computer Science, Data Science, Engineering, or another quantitative or technical discipline.

Core Technical Skills

• Data Platform: Databricks Lakehouse Platform, Delta Lake, Unity Catalog, Medallion Architecture
• Programming and Processing: Apache Spark, PySpark, Scala, SQL
• Data Engineering: ETL/ELT, Batch Processing, Real-Time Data Ingestion, Delta Live Tables, Databricks Workflows
• Cloud Platforms: Microsoft Azure, AWS, Google Cloud Platform
• Machine Learning and MLOps: MLflow, Feature Store, Model Serving, Model Monitoring, Automated Retraining
• Governance and Security: Data Lineage, Access Control, Data Quality, Data Observability
• DevOps and Automation: CI/CD, Terraform, Pulumi, dbt
• Leadership: Solution Architecture, Technical Consulting, Stakeholder Management, Team Mentoring, Presales Support

Work Environment

This is a senior-level technical leadership position based onsite in Dallas, Texas, five days per week. The role combines hands-on technical architecture, enterprise data engineering, cloud platform strategy, machine learning enablement, client advisory, and team leadership.

BuzzClan is committed to fostering a professional and inclusive work environment and providing equal employment opportunities to qualified candidates. Reasonable accommodations will be considered in accordance with applicable laws.