Principal Databricks Engineer

Position Title: Principal Databricks Engineer

Location: Remote

Position Status: Full Time – Direct Hire

Shape the Future of Our Enterprise Data Platform

We are seeking a Principal Databricks Engineer to become the technical owner and subject matter expert for our enterprise Databricks platform.

This is a rare opportunity to join at the beginning of a major transformation initiative and help build the foundation of a modern data and analytics ecosystem. As the first dedicated Databricks engineering hire, you will play a pivotal role in establishing platform standards, guiding architectural decisions, and ensuring the successful adoption of Databricks across the organization.

You'll work closely with leadership, data engineers, cloud teams, and an external implementation partner to evolve the platform from initial deployment through long-term enterprise ownership.

This role is ideal for someone who enjoys solving complex platform challenges, influencing technical direction, and taking ownership of a high-impact, enterprise-scale environment.

What You'll Do

Databricks Platform Ownership

• Serve as the primary technical expert for the enterprise Databricks environment.
• Own platform administration, configuration, architecture, governance, and operational support.
• Help define the long-term vision and roadmap for Databricks adoption across the organization.
• Partner with an external professional services team during implementation and gradually assume full platform ownership.

Architecture & Engineering

• Design scalable, secure, and maintainable Databricks environments.
• Establish standards for workspaces, compute resources, jobs, workflows, permissions, and integrations.
• Develop deployment, monitoring, and operational best practices.
• Support performance tuning, optimization, reliability, and cost management initiatives.
• Collaborate with cloud, infrastructure, and security teams to ensure enterprise-level scalability and compliance.

Governance & Best Practices

• Implement governance and access control frameworks using Unity Catalog.
• Establish security, data access, and platform administration standards.
• Create documentation, architecture guidance, and reusable patterns for engineering teams.
• Serve as a trusted advisor to leadership on Databricks capabilities and platform strategy.

Technical Leadership

• Lead technical discussions and architectural decision-making.
• Mentor team members and promote engineering best practices.
• Evaluate new Databricks capabilities and determine their applicability to business needs.
• Help drive adoption of modern data platform technologies across the enterprise.

Required Qualifications

Databricks Expertise

• Extensive hands-on experience administering and supporting Databricks in enterprise production environments.
• Strong knowledge of Databricks platform architecture, configuration, security, and governance.
• Experience implementing and managing Databricks workspaces, clusters, jobs, workflows, and integrations.
• Experience establishing platform standards, operational procedures, and engineering best practices.
• Ability to serve as an internal technical authority and trusted advisor for Databricks-related decisions.

Cloud & Data Engineering

• Strong experience with Azure and cloud-based data platforms.
• Advanced Python and SQL skills.
• Experience deploying and supporting modern data engineering solutions.
• Familiarity with CI/CD practices, infrastructure automation, monitoring, and operational support.
• Understanding of data governance, security, and enterprise architecture principles.

Leadership & Communication

• Proven ability to lead technical initiatives and influence architecture decisions.
• Strong communication and stakeholder management skills.
• Experience working across engineering, infrastructure, analytics, and business teams.
• Comfortable operating as a hands-on technical leader without direct people management responsibilities.

Preferred Qualifications

While not required, experience in the following areas is highly desirable:

• Databricks AI/BI Genie
• Databricks Mosaic AI
• AI agents and agent orchestration
• Generative AI applications
• Large Language Models (LLMs)
• Retrieval-Augmented Generation (RAG)
• Conversational analytics and AI-powered data experiences

Note: Deep Databricks platform expertise is significantly more important than advanced AI development experience. We are seeking a platform leader first and foremost.

Ideal Background

We're especially interested in professionals who have worked for:

• Databricks consulting organizations
• Cloud and data modernization consultancies
• Professional services firms implementing Databricks across multiple clients
• Enterprise organizations where they owned or led Databricks platform strategy

About Seneca Resources:

At Seneca Resources, we are more than just a staffing and consulting firm, we are a trusted career partner. With offices across the U.S. and clients ranging from Fortune 500 companies to government organizations, we provide opportunities that help professionals grow their careers while making an impact.

When you work with Seneca, you’re choosing a company that invests in your success, celebrates your achievements, and connects you to meaningful work with leading organizations nationwide. We take the time to understand your goals and match you with roles that align with your skills and career path. Our consultants and contractors enjoy competitive pay, comprehensive health, dental, and vision coverage, 401(k) retirement plans, and the support of a dedicated team who will advocate for you every step of the way.