ONE-Consultants

Position Summary
This role is for professionals seeking a long-term career with us; subcontractor applications will not be considered.
We are seeking an experienced&nbsp;
Senior Data Engineer
&nbsp;to design, build, and optimize modern
data platforms using the&nbsp;
Databricks Lakehouse Platform
. The ideal candidate will have&nbsp;
10+
years of data engineering experience
&nbsp;and deep expertise in&nbsp;
Lakehouse
Architecture
,&nbsp;
Medallion Architecture
, cloud-native data solutions, and enterprise-scale data
governance practices.
This role will be responsible for architecting scalable data solutions, implementing robust data
pipelines, enforcing data quality standards, and enabling analytics and AI workloads across the
organization. The successful candidate will collaborate closely with business stakeholders, data
architects, analytics teams, and platform engineers to deliver high-quality, secure, and reliable
data products.
Key Responsibilities
Data Platform Architecture &amp; Engineering
<ul>
<li>Design and implement scalable, high-performance data platforms using the Databricks Lakehouse architecture.</li>
<li>Lead the development of enterprise data solutions leveraging Bronze, Silver, and Gold layers following Medallion Architecture principles.</li>
<li>Build and maintain batch and streaming data pipelines for large-scale data processing.</li>
<li>Develop reusable data frameworks, accelerators, and engineering best practices.</li>
<li>Optimize data storage, processing, and query performance across cloud environments.</li>
</ul>
Databricks Development
<ul>
<li>Design, develop, and maintain Databricks workflows using PySpark, SQL, and Python.</li>
<li>Implement and manage Delta Live Tables (DLT) pipelines.</li>
<li>Utilize Unity Catalog for centralized governance, metadata management, access controls, and lineage tracking.</li>
<li>Support advanced analytics, machine learning, and AI workloads on the Databricks platform.</li>
<li>Monitor and troubleshoot production data pipelines and platform performance.</li>
</ul>
Data Governance &amp; Quality
<ul>
<li>Establish and enforce enterprise data governance standards and policies.</li>
<li>Implement data quality frameworks, validation rules, reconciliation processes, and monitoring solutions.</li>
<li>Ensure compliance with security, privacy, regulatory, and audit requirements.</li>
<li>Drive metadata management, lineage, cataloging, and data stewardship initiatives.</li>
<li>Collaborate with business and governance teams to improve trust and usability of enterprise data assets.</li>
</ul>
Cloud Engineering
<ul>
<li>Design and deploy cloud-native data solutions on Azure, AWS, or GCP.</li>
<li>Implement scalable and secure cloud architectures aligned with enterprise standards.</li>
<li>Optimize cloud resource utilization, performance, and cost management.</li>
<li>Integrate cloud-native services with Databricks-based solutions.</li>
</ul>
DevOps &amp; Automation
<ul>
<li>Implement CI/CD pipelines for data engineering workloads.</li>
<li>Automate deployment, testing, monitoring, and operational processes.</li>
<li>Apply Infrastructure as Code (IaC) practices using tools such as Terraform or similar.</li>
<li>Promote engineering excellence through code reviews, testing strategies, and release management processes.</li>
</ul>
Leadership &amp; Collaboration
<ul>
<li>Provide technical leadership and mentorship to junior and mid-level engineers.</li>
<li>Collaborate with architects, analysts, data scientists, product owners, and business stakeholders.</li>
<li>Contribute to platform roadmaps, architectural decisions, and technology evaluations.</li>
<li>Lead technical discussions and establish best practices across the data engineering organization.</li>
</ul>
Required Qualifications
<ul>
<li>Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related field.</li>
<li>10+ years of experience
&nbsp;in Data Engineering, Data Warehousing, or Big Data platforms.</li>
<li>5+ years of hands-on experience with Databricks
&nbsp;in enterprise environments.</li>
<li>Strong expertise in:</li>
</ul>
<ol>
<li>Lakehouse Architecture</li>
<li>Medallion Architecture</li>
<li>Delta Lake</li>
<li>Delta Live Tables (DLT)</li>
<li>Unity Catalog</li>
<li>Data Governance and Data Quality frameworks</li>
<li>Data Modeling and Data Warehousing concepts</li>
</ol>
<ul>
<li>Extensive experience with at least one major cloud platform:</li>
</ul>
<ol>
<li>Microsoft Azure</li>
<li>Amazon Web Services (AWS)</li>
<li>Google Cloud Platform (GCP)</li>
</ol>
<ul>
<li>Advanced programming experience with:</li>
</ul>
<ol>
<li>Python</li>
<li>SQL</li>
<li>PySpark</li>
</ol>
<ul>
<li>Strong understanding of CI/CD methodologies and DevOps practices.</li>
<li>Experience implementing Infrastructure as Code (Terraform).</li>
<li>Expertise in performance tuning, optimization, and troubleshooting of large-scale data pipelines.</li>
<li>Strong understanding of security, access controls, and compliance requirements in cloud environments.</li>
</ul>
Preferred Qualifications
<ul>
<li>Databricks certifications such as 'Databricks Certified Data Engineer Professional'</li>
<li>Cloud certifications (Azure, AWS, or GCP).</li>
<li>Experience with real-time streaming technologies and event-driven architectures.</li>
<li>Experience supporting AI/ML and Generative AI data platforms.</li>
<li>Familiarity with data observability tools and modern data quality frameworks.</li>
<li>Experience implementing enterprise metadata management and data lineage solutions.</li>
</ul>
Hiring Process
We aim to complete the interview process in October-November. The targeted onboarding is January 2027 or earlier.

Senior Data Engineer – Databricks Lakehouse Platform

Senior Data Engineer – Databricks Lakehouse Platform

This job is no longer available

Talents

Recruiters

About