Data Engineering Specialist (Databricks)

Accenture helps the world’s leading enterprises reinvent by building their digital core and unleashing the power of AI to create value at speed for organizations across industries. Our strategy is to be the reinvention partner of choice for our clients and lead in the safe, widespread adoption of AI, and to be the most client-focused, AI-enabled, great place to work in the world. We bring together the talent of our approximately 799,000 people with proprietary assets and platforms, deep process and industry expertise, and leading ecosystem relationships to deliver end-to-end solutions and measurable outcomes at scale. Through our Reinvention Services, we offer broad expertise across Cybersecurity, Digital Core, Finance, Industry and Enterprise, Song, Supply Chain and Engineering, and Talent, with advanced capabilities in AI and Data, Industry and Process, and Technology. We serve approximately 9,000 clients and generated approximately $70 billion in FY25 revenue. Visit us at accenture.com.

Role overview

Lead a delivery workstream within a client data engineering team migrating data from legacy warehouses to an Azure Databricks lakehouse. The role owns technical design, production-grade pipeline delivery, engineering standards, and coordination of a distributed engineering team. It also translates business requirements into technical solutions and communicates delivery trade-offs to client stakeholders.

Key responsibilities

• Lead a migration workstream from requirements and technical specifications through production deployment and stabilization.
• Design and build scalable batch and streaming pipelines using PySpark, Databricks SQL, and Lakeflow Jobs and Declarative Pipelines, including cleansing, validation, monitoring, and logging.
• Design Delta Lake medallion layers (bronze, silver, and gold) and dimensional models using fact and dimension tables.
• Optimize Databricks workloads for performance and cost through compute sizing, partitioning, liquid clustering, and shuffle optimization.
• Implement governance, security, access controls, and data quality standards using Unity Catalog.
• Plan migrations from legacy warehouses and ETL tools, including data reconciliation and cutover validation.
• Create and groom backlog items, estimate effort, track sprint delivery, and manage workstream risks, blockers, and dependencies.
• Review code and establish reusable engineering patterns, CI/CD practices, and documentation standards.
• Guide engineers across India, Canada, and Costa Rica and coordinate priorities and handoffs across time zones.
• Present technical options, including cost, performance, and timeline implications, to client stakeholders.

Required qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent relevant professional experience.
• 5+ years of data engineering experience, including 3+ years of hands-on production experience with Databricks and Apache Spark.
• Hands-on experience with Databricks workspace and compute management, Delta Lake, Unity Catalog, Lakeflow Jobs and Pipelines, Databricks SQL, notebooks, and Git folders.
• Advanced PySpark, including the DataFrame API and performance tuning, plus advanced SQL.
• Production experience designing dimensional models, including star schemas with fact and dimension tables.
• Experience migrating data from legacy data warehouses to a cloud data platform.
• Experience with Azure Data Factory and Azure Data Lake Storage Gen2.
• Python experience for data processing and automation.
• Working knowledge of data governance, data quality, data warehousing, and data architecture principles.
• Demonstrated experience leading a technical workstream, reviewing code, coordinating delivery, and guiding engineers with minimal supervision.
• Ability to explain technical trade-offs and summarize recommendations for client and senior stakeholders.

Preferred qualifications

• Databricks Certified Data Engineer Associate or Professional.
• Microsoft Certified: Azure Databricks Data Engineer Associate.
• Experience with IBM DataStage and SQL Server as legacy source systems.
• Experience with CI/CD for data pipelines using Databricks Asset Bundles, Git, or Azure DevOps.
• Experience with streaming ingestion using Structured Streaming or Auto Loader.
• Exposure to AWS, Google Cloud, or PLX.
• Client-facing consulting experience in digital analytics.

What we offer

• Health and Life Insurance
• Accenture days, 3 additional vacation days
• On site doctor
• Birthday leave
• Internet reimbursement