Shivam Choudhary

– Spearheaded the migration of 2,600+ data objects to Databricks Unity Catalog with zero disruption to global operations or downstream Azure Data Factory pipelines, managing a team of 11 data engineers. – Achieved 100% platform availability during high-risk cutovers for multi-petabyte Marketing and Supply Chain. – Reduced data discovery time by 30% by establishing a centralized metadata framework and lineage tracking in Azure Databricks. – Designed the phased migration blueprint, including rollback strategies and comprehensive dependency mapping. – Defined global standards for data cataloging and lineage to ensure a fully governed data environment. – Overheaded the transition from legacy Hive metastores to Unity Catalog, eliminating significant technical debt.

– Served as a Core Architect for Global Data Platforms, providing design authority for a petabyte-scale Azure Data Lake serving Finance, R&D, and Supply Chain. – Delivered a 25% reduction in cloud compute costs by re-architecting and optimizing high-latency complex Spark jobs in Azure Databricks within global pipelines. – Engineered an Executive Governance PowerBI Dashboard used by VPs and C-suite leaders for real-time visibility into project health and budget utilization. – Boosted system performance by 35% by implementing a centralized observability framework that eliminated processing bottlenecks. – Partnered with global business owners to deconstruct requirements into precise, scalable technical specifications into actionable Jira epics/tasks for the development team, serving as the primary technical bridge. – Conducted rigorous Pyspark code reviews and pair-programming sessions to help resolve deep-seated technical debt. – Facilitated architecture and design workshops with UK-based business, analytics, and IT stakeholders.

– Led a team of 12 data engineers delivering a large-scale Azure-based enterprise data lake for Unilever using Databricks, DataFactory, Python, Spark and SQL. – Managed production release coordination across 16+ teams, ensuring stable and on-time deployments. – Engineered a Strategic Data Lineage & Usage Platform using PowerBI, Databricks & Storage Logs, providing visibility into dataset consumption across global projects, the first Internal Chargeback Model to attribute cloud compute costs to specific business units. – Partnered with business stakeholders to translate requirements into scalable data solutions. – Owned Jira sprint planning, workload allocation, technical reviews, hiring candidates and delivery governance. – Provided technical leadership and mentoring for production releases and incident resolution.

– Built complex Azure Data Factory ingestion pipelines integrating data from global finance and supply chain applications. – Developed monitoring and observability accelerators adopted as part of Capgemini's flagship 890 Platform. – Delivered Azure Data Engineering classroom training to 60+ engineers across levels. – Developed scalable ETL frameworks using ADF, Databricks, Spark, SQL. – Implemented operational monitoring solutions for data pipelines using Azure Log Analytics and PowerBI.

– Senior developer for a flagship Azure project to develop E2E finance platform development using Azure Databricks, Datafactory and Pyspark. – Developed an automated testing framework to bring down manual testing efforts in SQL and PySpark. – Developed data quality accelerator as part of Azure Databricks COE. The accelerator was chosen as part of the company's key offering (Azure MX) and presented to Global VP of Insights and Data and Azure Ignite NZ.

– Build supply chain analytics platform using Cloudera Hadoop, Shell Scripting, SQL, Power BI.