Balaji Pothuganti
Data Engineer | Azure · Databricks · Microsoft Fabric | ETL/ELT · PySpark · Delta Lake · Medallion Architecture | DP-700 · PL-300 · Databricks Certified | Open to Work
Salary / Daily rate
Romford, UK
Freelance, open to permanent contract
Skills
About me
Intro
I'm a certified data engineer with 4+ years of end-to-end experience building scalable data pipelines, Lakehouse architectures, and cloud analytics platforms across Azure, Databricks, and Microsoft Fabric. At MDM IT Solutions (client: Saga Insurance), I've delivered data engineering solutions across CRM, financial compliance, and BI domains — processing 500GB+ daily datasets, automating ETL/ELT pipelines, and enabling real-time and batch analytics at enterprise scale. Core expertise: - Azure Data Factory (ADF v1/v2), Synapse Analytics, Microsoft Fabric, ADLS Gen2, OneLake - Databricks (PySpark, Delta Lake, Autoloader, Unity Catalog, MLflow) - Medallion Architecture (Bronze–Silver–Gold), schema evolution, Z-ordering, partitioning - Real-time analytics: Eventstream, KQL, streaming pipelines in Fabric and Power BI - Data governance: Unity Catalog, RBAC, IAM, Azure Purview, Key Vault - CI/CD automation via Azure DevOps and GitHub Actions - Power BI (DAX, RLS, semantic layer, executive dashboards) Measurable results I've delivered: - 45% reduction in ETL execution time processing 500GB daily datasets - 50% faster new data source onboarding via metadata-driven ADF frameworks - 70% improvement in report accuracy and refresh performance (Microsoft Fabric Lakehouse) - 48% reduction in large batch runtime during month-end closing cycles - 44% reduction in manual audit preparation through governed Power BI dashboards - 21% infrastructure cost saving through Databricks cluster and autoscaling optimisation - 65% query performance improvement using Z-ordering, caching, and schema enforcement - 30% reduction in CI/CD release rollback incidents via Azure DevOps automation Certifications: - Microsoft Fabric Data Engineer Associate (DP-700) - Power BI Data Analyst Associate (PL-300) - Databricks Certified Data Engineer Associate - MSc in Data Science — University of Hertfordshire (2025) Based in London | Available immediately | Open to roles UK-wide (relocation: yes) Graduate Visa valid until July 2027 Actively seeking: Data Engineer | Senior Data Engineer | Azure Data Engineer | Fabric Data Engineer Open to connecting with hiring managers, technical leads, and fellow data professionals.
Work experience
Azure Data Engineer
MDM IT Solutions
Mar 2025 - Present
1 year 8 months
• Collaborated with cross-functional stakeholders to define data platform requirements across ingestion, transformation, modelling, and reporting use cases. • Migrated workloads from Synapse to Microsoft Fabric, simplifying the data estate and reducing platform fragmentation. • Designed medallion architecture (bronze, silver, gold) within a unified OneLake lakehouse. • Re-engineered legacy SQL ETL into Spark-based pipelines, reducing daily processing time by 45% (~500GB/day). • Implemented Spark-based data transformations and pipelines in Fabric to process large datasets efficiently. • Developed metadata-driven pipelines using parameterised frameworks, reducing onboarding effort by 50%. • Built curated datasets and semantic models in Fabric for Power BI reporting. • Improved pipeline reliability through enhanced monitoring, retry logic, and workflow optimisation. • Managed Fabric capacity utilisation, balancing workloads for performance and cost control. • Implemented layered access control using workspace roles and item-level permissions. • Integrated Dataverse (Power Apps) as an operational data source for business workflows. • Built rule-based approval workflows using Power Automate / Logic Apps. • Enabled end-to-end visibility of approval processes, improving turnaround time.
MDM IT Solutions
Jul 2019 - Dec 2022
3 years 6 months
Azure Data Engineer
Jul 2021 - Dec 2022
1 year 6 months
• Designed Delta Lake architecture (bronze, silver, gold) in Azure Databricks. • Built modular PySpark transformation frameworks for scalable data processing. • Developed ADF pipelines to orchestrate Databricks jobs and workflows. • Implemented Databricks Workflows for end-to-end pipeline scheduling. • Optimised cluster usage (job vs all-purpose, autoscaling), reducing cost. • Used Databricks SQL (serverless) for BI workloads, separating compute for BI and ETL to improve cost efficiency. • Implemented incremental loads (MERGE, CDC), improving performance. • Tuned Spark performance (shuffle partitions, broadcast joins, caching). • Built data quality frameworks using PySpark and Python. • Integrated Logic Apps and Power Apps for workflow automation. • Optimised Delta tables (partitioning, OPTIMIZE, ZORDER). • Delivered curated datasets using star schema models for Power BI. • Implemented governance using Unity Catalog.
Data Engineer
Jul 2019 - Jul 2021
2 years 1 month
• Designed dimensional models (star/snowflake) for enterprise reporting. • Built ETL pipelines using SSIS integrating SAP and operational systems. • Developed complex SQL transformations (CTEs, window functions, joins). • Applied indexing and query optimisation techniques to improve performance. • Implemented surrogate keys and SCD frameworks for data consistency. • Built Power BI reports using Power Query and optimised data models. • Created DAX measures for KPIs, time intelligence, and analytics. • Designed interactive dashboards using slicers and drill-through features.
Education
University of Hertfordshire
Master of Science
2023 - 2025
2 years 1 month
Vasireddy Venkatadri International Technological University
Bachelor's Degree
2015 - 2019
4 years
Licenses & certifications
Microsoft Certified: Fabric Data Engineer Associate
Issued: Jun 2026