Rajachozhan Iyappan

– Collaborated with business stakeholders and analysts to gather data requirements, define KPIs, and build scalable data solutions using Azure Data Services, improving reporting turnaround time by 35% and enhancing cross-departmental alignment through effective communication and documentation practices. – Designed and implemented greenfield project using Databricks, Synapse, Microsoft Fabric, and Unity Catalog, enabling real-time analytics and reducing data latency and operational delays by 40%. – Leveraged PySpark in Databricks to optimize large-scale ETL pipelines, reducing data processing time by 50% for customer engagement datasets sourced directly from Dataverse and external APIs. – Developed automated ETL pipelines using Python and Azure Data Factory (ADF) that validate, test, and load data, eliminating manual effort and improving data accuracy by 30% across business domains. – Built metadata-driven pipelines with Delta Lake, Azure Purview, ADLS, and Python, enabling scalable, governed data movement in both Databricks and ADF environments with dynamic schema handling. – Integrated Power BI dashboards with curated Delta tables to deliver near real-time insights on CRM customer journeys, reducing dashboard refresh times and reporting delays by 45% across regions. – Architected a Delta Lake-based data lakehouse for storing structured and semi-structured data with schema enforcement and time travel, improving audit readiness and regulatory compliance for all departments. – Designed a hybrid data warehouse and data lake architecture using Synapse and ADLS, supporting both historical and real-time analytics, with a 40% performance improvement on large-scale queries. – Configured IAM and RBAC policies in Azure and Unity Catalog to enforce fine-grained access controls, ensuring role-based segregation and boosting data security compliance across data teams. – Enabled data quality monitoring by embedding custom Python scripts within ADF pipelines, detecting anomalies and reducing recurring data issues by 20% in weekly operational reports. – Built reusable ADF templates for parameterized pipelines, improving development consistency and boosting productivity by 30% across distributed engineering and analytics teams globally. – Automated pipeline deployment using Azure DevOps CI/CD, minimizing manual steps and reducing deployment time and errors by 90%, ensuring consistent releases across all environments. – Implemented incremental data loading and watermarking strategies in ADF and Databricks pipelines, ensuring efficient processing of only changed records and reducing overall data ingestion time by 35%.

– Collaborated closely with cross-functional teams and business users to define data strategy and reporting requirements, improving business alignment and reducing change requests by 25%. – Initiated and led a new project implementation using Databricks, Azure Synapse, Unity Catalog, PySpark, and Microsoft Fabric, delivering production-ready pipelines within 3 months and enabling rapid, scalable data insights for key business units and analytical workflows across the enterprise. – Used PySpark in Databricks for efficient batch processing of millions of records daily, reducing data ingestion time from 2 hours to 30 minutes and improving operational efficiency significantly. – Developed Python modules for ETL automation and unit test frameworks, enabling test-driven development and improving code reliability and reusability by 40% across multiple development environments, promoting standardized practices and reducing technical debt across data engineering teams. – Engineered metadata-driven frameworks in Databricks and ADF using Delta Lake and Python, allowing pipeline reuse across 6+ domains and reducing code duplication in multi-team setups. – Integrated Power BI dashboards and published them to workspaces connected to Delta tables, providing real-time KPIs and improving dashboard refresh efficiency by 50% across business operations. – Built scalable data pipelines to transfer data from Cosmos DB to Azure SQL DB, and later to Amazon S3/Redshift, effectively reducing data redundancy and overall cloud storage cost by 20%. – Designed a robust data lake architecture using ADLS and Delta Lake, supporting version control, schema evolution, and rollback, facilitating easy data governance and compliance auditing. – Configured RBAC and IAM policies in Unity Catalog and Azure, enforcing strong data governance practices and ensuring GDPR compliance across regional and international data zones. – Migrated legacy ETL jobs to Apache Airflow-orchestrated Databricks notebooks, resulting in a 60% improvement in pipeline monitoring, scheduling efficiency, and failure recovery response. – Integrated Azure DevOps pipelines for CI/CD deployment of notebooks and configuration files, reducing environment drift and improving release cycles across all data engineering environments. – Implemented custom alerting mechanisms for pipeline failures and SLA breaches using Azure Monitor and Log Analytics, cutting pipeline downtime and alert response time significantly.

– Led data requirement analysis with finance and supply chain stakeholders, improving project planning efficiency and reducing scope creep by 20% across multiple enterprise-wide data initiatives. – Spearheaded a new project implementation using Databricks, PySpark, Unity Catalog, and Synapse, resulting in an enterprise-grade data lake with a 99.9% uptime SLA and enhanced reliability. – Optimized large-scale SSIS to ADF pipeline migration using PySpark in Databricks for high-volume data loads, reducing daily job runtime by 55% and enabling faster downstream reporting. – Created Python-based test scripts to validate data lineage and integrity in ETL pipelines, boosting accuracy and expanding automation test coverage by 60% in critical production systems. – Built metadata-driven pipelines with Delta Lake, Azure Purview, and Python, facilitating complete data cataloging and promoting pipeline reusability across both SAP and SQL Server data sources, while ensuring enterprise-wide consistency, scalability, and governance in enterprise data operations. – Integrated Power BI with Delta Lake and SAP BO data layers, developing interactive, executive-level dashboards and reports, significantly improving data visibility and decision-making by 50%. – Designed a Delta Lake and ADLS-based data lake architecture to standardize data ingestion from SAP BW and Azure SQL, enhancing long-term data reusability and storage efficiency organization-wide. – Created Tabular and Multi-dimensional models in SQL Server Analysis Services (SSAS) for OLAP queries, reducing end-user reporting latency by 30% and improving analytics performance and user experience. – Established role-based access controls (RBAC) using Git and Azure policies, ensuring secure collaboration and strict compliance with data privacy adherence across cross-functional data engineering teams. – Migrated existing SAP BO reports to Power BI, building a centralized self-service analytics platform, drastically reducing IT support dependency for ad-hoc reporting across business units. – Deployed and maintained source control and versioning via Git, improving change management, enabling traceability, and reducing rollback efforts due to unexpected deployment errors and issues. – Conducted UAT, coordinated deployment, and provided post-production support, achieving a 95% first-time acceptance rate from key business stakeholders across finance and supply chain.