John Subhane

Data Engineering & Analytics (Databricks / dbt): • Design and management of a Data Lake on Azure (ADLS Gen2) integrated with Azure Databricks. • Development of data transformation pipelines with dbt on Databricks (star schemas, analytical tables, business marts). • Implementation of dbt data quality tests (not null, unique, relationships) to ensure reliability of exposed datasets. • Performance optimization using Delta Lake (partitioning, Z-ORDER, time travel management). • Standardization of Bronze / Silver / Gold layers to simplify data access for business teams. • Exposure of analytical datasets to Business Analysts for BI and exploratory analysis. Machine Learning & MLOps (MLflow): • Implementation and use of MLflow to track ML experiments (parameters, metrics, and artifacts). • Model versioning and management through MLflow Model Registry to streamline deployment and governance. • Close collaboration with Data Scientists to industrialize ML pipelines on Databricks. • Facilitation of the experimentation-to-production workflow in a reproducible and secure framework. Security, Governance & Operations: • Implementation of fine-grained access control (Azure RBAC, Databricks ACLs, Unity Catalog where applicable). • Securing sensitive data through access controls and environment separation. • Operational platform maintenance: monitoring, incident management, and continuous optimization. • Documentation of standards and support for users (BAs & Data Scientists) in leveraging the Data Lake.

• Developed data pipelines using Python, PySpark, and Scala for the ingestion, processing, and real-time analysis of operational data. • Implemented a Big Data architecture with Hadoop, Spark, and Kafka, and a microservices architecture using FastAPI to expose analytical APIs. • Orchestrated ETL workflows with Apache Airflow and monitored them through Grafana. • Automated recurring and administrative tasks using Shell scripts. • Deployed on Azure using AKS, Blob Storage, and infrastructure as code with Terraform. • Built CI/CD pipelines with GitLab for secure Docker deployments, including testing, vulnerability scanning, and secret management with Azure Key Vault. • Applied DevOps best practices to ensure infrastructure resilience, security, and scalability.

• Designed and deployed secure data pipelines using Python and Spark on Databricks, with real-time processing via Kafka and orchestration through Apache Airflow. • Integrated and modeled data in Snowflake to build key performance indicators (KPIs) related to flight and maintenance data. • Implemented serverless cloud architectures (Cloud Functions, Cloud Storage, Dataform/Cloud Data Fusion) alongside Databricks for data management and processing. • Adopted DevOps practices using GitLab CI/CD and Docker to automate testing, packaging, and deployment on GCP (Artifact Registry, Cloud Run / GKE). • Monitored data workflows and optimized performance through Grafana, with integrated alerting via Cloud Monitoring. • Applied security best practices (secret management, encryption, defense in depth) and maintained structured technical documentation on Confluence. • Led projects using Agile methodology (JIRA), with regular interactions with business teams to refine requirements.