Soheil Salmani

dbt Cloud configuration and DevOps • Set up dbt Cloud account from scratch, managing user permissions and setting up multiple environments (development, staging, and production). • Built CI workflows for data quality, including code validation, unit tests, and data tests, and `dbt_project_evaluator` to monitor adherence to dbt best practices. • Designed the deployment workflow across multiple environments. Data modeling and optimization • Designed and optimized data models to enhance performance and reliability for Qlik dashboards. • Shifted all metric computations and transformations from Qlik (QVDs) to dbt, centralizing business logic within the data warehouse. • Built and maintained metrics definitions using dbt Semantic Layer solution to standardize KPIs across teams. Training and knowledge sharing • Conducted training sessions on dbt best practices, data modeling layers (staging, intermediate, marts), and project structure. • Supported the upskilling of analytics and data teams through workshops and ongoing mentorship. Migration strategy and data mesh • Designed and executed a migration strategy leveraging dbt’s audit_helper to ensure a smooth transition. • Implemented data mesh principles, enabling collaboration across domains via dbt Mesh, data contracts and model versioning. Analytics engineering for finance team • Built and maintained a data pipeline to calculate the integrated margin using data coming from SAP, consolidating industrial, retail, and logistics margins into a unified financial metric.

Data mesh with dbt and Astronomer: • Migrate all Snowflake stored procedures/tasks to dbt models. • Set up and configured Airflow environments (development, staging, and production) in Astronomer. • Set up a data mesh with one dbt project for each feature team, and one dbt project for core models. • Enabled seamless collaboration between feature teams using data contracts and groups in dbt. • Implemented CI/CD workflows with GitLab CI to automate tests and deployments for all data projects. • Led the migration of all Airflow pipelines to Dagster as a PoC for a feature team. Data observability and semantic layer: • Configured Elementary for data observability on test results, run results and dbt artifacts. • Configured semantic models to test dbt semantic layer and Cube, and their integration with Tableau. • Presented a PoC on DataHub/OpenMetadata to have a centralized data catalog. Training data/analytics engineers on dbt and Airflow: • Conducted multiple workshops and presentations to keep data and analytics engineers updated on best practices and new features. • Developers reported a significant increase in productivity.

Generic dbt package: • Developed a generic dbt package to compute marketing insights across multiple sources (Google Ads, Google Search Console, etc.) and deployed it for 80+ clients. • Isolated client data by setting up separate environments per client on Google Cloud for enhanced security and scalability. • Implemented automated anomaly detection using Elementary to monitor and alert on data quality issues. • Built CI/CD workflows with GitLab CI to automate tests and enforce best practices via the `dbt_project_evaluator` package. • Integrated dbt models with a frontend analytics dashboard, enabling real-time insights for end-users. • Presented a PoC on SQLMesh to replace dbt. Orchestration and data observability: • Deployed and configured Dagster Cloud environments to orchestrate and monitor dbt pipelines. • Enabled data observability by configuring Elementary Cloud to track test results, run results, and dbt artifacts, improving monitoring and data quality insights. • Optimized 80+ data pipelines (each with ~150 dbt nodes) to reduce build time to under 15 minutes, significantly improving iteration speed.

Data architecture on AWS and Snowflake: • Designed the data architecture involving Snowflake, an Airflow environment, and a data lake using S3 and Glue. • Improved scalability by delegating the execution of tasks triggered by Airflow to ECS. • Configured the storage lifecycle in the data lake to avoid data loss. • Implemented security best practices for secret management, data encryption, and network access. • Set up everything by building and deploying Terraform modules. Data pipeline development: • Set up a new monorepo from scratch using Pants as the build system to easily format, lint, test, package, and distribute the code using a single CLI. • Configured DAGs to execute jobs in containers on ECS when ingesting data. • Configured data transformations in Airflow with dbt using Cosmos. • Built data extractors for partners' APIs. • Built a data transformer using the Presidio library for data anonymization (before ingesting data on S3). Data modeling and governance: • Implemented best practices for data modeling, data governance, and for structuring a dbt project. • Set up programmatically RBAC on Snowflake data objects using Permifrost and dbt grants. • Established a plan to migrate the old SQL models to dbt without causing friction for the analytics engineers/data analysts. CI/CD for data pipeline deployments: • Implemented CI/CD workflows with GitHub Actions. • Set up a multi-stage deployment (dev, staging, prod) for Airflow DAGs and dbt models. • Configured Git hooks using `pre-commit` to improve the commit quality submitted. Web app for exposing dashboards internally: • Created a React app using the Tableau Embedding API to expose dashboards internally through a lightweight web app. • Application entirely built using the Remix framework. LLM project to fact-check YouTube videos about Ledger: • Created a project during a Hackathon using Pinecone and OpenAI to assess the feasibility of the idea. • Transitioned the project to an ongoing initiative afterward.

Data architecture on Google Cloud Platform: • Built an Airflow infrastructure on Google Cloud Platform via Cloud Composer to manage all data pipelines, training and predict jobs developed in the lab. • Configured the infrastructure for autoscaling and access management (using IAM and RBAC). • Designed a project template for sharing common strategy for automatically pushing a DAG to Airflow thanks to the Jenkins' pipeline as code functionality. Data pipeline optimization and refactoring in sales forecast projects: • Refactored all data pipelines entirely using dbt to have a DAG-like execution model and increase readability and performance. • Implemented tests suites for all SQL (dbt tests) and Python (pytest) code. • Set up the documentation for the data models (dbt docs). • Built the CI/CD pipeline to run tests and deploy the project in a Kubernetes cluster (GKE) using Terraform. Chatbot for product information: • Designed a chatbot to easily retrieve product information in the form of cards in Google Chat. • Developed an API in Python using Flask. • Developed data connectors (APIs, BigQuery, Firestore) for data exposition. • Followed a TDD methodology using automated testing (with pytest and Nox) during the entire project. • Created a CLI (using Click) to control several aspects of the project via commands (manage data, train and store ML models, and test functionality via the terminal). • Set up the CI/CD pipeline using Jenkins, Cloud Run and Terraform. • Documented the code using Sphinx for future contributors. React application for image labelling: • Created a React application that will ask the user questions about the product they would be likely to buy against another if it was on sale. The goal was capture which products would be likely to cannibalize another. • Set up a Firestore database and its data models used by the app. • Designed the UI using the Material-UI framework.