Sr. Data Engineer (Databricks & Cloud Data Platforms)

• Role Overview: Semantic Layer & Data Modelling (Python-first) Design and implement the
• Data/Semantic Layer
• Python
• frameworks to define business metrics, dimensions, and KPIs as code.
• Build the "Gold" or "Serving" layer in Databricks using
• PySpark, embedding complex business logic directly into the data pipeline rather than downstream BI tools.
• Implement
• Metrics-as-Code
• practices to ensure consistent definitions across different analytical tools (Power BI, custom apps, data science models).
• Translate functional business requirements into Python-based transformation logic for the final serving layer.
• Data Pipelines & Databricks Design, build, and maintain scalable batch and streaming pipelines using
• Databricks Workflows
• Apache Spark.
• Develop robust ETL/ELT processes to ingest data from diverse sources (IoT, ERPs, APIs) and move it through the Medallion Architecture (Bronze $o$ Silver $o$ Gold).
• Optimize Python/Spark code for performance, partition management, and cost efficiency.
• Architecture & Governance Contribute to the design of Lakehouse architectures, ensuring the semantic layer supports both self-service BI and advanced analytics.
• Implement data governance and quality checks within the Python pipeline code.
• Ensure the semantic layer aligns with Clients data security standards (Row-Level Security, masking) and access controls.
• Collaboration Work closely with Business Analysts to understand metric definitions and codify them in Python.
• Collaborate with Data Scientists to expose semantic features for Machine Learning models.
• Mentor junior engineers for data modeling.
• Minimum
• 5 years
• of professiIn this role, you will play a key role in designing and implementing clients next-generation data platforms.
• You will move beyond traditional ETL by focusing on the
• Semantic Layer, using
• Python
• to define complex business logic, metrics, and data models as code.
• You will bridge the gap between raw data and business consumption, ensuring consistent, governed data delivery via Databricks.

Your Responsibilities: onal experience in Data Engineering.

• Strong expertise in Python
• for data manipulation and modelling (pandas, PySpark).
• Proven experience building
• Semantic/Serving layers
• programmatically (not just drag-and-drop BI modeling).
• Experience with
• Databricks
• and the
• Delta Lake
• ecosystem.
• Familiarity with "Data-as-Code" or "Metrics-as-Code" concepts (e.g., using dbt with Python models, or custom Python semantic frameworks).
• Strong understanding of dimensional modeling (Star Schema) and how to implement it via Spark/Python.

Technical skills Languages:

• Python (Advanced), SQL (Strong).
• Core Platform:
• Azure Databricks, Apache Spark (PySpark), Delta Lake.
• Semantic/Modeling:
• Building serving layers in PySpark, dbt (Python models), or Python-based metric layers.

Data Architecture:

• Medallion Architecture (Bronze/Silver/Gold), Dimensional Modeling.
• DevOps:
• CI/CD for data pipelines (Azure DevOps/GitHub Actions), Git flow.
• Concepts:
• ACID transactions, Time Travel, Unity Catalog, Governance.