Data Engineer

Job Description: Data Engineer

Position Title: Data Engineer

Department: Business Intelligence & Analytics

Employment Type: Full-time

Executive Summary & Role Intent

We are seeking an experienced Data Engineer to architect, build, and maintain the enterprise data foundation supporting our Business Intelligence, Analytics, and AI initiatives. This role is pivotal in transforming multi-source operational data into a secure, governed, high-performance Lakehouse ecosystem.

Beyond standard ETL tasks, you will be responsible for replacing manual data uploads with automated, monitored pipelines, implementing medallion architecture (Bronze, Silver, Gold), and ensuring every dataset across the organization is reliable, traceable, and performance-optimized.

Primary Responsibilities

1. Enterprise Lakehouse Architecture & Engineering

• Architect, deploy, and maintain the enterprise Lakehouse platform on Microsoft Fabric (including OneLake storage, Delta tables, Fabric Warehouse, and SQL endpoints).
• Manage workspace hierarchy, environment configurations, CI/CD deployment pipelines, and overall platform health across Development, Test, and Production environments.

2. Ingestion, Integration & Automation

• Design, build, and optimize scalable ingestion pipelines connecting diverse data sources: SaaS CRMs, REST APIs, client databases, SharePoint, Excel, and legacy flat files.
• Utilize Fabric Data Factory, Dataflows Gen2, PySpark notebooks, OneLake shortcuts, and database mirroring to eliminate manual workflows in favor of automated, scheduled, and fully monitored loads.

3. Data Modeling & Semantic Layer Alignment

• Model Silver and Gold data layers using dimensional modeling methodologies (Kimball), wide operational KPI tables, and Direct Lake-optimized datasets for Power BI.
• Collaborate directly with BI Managers and Analytics Engineers to ensure underlying schema designs natively support complex DAX measures and direct reporting needs.

4. Quality, Observability & Recovery

• Embed automated data validation checks, schema enforcement, reconciliation, and proactive alerting into every pipeline.
• Standardize business-rule exception handling (e.g., time/attendance anomalies) and maintain operational runbooks for data freshness, failure recovery, and incident response.

5. Governance, Security & Compliance

• Enforce data governance standards by maintaining dataset registries, data lineage, and metadata catalogs.
• Implement Role-Based Access Control (RBAC), Entra ID security groups, item-level permissions, and sensitivity labels to support auditability and strict client data segregation.

6. Performance Optimization & FinOps

• Tune Spark configurations, Delta indexing/z-ordering, and SQL queries to keep computational workloads operating within allocated Fabric Capacity Units (CU).
• Monitor capacity usage and optimize refresh schedules to ensure predictable, cost-effective scaling alongside growing data volumes.

7. AI Readiness & Cross-Functional Operations

• Structure governed datasets and semantic models to support Fabric Data Agents, Microsoft Copilot, and natural language query tools.
• Manage incoming engineering requests via an intake queue with structured SLA estimates, partnering across BI, Workforce Management, Finance, IT, and external client stakeholders.

Candidate Qualifications

Required Experience & Technical Proficiencies

• Experience: 3+ years of dedicated experience in data engineering, building, and maintaining enterprise-grade production pipelines.
• Core Languages: Advanced SQL (query optimization, window functions, incremental loading, complex joins) and Python for data engineering (PySpark, pandas, REST API integration).
• Platform Expertise: Hands-on experience with Microsoft Fabric or equivalent modern cloud data stacks (Azure Data Factory, Azure Synapse, Databricks, or Snowflake).
• Architecture: Proven knowledge of Medallion Architecture principles (Bronze/Silver/Gold layers) and dimensional data modeling (star/snowflake schemas).
• BI & Analytics: Working knowledge of Power BI (Direct Lake and Import modes, dataset refresh management, and functional DAX troubleshooting).
• DevOps & Automation: Experience with Git version control, deployment pipelines, CI/CD for data assets, and scripting via Python or PowerShell.
• Communication: Exceptional written and verbal English communication skills, with a track record of documenting technical architectures and presenting data logic to business stakeholders.

Preferred Qualifications

• Industry Domain: Experience processing operational datasets within BPO, logistics, supply chain, contact centers, or shared services environments (e.g., productivity, workforce management, attendance, and SLA data).
• Certifications: Microsoft Certified: Fabric Data Engineer Associate (DP-600) or Fabric Analytics Engineer Associate.
• Tools & Frameworks: Exposure to Azure DevOps, Agile/Scrum delivery, enterprise data catalogs (e.g., DataHub, Microsoft Purview), and AI-driven data tools.

Core Competencies & Engineering Philosophy

• Product Mindset: Treats data pipelines as critical business products complete with ownership, thorough tests, monitoring, and clear documentation.
• Business-Centric Modeling: Models data according to the business question being asked rather than the raw structure of source systems.
• Reusability & Efficiency: Standardizes single-source-of-truth tables to avoid redundant data duplication.
• Resource Awareness: Monitors compute cost and capacity utilization as rigorously as pipeline runtimes.