Data Engineer Advisor

Req ID: phone_number

We are currently seeking a Data Engineer Advisor to join our team in Chennai, Tamil Nadu (IN-TN), India (IN).

Technical Expertise

• Databricks Ecosystem: Mastery of Unity Catalog, Delta Lake, Photon engine, Serverless Compute, Databricks Asset Bundles (DABs), and Delta Live Tables (DLT).
• Core Languages: Advanced proficiency in Python (PySpark), SQL, and standard shell scripting languages (Bash / PowerShell).
• Infrastructure DevOps: Strong experience with Terraform, git workflows, and CI/CD tools (e.g., Azure DevOps, GitHub Actions, or GitLab CI).
• Cloud Platforms: Hand-on experience administering data infrastructure on at least one cloud provider (AWS, Azure, or GCP).
• Distributed Systems: Complete understanding of Apache Spark core concepts, memory management, optimizations (Z-order, data skipping), and tuning.

Key Responsibilities Platform Architecture Foundation Build

• Architect and provision the core Databricks ecosystem across multi-workspace environments using cloud-native services (Azure, AWS, or GCP).
• Establish shared data structures and optimized storage strategies utilizing the Delta Lake format, Auto Loader, and Delta Live Tables (DLT).
• Implement cost-management strategies, configuring optimal cluster sizing, autoscaling, and serverless SQL warehouses to balance performance against cloud spend.