Data Engineer

Job Summary Were seeking an experienced and skilled Data Engineer to work with our Consumer-Packaged Goods (CPG) clients and help build reliable, scalable data pipelines that drive strategic insights. This role is suited for someone who combines strong technical foundations with a solid understanding of how data enables analytics and business value. This is a dynamic, client-facing role that may involve working across adjacent areas such as data analytics, AI or even data science use cases based on project needs. This person will wear many hats in the role, but much of the focus will be on building out our Python ETL processes and automation. What You'll Do: • Client Interface: Collaborate directly with client teams, often interfacing with senior stakeholders from global CPG firms to understand data requirements and deliver engineering solutions. • Data Pipeline Development: Design and maintain robust ETL/ELT workflows across Azure Data Factory, Databricks, and related cloud platforms. • API Integrations: Ingest data from external APIs and web-sources including web scraping, and ensure error handling, backfilling, and retry mechanisms are in place for resilience. • Automation: Streamline client reporting through automation for enhanced efficiency. • AI-Enabled Solutions: Explore and leverage AI capabilities to develop intelligent and automated solutions that simplify data processes, accelerate analysis, improve operational efficiency, and address evolving client needs. • Utilities & Reusability: Build and maintain utility functions, helper scripts, and custom Python libraries to promote code reuse, modularity, and consistency across projects. • Collaboration with Analysts: Partner with analytics teams to structure raw data for reporting (Power BI, Tableau) and downstream insights. • Scalability & Governance: Ensure data quality, security, and reusability, while optimizing performance and adhering to best practices in data governance. • Cross-functional Flexibility: Opportunity to contribute in analytical or developmental tasks such as building dashboards, web-based tools, or supporting ML workflows when required. Experience You'll Need: • Education: Graduate/Postgraduate from Tier 1/Tier 2 institutions. • Professional Track: 4+ years of experience in Data Engineering/Data Analytics. • Technical Expertise: o Hands-on experience with Azure - ADF, Databricks, Key Vault, Blob/ADLS o Strong proficiency in SQL, Python (with libraries like pandas, requests, pySpark). o Ability to design reusable utility functions and develop custom Python modules/libraries to streamline repetitive tasks and improve code maintainability. o Experience with API-based data ingestion, data cleaning, and transformation. o Familiarity with version control (Git), modular code structuring, and collaborative development practices. • Communication Prowess: Good communication skills, with the ability to explain complex technical concepts clearly to both technical and non-technical stakeholders. • Adaptability: Willingness to wear multiple hats and contribute to broader data and analytics