AI Data Engineer

About the Role

We are looking for an AI Data Engineer to build and manage data platforms that power Generative AI, RAG, and intelligent automation solutions. The ideal candidate should have strong experience in Azure data services, Python, data pipelines, and search technologies, with exposure to AI-driven applications.

Key Responsibilities

Data Engineering & Integration

• Design and develop scalable data ingestion pipelines using Azure services.
• Integrate enterprise data sources such as SharePoint, Azure DevOps, databases, and document repositories.
• Build document processing and data transformation workflows for AI applications.
• Ensure data quality, scalability, and reliability across platforms.

RAG & Search Solutions

• Build and maintain Retrieval Augmented Generation (RAG) solutions.
• Manage vector databases and Azure AI Search indexes.
• Optimize document chunking, embeddings, and search relevance.
• Implement hybrid search and semantic search capabilities.

AI & Automation

• Support AI-powered applications using Azure OpenAI and related services.
• Collaborate with AI teams to enable knowledge retrieval and intelligent automation.
• Contribute to Agentic AI and workflow orchestration initiatives using tools such as LangGraph or LangFlow.

Security & Governance

• Implement RBAC, Row-Level Security, and data governance controls.
• Ensure compliance with enterprise security and privacy requirements.

Monitoring & Operations

• Monitor data pipelines and AI services using Azure Monitor and Log Analytics.
• Support troubleshooting, performance tuning, and platform reliability.

Required Skills

Must Have

• Strong Python and SQL programming skills.
• Experience with Azure Data Services (Databricks, Synapse, Data Factory, Functions).
• Experience building data pipelines and ETL/ELT solutions.
• Hands-on experience with Azure AI Search or similar search platforms.
• Understanding of RAG, embeddings, and vector search concepts.
• Experience with REST APIs and data integration.
• Knowledge of security and governance best practices.

Good to Have

• Azure OpenAI Service.
• LangChain.
• LangGraph.
• LangFlow.
• Agentic AI or Multi-Agent Systems.
• Dagster or workflow orchestration frameworks.
• Azure AI Foundry.

Preferred Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, IT, or related field.
• Azure Data Engineer (DP-203) or Azure AI Engineer (AI-102) certification preferred.

Success Measures

• Reliable and scalable AI-ready data platform.
• High-quality search and retrieval performance.
• Secure and compliant data environment.
• Successful delivery of RAG and AI automation solutions.
• Reduced operational effort through automation.

Why this version works better

Must-have skills (easy to find):

• Python
• SQL
• Azure Databricks
• Synapse
• Data Pipelines
• Azure AI Search
• RAG Basics

Good-to-have skills (bonus):

• LangGraph
• LangFlow
• Agentic AI
• CrewAI
• AutoGen

This will increase your candidate pool by 3-5x while still allowing you to identify candidates who can grow into Agentic AI initiatives later.