AI and Data Engineer

Responsibilities

• 1. Data Cleaning, Curation & Pipeline Development
• Clean, normalize, and validate unstructured or messy data to ensure high quality inputs for analytics and AI models.
• Write and optimize DAX queries to programmatically extract and aggregate tabular data directly from Power BI reports and datasets.
• Write performant SQL queries, stored procedures, and schema migrations across both PostgreSQL and SQL Server (SSMS).
• Build, automate, and monitor reliable data pipelines (ETL/ELT) using Python and Azure cloud services.
• 2. AI, GenAI & Model Performance Optimization
• Assist in developing and maintaining AI/GenAI workflows (RAG pipelines, prompt templates, vector search integrations with Azure OpenAI).
• Profile and optimize inference speed and resource usage for Machine Learning and GenAI systems:
• Implement caching, prompt token reduction, and smart chunking strategies to minimize LLM latency and API bills.
• Apply model optimization techniques (such as ONNX runtime conversion, quantization, batching, and GPU/CPU profiling).
• Track and monitor model accuracy, latency, and throughput in staging and production.
• 3. Statistical Analysis & Validation
• Apply fundamental statistical concepts (summary statistics, distributions, hypothesis testing, data sampling, regression) to audit data distributions and evaluate model outputs.
• Identify anomalies, outliers, and data drift across pipeline stages.
• 4. Version Control & CI/CD
• Maintain clean, documented code and participate in peer code reviews using Git and GitHub.
• Work with automated build and deployment pipelines (GitHub Actions or Azure DevOps).

Qualifications

• Required Qualifications & Technical Skills
• Experience: 2 to 4 years of hands-on experience in data engineering, analytics engineering, or applied machine learning.
• Databases & Querying:
• Solid working proficiency in both PostgreSQL and SQL Server Management Studio (SSMS), including writing complex joins, indexing, and analyzing basic query execution plans.
• Practical experience writing and debugging DAX queries to fetch data out of Power BI models.
• Programming & Data Wrangling:
• Strong Python skills with libraries like Pandas, NumPy, and SQLAlchemy for data manipulation and cleaning.
• AI & Optimization Foundations:
• Understanding of ML/DL lifecycle and basic Generative AI concepts (LLM APIs, embeddings, vector search).
• Hands-on exposure to latency optimization (caching, batch inference, token usage control, ONNX/runtime speedups).
• Mathematics & Statistics: Solid understanding of probability, distributions, data validation methods, and evaluation metrics (precision, recall, MSE, perplexity).
• Cloud & Collaboration Tools:
• Exposure to Microsoft Azure services (Azure Data Factory, Blob Storage / ADLS, Azure OpenAI).
• Daily proficiency with Git and GitHub (branching, PR workflows, resolving merge conflicts).

Required Skills

• Data engineering, analytics engineering, or applied machine learning experience (2 to 4 years).
• PostgreSQL and SQL Server Management Studio (SSMS), including complex joins, indexing, and basic query execution plan analysis.
• DAX queries to fetch data out of Power BI models.
• Python (Pandas, NumPy, SQLAlchemy) for data manipulation and cleaning.
• Understanding of ML/DL lifecycle and basic Generative AI concepts (LLM APIs, embeddings, vector search).
• Latency optimization (caching, batch inference, token usage control, ONNX/runtime speedups).
• Probability, distributions, data validation methods, and evaluation metrics (precision, recall, MSE, perplexity).
• Microsoft Azure services (Azure Data Factory, Blob Storage / ADLS, Azure OpenAI).
• Git and GitHub (branching, PR workflows, resolving merge conflicts).

Preferred Skills

• Familiarity with tools like DAX Studio or Tabular Editor.
• Experience with LangChain, LlamaIndex, or Azure AI Search.
• Any foundational Microsoft Azure certification (e.g., DP-900, DP-203, or AI).