Sr Fullstack Data & AI Search Engineer

Job Summary

Role: Senior Fullstack Data AI Search Engineer

We are looking for a hands-on Data AI Search Engineer to design and deliver a production-grade, AI-augmented enterprise search capability for a large international organisation. The engagement covers the full pipeline from raw data ingestion through to AI-generated, grounded answers surfaced via a conversational or search interface.

The right candidate combines deep Elasticsearch engineering with practical experience building Retrieval-Augmented Generation (RAG) pipelines and agentic AI workflows. This is an individual contributor role with direct impact on a critical knowledge management platform.

Role Overview

Experience: Senior (8+ years total; 6+ years in enterprise search / RAG / LLM applications)

Profile: Senior Data AI Search Engineer with hands-on expertise in RAG pipelines and agentic AI workflows.

Primary Focus: Elasticsearch + RAG + agentic AI workflows

Key Responsibilities

• Design and build scalable ingestion pipelines and connectors from enterprise sources including SharePoint, Liferay, web crawls, Data Lakes, and corporate systems into Elasticsearch or equivalent search indexes.
• Support batch, incremental, and near-real-time indexing; implement change tracking, version management, source provenance, access permission mapping, and deletion event handling to keep the index accurate.
• Build document conversion pipelines for PDF, Word, Excel, PowerPoint, HTML, email, and scanned content; convert to structured Markdown and vector embeddings using tools such as Marker, Docling, or equivalent frameworks.
• Design semantic chunking strategies (chunk size, overlap, section-aware splitting, heading preservation, table handling) and implement metadata extraction, enrichment, and deduplication during ingestion.
• Develop hybrid search capabilities combining BM25 keyword search, semantic vector search, metadata filtering, and contextual retrieval.
• Build re-ranking pipelines using embedding models, cross-encoders, or custom ranking logic to improve result relevance.
• Implement advanced retrieval techniques: query rewriting, query expansion, multi-query retrieval, parent-child retrieval, contextual document embeddings, and contextual compression.
• Enforce security controls so users retrieve only content they are authorised to access.
• Design and build the end-to-end RAG pipeline connecting enterprise search to large language models for grounded answer generation.
• Implement agentic workflows where the AI can invoke tools, call enterprise APIs, perform multi-step reasoning, and refine searches iteratively to answer complex queries.
• Engineer prompt orchestration patterns: system prompts, retrieval prompts, guardrails, context assembly, response formatting, and fallback strategies for low-confidence or ambiguous queries.

Technical Requirements

Core Search Engineering

• Deep, hands-on Elasticsearch experience: query DSL, BM25 tuning, function_score, boosting and decay functions, multi-field matching.
• Index and data modelling: field type selection, custom analyzers and tokenizers per content type (code, prose, structured records, multimedia).
• Cluster operations: shard strategy, index sizing, reindexing, query latency tuning, and cluster health management.
• Search evaluation and relevance testing: building ground-truth benchmarks, measuring precision/recall, NDCG, and iterating against them.
• Experience with Elasticsearch, OpenSearch, Azure AI Search, or equivalent enterprise search platforms.

Data and Ingestion Engineering

• Proven experience building or configuring connectors for SharePoint, Liferay, databases, and Azure Data Lake including incremental sync, CDC, rate limiting, and API edge-case handling.

Disclaimer: This job posting & Location has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.