Python AI Engineer

Home - Careers - Python AI Engineer

Python AI Engineer

Home - Careers - Python AI Engineer

Python AI Engineer

Learn more about us

LOCATION: Hanoi, HCMC Job Description We are looking for a Python AI Engineer to join us in building and advancing our AI Indexing & Search Pipeline. In this role, you will be responsible for designing and implementing end-to-end pipelines for data crawling, content processing, vector embedding generation, and indexing to power high-performance semantic search capabilities.

• Pipeline Development & Core Architecture:

• Develop, optimize, and maintain the Python-based Batch AI Indexer system.

• Build and operate an end-to-end data pipeline

• Build and optimize Semantic Search / Vector Search capabilities.

• Data Crawling & Processing:

• Crawl and collect content from diverse web sources (Homepage, Naver Blog, and other relevant web data sources).

• Optimize text chunking strategies and data preprocessing tailored for language models.

• AI Models & Search Engine Integration:

• Utilize OpenAI Embeddings for content vectorization.

• Integrate and manage Elasticsearch 8.x for vector storage and search (working extensively with 1536-dimensional dense_vector and the Korean Nori Analyzer).

• Utilize FAISS for local vector search experiments and rapid prototyping.

• Batch, State & Reliability Management:

• Design and implement mechanisms for Incremental Indexing and Full Re-indexing.

• Manage incremental batch states using Redis.

• Track batch execution history and metadata using PostgreSQL.

• Implement robust retry mechanisms and error-handling logic to ensure high system reliability.

• Scheduling, Deployment & Operations:

• Configure automated job scheduling using APScheduler or Linux Crontab.

• Deploy and operate batch applications on Linux environments using systemd and Python virtual environments (venv).

• Perform logging, monitoring, and troubleshooting for batch job operations.

• Collaborate closely with Backend, AI, and DevOps team members to deploy and scale system operations.

Required Skills And Experience

• Core Experience & Technical Skills:

• 2–4+ years of strong hands-on development experience in Python.

• Proficiency in web scraping and crawling libraries (e.g., BeautifulSoup, Scrapy, Playwright, or Selenium).

• Practical experience with Elasticsearch 8.x (specifically Vector Search, dense_vector, kNN search, and custom analyzer configurations like Nori).

• Deep understanding and hands-on experience with Embedding models (e.g., OpenAI Embeddings, HuggingFace) and text chunking/preprocessing techniques.

• Databases & System Architecture:

• Solid skills in Redis (for caching/state management) and PostgreSQL (or equivalent relational databases).

• Strong knowledge of Batch Processing architectures, state management, idempotency, and error handling within data pipelines.

• Experience with FAISS or other vector search libraries for local experimentation.

• DevOps & Environment:

• Proficient in Linux environments, application deployment via systemd, and Python virtual environments.

• Experience with scheduling tools (crontab, APScheduler).

• Strong command of Git, logging frameworks, and system troubleshooting.

• Nice-to-Have Qualifications:

• Experience with Korean Natural Language Processing (Korean NLP).

• Experience with LLM orchestration frameworks such as LangChain or LlamaIndex.

• Hands-on experience with Docker, Kubernetes, or CI/CD pipelines.

Benefits

• Competitive and negotiable salary (UP TO 50M)
• Probation salary is 100% of official salary
• 13th-month salary and performance review twice a year
• Bonus for special occasions each year (Labor Day, National Day, Solar New year, Lunar New Year)
• Project bonus
• Employee’s professional certification and training allowances subject to company regulations
• Opportunity to participate in internal training programs on AI, foreign languages, and professional development
• Health Care Insurance
• Annual Health Assessment
• Social, health and unemployment insurance following government policy
• Enjoy company summer trips and other team building activities held monthly and quarterly
• Professional, creative and dynamic working environment
• Work five days per week with flexible check-in time

CONTACT

• Send CV to: [email protected]
• Hotline: +842466666100 (Recruitment Team)
• Facebook GEM Careers: https://www.facebook.com/tuyendungGEM
• Linkedin GEM Careers: https://www.linkedin.com/company/gem-careers-hiring/