Principal AI Architect

We're looking for a seasoned Software Architect with deep expertise in cloud-native enterprise systems and Generative AI. You will define and deliver scalable, secure, and production-grade GenAI architectures including multi-agent, RAG, LLMOps and AgentOps systems and lead cross-functional teams to build and operate them. This role combines hands-on technical leadership, systems thinking, and strong stakeholder management.

Role & responsibilities

Architecture & System Design

• Design scalable, modular, and cloud-native architectures for GenAI applications (microservices, event-driven, serverless)
• Define system boundaries, data flows, orchestration, and integration patterns for LLMs, vector DBs, embedding services, and tool integrations
• Produce architecture artifacts (Layered Architecture Diagrams, C4 Models, DFDs, Class, Sequence, ER & Use Case diagrams, different types of blueprints, API contracts, design and trade-off decisions)

GenAI & Agentic Systems

• Architect and deliver Retrieval-Augmented Generation (RAG) pipelines, Natural Language to SQL Flows, fine-tuning strategies, multi-modal capabilities, and tool augmented agents
• Design agent orchestration and multi-agent frameworks enabling planning, reasoning, and secure tool invocations, implement and design Agent prototypes and Communication Protocols
• Define prompt engineering standards, memory models (episodic/semantic/procedural), and context management.

LLMOps & AgentOps

• Define and implement model lifecycle pipelines: training, fine-tuning, validation, deployment, rollback, and monitoring

• Build AgentOps processes for agent lifecycle, behavior tracking, governance and performance optimization

• Automate CI/CD for models, agents and services (MLflow, TFX, BentoML, custom pipelines). Integration, Security & Compliance

• Integrate GenAI services with enterprise systems (ERP, CRM, data lakes, APIs) using secure, scalable interfaces

• Ensure secure access controls, data privacy, encryption, and compliance (GDPR, HIPAA, SOC2)

• Define responsible AI practices: bias mitigation, explainability, audit trails, and output governance

• GenAI security classify, encrypt & sign data/models; enforce least-privilege with short-lived creds and CI/CD security gates; telemetry, drift/hallucination alerts, kill switch & runbooks

• Agentic AI security - provable agent identity/attestation, tool allowlist + human gate for high-risk actions; ephemeral scoped tokens, sandboxed execution, and replayable audit traces

Observability, Ops & Cost Optimization

• Define telemetry, tracing, and logging for models and agents; monitor performance, drift, hallucination rates and user feedback loops
• Build dashboards, alerts and runbook guidance for operational health
• Design systems for cost efficiency (autoscaling, spot instances, serverless choices) and support FinOps practices. Leadership, Collaboration & Documentation
• Lead cross-functional teams (product, data science, AI engineers, platform) through architecture reviews, workshops, and technical decisioning
• Maintain architectural standards, documentation, playbooks, and pattern libraries for GenAI systems. • Mentor engineers and evangelize best practices across the organization

Required qualifications & experience

• 14+ years software engineering experience with 7+ years in architecture or senior technical leadership roles (or equivalent)
• Proven track record designing and delivering cloud-native, production systems at enterprise scale
• Hands-on experience with GenAI/LLM systems, RAG, NL-SQL, agentic frameworks or similar productionized AI applications
• Strong knowledge of system design patterns (microservices, event-driven, CQRS, hexagonal architecture), and Low-Level Design Patterns
• Experience integrating ML/LLM services with enterprise data platforms and APIs while meeting security/compliance requirements
• Solid engineering background in at least two languages (Python, TypeScript, Go, Java, C#) and familiarity with modern frameworks.

Technical skills & technologies (comprehensive)

Cloud & Infra: AWS / Azure / GCP; Kubernetes, Docker, serverless (Lambda, Functions, Cloud Run), GPU instances

GenAI & ML: Hugging Face Transformers, OpenAI APIs, ,LangChain, LlamaIndex, Semantic Kernel, Haystack

Vector Stores: FAISS, Pinecone, Weaviate, Chroma, Postgres+pgVector, and other cloud vector stores LLMOps / MLOps: Custom Development of Ops Pipelines, MLflow, TFX, BentoML, Kubeflow

Data & Integration: Kafka, Spark, Airflow, Flink, ETL/ELT concepts, data lakes, API gateways (Apigee etc)

DevOps & IaC: Terraform, Pulumi, CloudFormation, GitHub Actions, Jenkins

Observability & Security: Prometheus, Grafana stack, OpenTelemetry, Jaeger, ELK, Datadog; Vault

IAM, LDAP/OAuth2/OIDC/SAML Connect, Snyk, SonarQube, SAST/SCA in pipelines, OWASPs, CWEs, CVEs

Databases & Storage: Relational (RDS/Cloud SQL), NoSQL (Mongo, DynamoDB, Cosmos DB), Redis, S3/Blob/GCS, ORM/ODM frameworks

Agent frameworks / tools: Understanding Basics of Agents required, Langgraph, Autogen, AutoGPT, AgentVerse, MetaGPT, CrewAI etc

Performance & scalability: SSR/ISR, caching strategies (CDN, edge), lazy loading, bundle optimization, performance budgets

Realtime & async: WebSockets, SSE, message brokers (Kafka, RabbitMQ), background workers.

Frontend frameworks: React (Next.js), Angular, Vue; component libraries and state (Redux/RTK, Context, Pinia, Zustand)

Styling & UI tooling: Component Libraries, Accessibility best practices, Responsive UI Frontend build & tooling: Vite, Webpack, Storybook, UI Frameworks.

Backend frameworks: Node.js/Express, FastAPI, serverless functions (AWS Lambda, Cloud Functions)

API design & integration: REST, gRPC, OpenAPI/Swagger, API versioning and contract testing, GraphQL(Optional) UX & product mindset: Design-system familiarity, usability, accessibility, and working with designers

Behavioral & leadership skills

• Strategic thinking with the ability to align architecture to product and business goals
• Excellent communicator: simplify complex technical concepts for technical and non-technical stakeholders
• Strong mentorship skills -able to raise team capability in GenAI architecture and engineering
• Pragmatic decision-maker with a bias for measurable outcomes and trade-off analysis
• High attention to detail, ownership, and accountability for reliability, security, and cost.

Nice-to-have

• Experience operating LLMs in regulated industries (Pharma/Retail/CPG)
• Familiarity with prompt auditing, hallucination detection, and automated quality checks
• Background in knowledge engineering, semantic search, or knowledge graphs
• Academic background in CS, ML, or equivalent applied experience
• Mobile & cross-platform (optional): React Native, Flutter basics for mobile integration

Deliverables & success metrics (examples)

• Production-ready GenAI architecture and deployment runbook
• Deployed RAG/agent pipeline with observable SLOs and monitoring dashboards
• Reduced model hallucination/incidents and measurable improvement in retrieval quality
• Architecture decision records (ADRs), standards library, and cross-team onboarding materials
• Cost targets achieved through optimized infra and autoscaling policies

Being a Mathemagician

• Embody MathCos culture and way of working
• Demonstrate ownership and strive for excellence in delivering results
• Actively engage and contribute to initiatives fostering company growth
• Support diversity and appreciate different perspectives