Full Stack • Java • System Design • Cloud • AI Engineering

Enterprise AI Assistant System Design

Design a scalable Enterprise AI Assistant — covering LLM serving infrastructure, RAG (Retrieval-Augmented Generation) pipelines, vector databases, prompt management, conversation context, multi-tenant data isolation, rate limiting, cost control, observability, and the safety and compliance requirements that govern AI in production.

1-Hour Interview Roadmap

Time Topic
0 – 5 min Requirements clarification
5 – 10 min Capacity estimation + cost model
10 – 18 min High-level architecture + LLM serving
18 – 28 min RAG pipeline — ingestion, chunking, embedding, vector search
28 – 36 min Conversation management — context windows, memory, session state
36 – 43 min Multi-tenancy, data isolation, access control
43 – 50 min Rate limiting, cost controls, prompt safety
50 – 56 min Observability, evaluation, model upgrades
56 – 60 min Trade-offs + common interview mistakes

What Are We Building?

An Enterprise AI Assistant that allows employees and customers to query organizational knowledge, get answers grounded in company documents, execute business workflows via natural language, and receive contextually appropriate responses — while maintaining data isolation between tenants, controlling costs, enforcing safety policies, and providing the observability required to improve the system continuously.

Enterprise AI assistants are not chatbots wired to GPT. The architecture must solve a distinct set of problems that are absent in consumer AI products:

Why enterprise AI is harder than consumer AI:

Dimension Consumer AI Enterprise AI
Knowledge source Model training data Private company documents, APIs, data
Data isolation Single tenant Strict per-tenant, per-role boundaries
Compliance Best-effort SOC 2, GDPR, HIPAA, financial regs
Answer accountability Attribution optional Every answer must be traceable to source
Cost model Pay per use, individual Enterprise budget, per-team allocation
Model updates Transparent improvement Change management, regression testing

The five challenges that make this hard:

  1. Grounding responses in private data — an LLM trained on public data knows nothing about a company's internal policies, products, or proprietary information. Retrieval-Augmented Generation (RAG) must bridge this gap without hallucinating.
  2. Context window management — LLMs have finite context windows (4K–128K tokens). Long conversations, large retrieved documents, and complex prompts compete for space. The system must manage context intelligently without losing relevant information.
  3. Multi-tenant data isolation — a financial firm with 50 business units cannot have one BU's assistant leak documents to another BU. Vector store queries, document access, and LLM context must be strictly scoped by tenant and role.
  4. Cost and latency at scale — LLM inference is expensive (~$0.002–$0.06 per 1K tokens). An enterprise with 10,000 employees making 50 queries/day at 2,000 tokens/query spends $20,000–$600,000/month. Cost control, caching, and model routing are architectural requirements.
  5. Evaluation and quality assurance — LLM outputs are probabilistic. The system must continuously measure answer quality, detect regressions when models or prompts change, and provide feedback mechanisms for users to flag incorrect answers.

Functional Requirements

Query and conversation:

  • Accept natural language queries from authenticated users
  • Return grounded responses with source citations from retrieved documents
  • Maintain conversational context across multiple turns in a session
  • Support follow-up questions that reference previous answers

Knowledge base management:

  • Ingest documents (PDFs, Word, web pages, structured data) into a searchable knowledge base
  • Index documents with access control metadata (org, team, role, classification)
  • Update and re-index documents when source changes
  • Support multi-tenant knowledge bases with strict isolation

Tool use / function calling:

  • Execute structured actions via tool calls (search CRM, run SQL query, fetch API data)
  • Chain multiple tool calls to answer complex queries
  • Present tool results as part of a coherent natural language response

Safety and compliance:

  • Filter inputs for prompt injection and jailbreak attempts
  • Filter outputs for PII, confidential data, and policy violations
  • Maintain full audit log of every query, retrieval, and response
  • Support opt-out of specific content types per tenant policy

Cost management:

  • Enforce per-user, per-team token budgets
  • Route queries to cheaper models when quality requirements are lower
  • Cache frequent responses to avoid redundant LLM calls

Non-Functional Requirements

Attribute Target
Response latency p50 < 3 seconds (streaming first token < 1 second)
Retrieval latency p99 < 200 ms for vector search
Availability 99.9% (LLM provider degradation handled via fallback models)
Throughput 1,000 concurrent queries
Context window Support up to 128K token context for enterprise models
Data isolation Zero cross-tenant data leakage — enforced at query and index level
Audit log retention 3 years minimum
Response grounding > 90% of factual claims attributed to a retrieved source

Capacity Estimation

Baseline assumptions:

  • 5,000 active enterprise users
  • 20 queries/user/day = 100,000 queries/day
  • Average query: 500 tokens input (user message + context) + 800 tokens output = 1,300 tokens/query
  • Document corpus: 500,000 documents, average 5 pages, 1,000 tokens/page = 2.5 billion tokens indexed
  • Average retrieval: 5 chunks × 500 tokens = 2,500 tokens of context added per query

LLM cost:

100,000 queries/day × 1,300 tokens = 130M tokens/day
GPT-4o: $2.50/1M input + $10/1M output
Input: (500 + 2,500) = 3,000 tokens/query
Output: 800 tokens/query
Cost/query: (3,000 × $2.50 + 800 × $10) / 1,000,000 = $0.0075 + $0.008 = $0.0155
Daily cost: 100,000 × $0.0155 = $1,550/day = ~$47K/month

This is substantial. Cache hit ratio of 30% (common queries answered from cache) saves ~$14K/month. Model routing (use GPT-3.5 for simple queries, GPT-4o for complex) reduces average cost by ~40%.

Vector database sizing:

500,000 documents × 5 pages × 400 word chunks = 2.5M chunks
Each chunk embedded as 1,536-dimensional vector (OpenAI text-embedding-3-large)
Storage: 2.5M × 1,536 × 4 bytes = 15.4 GB of vectors
+ metadata: 2.5M × 500 bytes = 1.25 GB
Total: ~17 GB — fits comfortably in a single Pinecone/Qdrant namespace

Throughput:

100,000 queries/day / 86,400 = ~1.2 QPS average
Peak (business hours): 4× = ~5 QPS
With 1,000 concurrent: at 3s avg response = 333 QPS throughput needed
LLM API (e.g., Azure OpenAI): rate limits at ~1,000 RPM = 16 RPS → need rate limiting / queuing

High-Level Architecture

flowchart TD
    subgraph Client
        WEB[Web App]
        API_GW[API Gateway]
    end

    subgraph Query Pipeline
        QUERY[Query Service]
        SAFETY_IN[Input Safety Filter]
        RETRIEVAL[Retrieval Service]
        CONTEXT[Context Builder]
        LLM_ROUTER[LLM Router]
        SAFETY_OUT[Output Safety Filter]
    end

    subgraph LLM Providers
        GPT4[Azure OpenAI GPT-4o]
        GPT35[Azure OpenAI GPT-3.5]
        LOCAL[Self-hosted Llama / Mistral]
    end

    subgraph Knowledge Base
        INGEST[Document Ingestion Service]
        CHUNK[Chunking Service]
        EMBED[Embedding Service]
        VSTORE[(Vector Store - Qdrant)]
        DOCSTORE[(Document Store - S3 + PG)]
    end

    subgraph State
        SESSION[(Session Store - Redis)]
        CACHE[(Response Cache - Redis)]
        AUDIT[(Audit Log - PostgreSQL)]
    end

    WEB --> API_GW
    API_GW --> QUERY
    QUERY --> SAFETY_IN
    SAFETY_IN --> RETRIEVAL
    RETRIEVAL --> VSTORE
    RETRIEVAL --> CONTEXT
    CONTEXT --> SESSION
    CONTEXT --> LLM_ROUTER
    LLM_ROUTER --> GPT4
    LLM_ROUTER --> GPT35
    LLM_ROUTER --> LOCAL
    LLM_ROUTER --> SAFETY_OUT
    SAFETY_OUT --> CACHE
    SAFETY_OUT --> AUDIT

    INGEST --> CHUNK
    CHUNK --> EMBED
    EMBED --> VSTORE
    INGEST --> DOCSTORE

Core Service Responsibilities

Service Responsibility
API Gateway Auth, rate limiting per user/team, request routing
Query Service Orchestrate the full query pipeline; return streaming response
Input Safety Filter Detect prompt injection, jailbreaks, PII in user input
Retrieval Service Vector search with tenant/role access filters; return top-K relevant chunks
Context Builder Assemble system prompt + conversation history + retrieved chunks within token limit
LLM Router Select model based on query complexity, cost tier, latency SLA
Output Safety Filter Detect PII, confidential data markers, policy violations in LLM output
Document Ingestion Accept new documents, extract text, trigger chunking and embedding pipeline
Chunking Service Split documents into overlapping chunks with metadata
Embedding Service Call embedding API (or local model), store vectors in vector store
Vector Store Sub-200 ms nearest-neighbor search with metadata filtering
Session Store (Redis) Conversation history per session, token count tracking
Response Cache Semantic cache — deduplicate similar queries against stored responses
Audit Log Every query, retrieval result, and response stored for compliance

Conversation State Machine

stateDiagram-v2
    [*] --> SESSION_CREATED: user starts conversation

    SESSION_CREATED --> QUERY_RECEIVED: user sends message
    QUERY_RECEIVED --> INPUT_FILTERED: safety filter evaluated
    INPUT_FILTERED --> QUERY_BLOCKED: safety violation detected
    INPUT_FILTERED --> RETRIEVING: safe input

    RETRIEVING --> CONTEXT_BUILT: chunks retrieved + history assembled
    CONTEXT_BUILT --> LLM_CALLED: prompt sent to LLM
    LLM_CALLED --> STREAMING: first token received
    STREAMING --> OUTPUT_FILTERED: full response received
    OUTPUT_FILTERED --> RESPONSE_BLOCKED: output violation detected
    OUTPUT_FILTERED --> RESPONSE_DELIVERED: safe output

    RESPONSE_DELIVERED --> QUERY_RECEIVED: user sends follow-up
    RESPONSE_DELIVERED --> SESSION_IDLE: no follow-up

    SESSION_IDLE --> SESSION_EXPIRED: idle timeout reached
    SESSION_IDLE --> QUERY_RECEIVED: user returns

    QUERY_BLOCKED --> QUERY_RECEIVED: user rephrases
    RESPONSE_BLOCKED --> RESPONSE_DELIVERED: fallback response sent

    SESSION_EXPIRED --> [*]

Step 1: Document Ingestion and RAG Pipeline

The Retrieval-Augmented Generation (RAG) pipeline is the core of the enterprise AI assistant. Without it, the LLM knows nothing about the company's private data.

Ingestion Pipeline

sequenceDiagram
    participant SRC as Document Source
    participant ING as Ingestion Service
    participant CHUNK as Chunking Service
    participant EMBED as Embedding Service
    participant VS as Vector Store
    participant DS as Document Store

    SRC->>ING: new document (PDF/Word/HTML)
    ING->>ING: extract text (pdfminer / docx2txt / BeautifulSoup)
    ING->>DS: store original + extracted text (S3 + PostgreSQL)
    ING->>CHUNK: chunk_document(text, metadata)
    CHUNK->>CHUNK: split into 400-token chunks with 50-token overlap
    CHUNK-->>ING: chunks[] with positional metadata
    ING->>EMBED: embed(chunks[])
    EMBED->>EMBED: call text-embedding-3-large API (batched)
    EMBED-->>ING: vectors[]
    ING->>VS: upsert(vectors[], metadata={tenant_id, doc_id, access_roles, ...})
    ING->>DS: update document status = INDEXED
    ING->>KAFKA: document.indexed event

Chunking Strategy

Chunking strategy significantly impacts retrieval quality. The goal: chunks that are semantically coherent and large enough to be informative but small enough to be precise.

Fixed-size chunking (baseline):

  • Split by token count (e.g., 400 tokens), with 50-token overlap
  • Simple and predictable, but can split sentences or logical units mid-thought

Semantic chunking (preferred for structured documents):

  • Split at natural boundaries: headings, paragraph breaks, section markers
  • Produces chunks that correspond to coherent logical units
  • For PDFs: respect page boundaries + paragraph breaks
  • For HTML: split at <h1>/<h2>/<p> tags

Chunk metadata (essential for RAG quality):

{
  "chunk_id": "chk_abc123",
  "doc_id": "doc_xyz789",
  "tenant_id": "tenant_acme",
  "access_roles": ["employees", "finance"],
  "source_type": "policy_document",
  "document_title": "Expense Reimbursement Policy v3.2",
  "section": "International Travel",
  "page_number": 4,
  "chunk_index": 12,
  "token_count": 387,
  "last_updated": "2026-06-01",
  "text": "International travel requires pre-approval from the department head for expenses exceeding $5,000..."
}

The access_roles field is the foundation of multi-tenant data isolation — the vector search filter enforces it at query time.

Step 2: Retrieval with Access Control

The Retrieval Service converts the user's query into a vector and finds the most semantically similar chunks in the vector store, filtered by the caller's tenant ID and access roles.

sequenceDiagram
    participant QS as Query Service
    participant RS as Retrieval Service
    participant EMBED as Embedding Service
    participant VS as Vector Store
    participant DS as Document Store

    QS->>RS: retrieve(query_text, tenant_id, user_roles, top_k=5)
    RS->>EMBED: embed(query_text)
    EMBED-->>RS: query_vector (1536-dim)
    RS->>VS: search(query_vector, filter={tenant_id, roles}, top_k=5)
    VS-->>RS: top_k chunks with scores + metadata
    RS->>DS: fetch_full_context(doc_id) for top chunks (optional)
    RS-->>QS: retrieved_chunks[] with source attribution

Hybrid Search (Dense + Sparse)

Pure vector search (dense) misses exact keyword matches. Pure keyword search (sparse, BM25) misses semantic similarity. Hybrid search combines both:

  • Dense retrieval: semantic similarity via embedding vectors → finds conceptually related content
  • Sparse retrieval: BM25 keyword matching → finds exact term matches (product codes, names, dates)
  • Reciprocal Rank Fusion (RRF): merges the two ranked lists into a single ranked result

Hybrid search significantly improves retrieval recall for enterprise queries that mix natural language with specific terminology.

Re-ranking

After retrieving top-K candidates (e.g., top 20), a cross-encoder re-ranker scores each candidate against the full query for relevance. The re-ranker is more accurate than embedding similarity alone but too expensive to run against the full corpus — hence the two-stage design:

  1. Stage 1: Embedding-based ANN search → retrieve top-20 candidates in ~50 ms
  2. Stage 2: Cross-encoder re-ranking of top-20 → select top-5 in ~100 ms

This gives near-cross-encoder accuracy at a fraction of the cost.

Step 3: Context Window Management

The assembled prompt must fit within the LLM's context window while including:

  • System prompt (persona, instructions, safety rules): ~500 tokens
  • Conversation history (last N turns): variable
  • Retrieved chunks (5 × ~400 tokens = 2,000 tokens)
  • User's current query: ~100 tokens
  • Output reservation: ~1,000 tokens

For a 16K context window: 500 + history + 2,000 + 100 + 1,000 = 3,600 + history → up to 12,400 tokens for history.

Context Management Strategy

Conversation summarization: When history exceeds the budget, instead of truncating old turns (which loses context), summarize them:

[System: You are an enterprise AI assistant...]
[History summary: The user asked about expense reimbursement limits for international travel. We covered the $5,000 pre-approval threshold and the documentation requirements.]
[Recent turns: last 3 full exchanges]
[Retrieved context: 5 chunks]
[User: What about meal allowances?]

Retrieval in conversation: For follow-up questions, the retrieval query must incorporate the conversation context, not just the latest message. Technique: query rewriting — use the LLM to rewrite the follow-up question as a standalone query:

  • User turn 1: "What is the expense reimbursement policy?"
  • User turn 2: "And what about for managers?"
  • Standalone query for retrieval: "What are the expense reimbursement policies for managers?"

This prevents poor retrieval caused by ambiguous follow-up questions.

Step 4: LLM Router

Not every query needs the most powerful (and expensive) model. The LLM Router classifies queries by complexity and routes them to the appropriate model tier:

Model Tiers

Tier Model Cost (per 1K tokens) Use Case
Tier 1 GPT-3.5 / Claude Haiku $0.0002–0.0005 Simple Q&A, factual lookup, summarization
Tier 2 GPT-4o / Claude Sonnet $0.002–0.006 Multi-step reasoning, complex queries, synthesis
Tier 3 GPT-4o (32K) / Claude Opus $0.01–0.06 Complex analysis, long documents, code gen
Local Llama-3 / Mistral Infrastructure cost PII-sensitive, air-gapped, cost cap exceeded

Routing Logic

def route_query(query, context):
    # Rule-based routing (fast)
    if context.contains_pii and not context.pii_allowed_for_external:
        return ModelTier.LOCAL  # never send PII to external providers
    
    if context.user_token_budget_exceeded:
        return ModelTier.TIER_1  # degrade gracefully
    
    # Complexity classifier (lightweight ML model)
    complexity = classify_complexity(query)  # SIMPLE, MEDIUM, COMPLEX
    
    routing_map = {
        "SIMPLE": ModelTier.TIER_1,
        "MEDIUM": ModelTier.TIER_2,
        "COMPLEX": ModelTier.TIER_3
    }
    return routing_map[complexity]

Fallback Chain

If the primary model is unavailable or rate-limited:

GPT-4o → Azure OpenAI fallback region → Claude Sonnet → GPT-3.5

Fallback is transparent to the user. Degraded responses (from a lower-tier model) are flagged in the audit log but not shown to the user unless quality drops below a threshold.

Step 5: Prompt Engineering and Safety

Prompt Structure

Every LLM call uses a structured prompt:

[SYSTEM]
You are an AI assistant for Acme Corp. You answer questions based only on the context provided.
If the answer is not in the provided context, say "I don't have information about that in my knowledge base."
Never reveal confidential information marked [CONFIDENTIAL].
Always cite your sources using the format [Source: {document_title}, {section}].

[CONTEXT]
Source 1: [Expense Policy v3.2, International Travel]
"International travel requires pre-approval from the department head for expenses exceeding $5,000..."

Source 2: [Expense Policy v3.2, Meal Allowances]
"Meal allowances for international travel are $75/day for destinations in North America and $100/day for other regions..."

[CONVERSATION]
User: What is the expense reimbursement policy for international travel?
Assistant: International travel expenses require department head pre-approval for amounts exceeding $5,000 [Source: Expense Policy v3.2, International Travel]. ...

[USER]
What are the meal allowances for a trip to London?

The system prompt enforces grounding ("answer only from context"), citation format, and confidentiality handling. These constraints are not negotiable per tenant.

Input Safety Filtering

Before the query reaches retrieval, an input safety filter checks for:

  • Prompt injection: "Ignore previous instructions and..." / "You are now DAN..."
  • Jailbreak attempts: instructions to override safety constraints
  • PII in input: SSNs, credit card numbers, health information
  • Policy violations: queries about competitors, restricted topics per tenant policy

Detection approach:

  • Rule-based patterns for obvious prompt injections (regex + keyword matching, ~0.5 ms)
  • Classifier model for subtle jailbreaks (fine-tuned small model, ~20 ms)
  • PII detection (Presidio or equivalent, ~10 ms)

If blocked, return a structured error:

{
  "error": "QUERY_BLOCKED",
  "reason": "INPUT_POLICY_VIOLATION",
  "message": "This query contains content that cannot be processed. Please rephrase.",
  "blocked_at": "2026-07-17T10:30:00.000Z"
}

Output Safety Filtering

The LLM output is filtered before delivery for:

  • PII leakage: did the model extract and repeat a user's SSN from a retrieved document?
  • Confidential data markers: did the model quote a chunk marked [CONFIDENTIAL]?
  • Hallucination detection: does the response contain claims not supported by retrieved chunks?

Hallucination detection is done via attribution scoring: every factual claim in the response is matched against the retrieved chunks. Claims not attributed to any chunk are flagged and optionally suppressed.

Step 6: Multi-Tenancy and Data Isolation

Vector Store Namespacing

Each tenant gets a dedicated namespace in the vector store. Queries are always scoped to the caller's namespace — a query from Tenant A can never retrieve vectors indexed under Tenant B.

def retrieve_chunks(query_vector, tenant_id, user_roles, top_k):
    return vector_store.query(
        vector=query_vector,
        top_k=top_k,
        filter={
            "tenant_id": {"$eq": tenant_id},
            "access_roles": {"$in": user_roles}
        },
        namespace=f"tenant_{tenant_id}"
    )

Namespaces enforce isolation at the storage layer — even a bug in the application code cannot return cross-tenant results because the vector store rejects queries against foreign namespaces.

Document Access Control

Documents are tagged with:

  • tenant_id: which organization owns this document
  • access_roles: which roles can see this document (["all_employees"], ["finance"], ["executives"])
  • classification: document sensitivity (PUBLIC, INTERNAL, CONFIDENTIAL, SECRET)

At ingestion time, the ACL is encoded into the chunk metadata and stored in the vector index. At query time, the user's roles are fetched from the identity provider and used as a filter on every vector search.

LLM Context Isolation

The assembled prompt never contains chunks from other tenants. The context builder enforces:

  1. Retrieval only fetches chunks for the current user's tenant + roles
  2. The system prompt explicitly states the user's tenant context
  3. Conversation history is stored per (tenant_id, session_id) — no shared sessions across tenants

Database Schema

PostgreSQL: Documents, Chunks, Sessions, Audit

-- Document registry
CREATE TABLE documents (
    doc_id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id        UUID NOT NULL,
    title            VARCHAR(500) NOT NULL,
    source_type      VARCHAR(50)  NOT NULL,   -- POLICY, FAQ, PRODUCT, REPORT
    source_url       VARCHAR(2000),
    s3_key           VARCHAR(500),
    mime_type        VARCHAR(100) NOT NULL,
    classification   VARCHAR(20)  NOT NULL DEFAULT 'INTERNAL',
    access_roles     TEXT[]       NOT NULL DEFAULT '{"all_employees"}',
    status           VARCHAR(20)  NOT NULL DEFAULT 'PENDING',  -- PENDING, INDEXED, FAILED
    chunk_count      INT,
    token_count      INT,
    version          INT          NOT NULL DEFAULT 1,
    created_at       TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    indexed_at       TIMESTAMPTZ,
    updated_at       TIMESTAMPTZ  NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_docs_tenant_status ON documents (tenant_id, status, updated_at DESC);
CREATE INDEX idx_docs_source_type   ON documents (tenant_id, source_type);

-- Conversation sessions
CREATE TABLE ai_sessions (
    session_id       UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id        UUID NOT NULL,
    user_id          UUID NOT NULL,
    title            VARCHAR(200),
    message_count    INT  NOT NULL DEFAULT 0,
    total_tokens     INT  NOT NULL DEFAULT 0,
    last_activity_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    expires_at       TIMESTAMPTZ NOT NULL
);

CREATE INDEX idx_sessions_user    ON ai_sessions (tenant_id, user_id, last_activity_at DESC);
CREATE INDEX idx_sessions_expiry  ON ai_sessions (expires_at) WHERE expires_at > NOW();

-- Conversation messages
CREATE TABLE ai_messages (
    message_id       UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    session_id       UUID NOT NULL REFERENCES ai_sessions(session_id),
    tenant_id        UUID NOT NULL,
    user_id          UUID NOT NULL,
    role             VARCHAR(10) NOT NULL,     -- USER, ASSISTANT, SYSTEM
    content          TEXT NOT NULL,
    token_count      INT,
    model_used       VARCHAR(50),
    latency_ms       INT,
    retrieved_chunks JSONB,                    -- chunk IDs used in this turn
    safety_flags     JSONB,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_messages_session ON ai_messages (session_id, created_at ASC);

-- Token budget tracking
CREATE TABLE token_budgets (
    budget_id        UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id        UUID NOT NULL,
    scope            VARCHAR(20) NOT NULL,     -- USER, TEAM, TENANT
    scope_id         UUID NOT NULL,
    period           VARCHAR(10) NOT NULL,     -- DAILY, MONTHLY
    budget_tokens    BIGINT NOT NULL,
    used_tokens      BIGINT NOT NULL DEFAULT 0,
    period_start     DATE NOT NULL,
    period_end       DATE NOT NULL,
    UNIQUE (tenant_id, scope, scope_id, period, period_start)
);

CREATE INDEX idx_budgets_scope ON token_budgets (tenant_id, scope, scope_id, period_end);

-- Audit log
CREATE TABLE ai_audit_log (
    audit_id         UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id        UUID NOT NULL,
    user_id          UUID NOT NULL,
    session_id       UUID,
    event_type       VARCHAR(50) NOT NULL,    -- QUERY, RETRIEVAL, RESPONSE, BLOCK
    query_hash       CHAR(64),                -- SHA-256 of query (for dedup analytics)
    model_used       VARCHAR(50),
    tokens_input     INT,
    tokens_output    INT,
    retrieved_doc_ids UUID[],
    safety_flags     JSONB,
    response_latency_ms INT,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW()
) PARTITION BY RANGE (created_at);

API Design

POST /v1/ai/query — Submit a Query

Request:

{
  "session_id": "sess_a1b2c3d4",
  "message": "What is the meal allowance for international travel to London?",
  "options": {
    "stream": true,
    "top_k": 5,
    "include_sources": true
  }
}

Response (streaming, Server-Sent Events):

data: {"type":"token","content":"Meal"}
data: {"type":"token","content":" allowances"}
data: {"type":"token","content":" for international"}
...
data: {"type":"complete","message_id":"msg_b2c3d4e5","sources":[
  {"doc_id":"doc_abc","title":"Expense Policy v3.2","section":"Meal Allowances","relevance_score":0.94},
  {"doc_id":"doc_def","title":"International Travel Guidelines","section":"Per Diem Rates","relevance_score":0.87}
],"tokens_used":1240,"model":"gpt-4o"}

POST /v1/ai/sessions — Start a Session

{
  "context": {
    "topic": "expense-policy",
    "persona": "hr-assistant"
  }
}

Response:

{
  "session_id": "sess_a1b2c3d4",
  "expires_at": "2026-07-17T18:00:00.000Z",
  "token_budget_remaining": 45000
}

GET /v1/ai/sessions/{session_id}/history — Get Conversation History

Returns the full turn-by-turn conversation with source attributions per message.

POST /v1/knowledge/documents — Ingest a Document

{
  "title": "Expense Reimbursement Policy v3.2",
  "source_type": "POLICY",
  "classification": "INTERNAL",
  "access_roles": ["all_employees"],
  "file_id": "fil_xyz789"
}

Response:

{
  "doc_id": "doc_abc123",
  "status": "PROCESSING",
  "estimated_index_time_seconds": 30
}

Kafka Event Architecture

Topic Producer Consumers Retention
ai.document.ingested Ingestion Service Chunking, Analytics 30 days
ai.document.indexed Embedding Service Ingestion Service (status update) 30 days
ai.query.received Query Service Analytics, Rate Limiter 7 days
ai.query.completed Query Service Analytics, Billing, Eval Pipeline 90 days
ai.query.blocked Safety Filter Security Alerting, Analytics 90 days
ai.budget.exceeded Budget Service Query Service, Notifier 7 days

Response Caching

A semantic cache avoids calling the LLM for near-duplicate queries:

sequenceDiagram
    participant QS as Query Service
    participant CACHE as Semantic Cache
    participant EMBED as Embedding Service
    participant LLM as LLM

    QS->>EMBED: embed(query)
    QS->>CACHE: lookup(query_vector, threshold=0.95)
    alt Cache hit (similarity > 0.95)
        CACHE-->>QS: cached_response + age
        QS->>QS: validate freshness (< 24h for policy docs)
        QS-->>CLIENT: cached response (latency: ~50ms)
    else Cache miss
        QS->>LLM: full pipeline
        LLM-->>QS: response
        QS->>CACHE: store(query_vector, response, ttl=86400)
        QS-->>CLIENT: fresh response
    end

Cache key: embedding vector of the query (not exact string match) — similar queries hit the same cache entry.

Cache invalidation: When a source document is updated and re-indexed, all cache entries that used chunks from that document are invalidated. The cache stores {chunk_ids_used} per entry for targeted invalidation.

Observability and Evaluation

Key Metrics

Metric Alert Threshold
ai.query.latency_p99_ms > 10,000 ms
ai.retrieval.latency_p99_ms > 500 ms
ai.query.block_rate > 2% (sudden spike = attack)
ai.query.error_rate > 1%
ai.cache.hit_rate < 20% (optimization opportunity)
ai.token.daily_cost_usd > $2,000/day
ai.retrieval.avg_relevance_score < 0.7 (retrieval quality degrading)
ai.user.thumbs_down_rate > 5% (quality issue)
ai.document.index_lag_minutes > 30 min (indexing backlog)

Evaluation Pipeline

LLM output quality cannot be measured with traditional metrics (precision, recall). Use:

Human feedback signals:

  • Thumbs up/down on each response
  • Flagging incorrect or harmful responses
  • Annotation for ground truth answers on sampled queries

Automated evaluation (LLM-as-judge):

  • Use a separate LLM to score responses on: factual accuracy, groundedness (is the answer supported by retrieved chunks?), relevance, helpfulness
  • Run on 5% sample of production queries

Retrieval evaluation:

  • Mean Reciprocal Rank (MRR) of correct source in retrieved chunks
  • NDCG@5 on benchmark query sets with known correct sources

When prompt changes or model updates are deployed, run the evaluation benchmark before and after and require that scores do not regress.

Design Trade-offs

1. RAG vs Fine-Tuned Model

Option A (RAG): Keep the base LLM unchanged. Inject relevant knowledge via retrieval at query time.

Option B (Fine-tuning): Train the LLM on company data. Knowledge is baked into model weights.

Decision: RAG for all enterprise use cases. Fine-tuning is expensive (~$1K–$100K), slow (days to train), and makes knowledge updates require a full retraining cycle. When an employee updates a policy document, RAG reflects it in minutes (re-index). Fine-tuning cannot reflect it until the next training run. RAG also provides citation — where did the answer come from? Fine-tuning embeds knowledge invisibly into weights, making it impossible to attribute or verify.

2. Shared Vector Store vs Per-Tenant Vector Store

Option A (Shared, namespace isolation): All tenants share a single vector store deployment; namespaces enforce isolation.

Option B (Per-tenant deployment): Each tenant gets a dedicated vector store instance.

Decision: Shared with namespace isolation for most tenants; dedicated instances for tenants with strict data residency or compliance requirements. Namespace isolation in a shared deployment (Qdrant, Pinecone) provides strong logical isolation at dramatically lower infrastructure cost. Per-tenant deployments are available as a premium tier for regulated industries (healthcare, government) with data residency requirements.

3. External LLM API vs Self-Hosted Model

Option A (External API): Azure OpenAI, Anthropic. Highest quality, managed, per-token pricing.

Option B (Self-hosted): Llama-3, Mistral on company infrastructure. Fixed cost, full data control.

Decision: External API as primary with self-hosted as fallback for sensitive data. External LLM APIs provide state-of-the-art quality with no operational overhead. The concern is data residency — queries containing sensitive employee or customer data should not leave the company's cloud boundary. The LLM router sends PII-sensitive queries to a self-hosted model on the company's VPC. Non-sensitive queries use the external API for quality.

4. Streaming vs Batch Response

Option A (Streaming, Server-Sent Events): First token delivered in < 1 second. User perceives fast response even if total generation takes 5 seconds.

Option B (Batch): Wait for complete response before sending. Simpler implementation.

Decision: Always stream. Users will not wait 5 seconds staring at a loading spinner. LLM responses feel immediate when the first word appears in under a second. Streaming is mandatory for any AI assistant that generates responses longer than a few sentences. The engineering cost is modest — SSE is well-supported in all major frameworks.

Common Interview Mistakes

  1. Treating RAG as a simple search-and-inject operation. "We retrieve documents and put them in the prompt" misses chunking strategy, hybrid search, re-ranking, query rewriting for follow-up questions, context window budgeting, and cache invalidation. Each of these has significant impact on quality.

  2. Ignoring multi-tenancy entirely. Enterprise AI is inherently multi-tenant. An answer that doesn't address how vectors from one tenant are isolated from another, or how access roles filter retrieval, is not an enterprise design.

  3. Not discussing the context window budget. Dumping 20 retrieved chunks into the prompt without managing token limits will fail at runtime. The context builder and its budget management logic must be discussed explicitly.

  4. Forgetting cost. LLM API costs are significant at scale. An enterprise AI assistant that doesn't discuss token budgets, cost per query, caching strategy, and model routing will run over budget in production.

  5. Proposing fine-tuning for knowledge updates. Proposing to fine-tune the model every time a policy document changes shows a misunderstanding of how RAG works. RAG is the correct choice for dynamic enterprise knowledge.

  6. Missing the safety filter layer. An enterprise AI that can be jailbroken into revealing another tenant's data, or that repeats PII from retrieved documents, is a compliance failure. Input and output safety filters must be part of the design.

  7. Describing retrieval as a single vector lookup. Advanced retrieval involves embedding the query, hybrid dense+sparse search, re-ranking, and query rewriting for conversational follow-ups. A design that only mentions "vector similarity search" is incomplete.

  8. Not addressing hallucination. When the LLM generates a confident but incorrect answer, the system has no way to detect it without an attribution check. Output grounding verification (does every claim trace to a retrieved chunk?) and user feedback mechanisms (thumbs down) are required components.

Summary

flowchart LR
    A[User Query] --> B[Input Safety Filter]
    B -->|safe| C[Query Rewriting]
    C --> D[Hybrid Retrieval]
    D --> E[Re-ranking]
    E --> F[Context Builder]
    F --> G[LLM Router]
    G --> H[LLM API]
    H --> I[Output Safety Filter]
    I --> J[Streaming Response + Citations]
    J --> K[Audit Log + Cache]
    B -->|blocked| L[Policy Error]
    I -->|blocked| M[Fallback Response]

Design Principles:

  • RAG over fine-tuning for enterprise knowledge — dynamic documents require real-time indexing, not periodic model retraining; RAG is the only viable architecture for frequently changing organizational knowledge
  • Multi-tenancy must be enforced at the storage layer — application-level filtering is insufficient; vector store namespaces and access role filters at the index level are required
  • Every query has a cost — design token budgets, semantic caching, and model routing from day one; LLM costs will exceed expectations without explicit controls
  • Streaming is mandatory — users will not wait for complete responses; first-token latency under 1 second is a non-negotiable UX requirement
  • Safety is a pipeline, not a feature — input filtering, output filtering, and attribution checking must be separate stages with independent failure modes; a single safety check is insufficient
  • Evaluation is continuous — LLM quality is probabilistic and degrades with model updates, prompt changes, and knowledge drift; an automated evaluation pipeline is required to maintain quality in production