Skip to main content

Case Study: Real-Time AI Search Engine

The Problem​

A fintech startup needs to build a real-time market intelligence platform that lets analysts ask natural language questions about live market data, news, and company filings.

Constraints given in the interview:

  • Data freshness: queries must reflect information from the last 5 minutes
  • Scale: 10,000 concurrent users, 50,000 queries/hour
  • Accuracy: financial data cannot be hallucinated
  • Latency: p95 response time under 3 seconds

The Interview Question​

"Design a system that lets users ask 'What is the sentiment around Tesla in the last hour?' and get an accurate, sourced answer in under 3 seconds."

Solution Architecture​

Key Design Decisions​

1. Why Kafka for Ingestion?​

The interviewer wants to know you understand streaming vs batch.

Answer: Kafka provides exactly-once delivery and allows multiple consumers. We have one consumer writing to the vector DB and another to Elasticsearch. If the vector indexing falls behind, the full-text index still serves queries. This is the dual-write pattern for resilience.

2. Why Hybrid Search (Vector + Full-Text)?​

Answer: Financial queries mix semantic ("sentiment around Tesla") with keyword ("TSLA 10-K filing"). Pure vector search would miss exact ticker matches. We use Reciprocal Rank Fusion (RRF) to combine results.

3. Why GPT-4o-mini Instead of GPT-4o?​

Answer: For a 3-second p95 latency target at 50K queries/hour, we need fast generation. GPT-4o-mini gives us 100+ tokens/second vs 40 tokens/second for GPT-4o. The reranker handles accuracy; the LLM only synthesizes already-verified content.

Handling the Freshness Requirement​

The hardest part of this problem is ensuring the index reflects data from the last 5 minutes.

Solution: TTL-Based Indexing

# Each document gets a timestamp field
doc = {
    "content": "Tesla announces new factory...",
    "timestamp": datetime.now(UTC),
    "source": "Reuters",
    "ttl_hours": 24  # Auto-delete after 24 hours
}

# Query filters to last N minutes
def search_recent(query: str, minutes: int = 60):
    cutoff = datetime.now(UTC) - timedelta(minutes=minutes)
    return vector_db.search(
        query=query,
        filter={"timestamp": {"$gte": cutoff}}
    )

Cost Analysis​

ComponentMonthly Cost (at 50K queries/hour)
Kafka (MSK)$2,500
Qdrant (managed)$1,800
Elasticsearch$2,000
GPT-4o-mini (generation)$3,500
Cross-encoder reranking$800
Total$10,600/month

Interview Follow-Up Questions​

Q: How do you prevent hallucinated financial data?

A: Three layers: (1) The LLM only summarizes retrieved content, never generates facts. (2) Every claim must cite a source document. (3) A post-generation validator checks that any number in the response exists verbatim in a source.

Q: What if Kafka falls behind during a news spike?

A: We implement backpressure with consumer lag monitoring. If lag exceeds 2 minutes, we shed load on the ingestion side using sampling. Real-time queries hit a "recent" index with only the last hour of data; batch jobs backfill the full index.

Key Takeaways for Interviews​

  1. Real-time AI search requires streaming infrastructure, not batch ETL
  2. Hybrid search (semantic + keyword) outperforms pure vector for structured domains
  3. Latency budgets drive model selection: use fast models for synthesis, save expensive models for reasoning
  4. Freshness is a filter, not a feature: implement at the index level, not the prompt level

Related chapters: Hybrid Search, Serving Infrastructure