Case Study: AI-Powered Customer Support
The Problem
An e-commerce company handles 2 million support tickets per month. They want an AI system that can automatically resolve 60% of tickets without human intervention, while seamlessly escalating complex issues.
Constraints given in the interview:
- 24/7 operation across 12 languages
- Must integrate with existing Zendesk and Salesforce
- Cannot make false promises (refunds, shipping dates)
- Human agents must be able to take over mid-conversation
- Cost target: $0.05 per resolved ticket
The Interview Question
"Design a customer support AI that handles 'Where is my order?' automatically but knows when to escalate 'I want to sue you for fraud' to a human."
Solution Architecture
Key Design Decisions
1. Three-Tier Routing (Auto / Hybrid / Escalate)
Answer: Not all tickets are equal. We classify into three paths:
| Path | Criteria | Example | Human Involvement |
|---|---|---|---|
| Auto | High confidence, low risk | "Where is my order?" | None |
| Hybrid | Medium confidence or medium risk | "I want a refund" | Reviews AI draft |
| Escalate | Legal, threats, VIP, low confidence | "This is fraud" | Full human handling |
2. Tool-Based Resolution, Not Pure Generation
Answer: The AI does not "know" where the order is. It calls the Order API tool. This is critical for accuracy:
@tool
def get_order_status(order_id: str) -> dict:
"""Retrieve real-time order status from OMS."""
order = oms_client.get_order(order_id)
return {
"status": order.status,
"shipped_date": order.shipped_at,
"estimated_delivery": order.eta,
"tracking_url": order.tracking_url
}
The LLM orchestrates tools but never fabricates data.
3. Why Safety Check Before Send?
Answer: Even auto-resolved tickets go through a safety filter:
- Promise Detection: Flags statements like "I guarantee" or "We will pay"
- Sentiment Mismatch: Catches if AI sounds happy when customer is angry
- PII Leak: Ensures no internal notes or other customer data appear
- Competitor Mention: Flags if AI recommends a competitor
The Escalation Intelligence
The hardest part is knowing when to escalate. We use a confidence score with multiple signals:
Key insight: A VIP customer asking a simple question still goes to Hybrid path because the cost of a mistake is higher.
Multilingual Support
12 languages without 12 separate models:
Why not native multilingual models?
Cost. GPT-4o handles all 12 languages well. Using specialized models per language would require 12 deployments. Translation adds latency but keeps infrastructure simple.
Human Takeover (Mid-Conversation)
When a human takes over, they need full context:
def handoff_to_human(conversation_id: str, agent_id: str):
conversation = get_conversation(conversation_id)
# Generate summary for human agent
summary = llm.generate(f"""
Summarize this conversation for a human agent:
- Customer issue
- What AI already tried
- Why escalation happened
Conversation:
{conversation.messages}
""")
# Create handoff package
return {
"summary": summary,
"customer_sentiment": conversation.sentiment,
"attempted_solutions": conversation.tool_calls,
"full_transcript": conversation.messages,
"customer_tier": conversation.customer.tier
}
Cost Analysis
| Component | Cost per Ticket |
|---|---|
| Intent classification (GPT-4o-mini) | $0.002 |
| Tool calls (Order API, FAQ search) | $0.001 |
| Response generation (GPT-4o-mini) | $0.008 |
| Safety check | $0.003 |
| Translation (if needed, 30% of tickets) | $0.004 |
| Average total | $0.018 |
At a 60% auto-resolution rate: $0.03 per resolved ticket (well under $0.05 target)
Interview Follow-Up Questions
Q: What if the AI keeps apologizing but never actually helps?
A: We track "resolution effectiveness" not just "response sent." If a customer replies again within 24 hours on the same issue, that ticket is marked as "unresolved" and the AI pattern is flagged for review. We also run weekly analysis: "What phrases correlate with customer follow-ups?"
Q: How do you handle a customer who insists on talking to a human?
A: Explicit escalation phrases ("talk to a human", "speak to manager") trigger immediate handoff regardless of confidence score. We never argue with escalation requests.
Q: What about customers who try to jailbreak the support AI?
A: Input sanitization plus strict tool-only responses. The AI cannot be prompted to reveal system prompts because it does not generate free-form answers: it calls tools and summarizes their outputs. The system prompt is also extremely narrow: "You help with order issues for [Company]. You cannot discuss other topics."
Key Takeaways for Interviews
- Tiered routing balances automation with risk: not every ticket should be auto-resolved
- Tool-based grounding prevents hallucination: the AI retrieves facts, it does not generate them
- Confidence is multi-dimensional: intent clarity + sentiment + customer tier + topic risk
- Human handoff needs context: summarize, do not just dump the transcript
Related chapters: Human-in-the-Loop Patterns, Guardrails Implementation