YundaDeskThe most common mistake when starting a knowledge base project isn’t delay — it’s trying to make it perfect in one pass. Teams try to launch before policies are reconciled, and let AI answer customers before anyone has reviewed the answers. The result: AI support that hedges, answers slowly, and needs constant correction from agents.
A more reliable approach is to break the work into 14 days. Spend the early days auditing and cleaning content, the middle days structuring answers and loading them into the knowledge base, and the final days verifying with a small test run. AI support should pull from the knowledge base for every answer, and hand off to a human agent when it can’t find a confident answer, when the customer asks for one, or when the topic is high-risk — that boundary needs to be set on day one, not patched in after launch.
Days 1–2: Audit what you already have
Before writing anything new, lay out everything that already exists.
- Storefront pages: shipping, returns, FAQ, size guides
- Support handbooks, script docs, internal wikis
- Conversation logs from your last peak season and the past month
- Temporary promotion rules (flag which ones have expired)
Tag each source with one of three statuses: usable as-is (accurate, currently valid), needs an update (outdated or conflicts with another document), or discard (expired campaign pages, verbal-only promises from individual agents). A common trap for cross-border sellers is having three different versions of the same return policy — one on the website, one in the handbook, one in a veteran agent’s memory. The job on days 1–2 is to surface those conflicts, not merge them prematurely.
Days 3–4: Reverse-engineer your top 40 questions from real conversations
Instead of guessing what customers ask, export conversation logs from your last peak season or recent weeks and cluster them by topic.
- Export conversations and strip out small talk, ads, and system notifications.
- Cluster by shipping, returns, sizing, promo codes, pre-sale questions, and complaints.
- Pull the most frequent questions from each cluster until you have around 40.
- Flag which ones AI can answer directly and which must route to a human — refunds, compensation, price changes, and escalated complaints.
You’re not aiming to cover every possible question with these 40 — you’re catching the loudest, most repeated ones first. This step also bakes the human-handoff boundary into the knowledge base design from the start, instead of discovering gaps after launch.
How agent first response can fall after the 14-day rollout (illustrative)
Days 5–7: Rewrite content into structured answers
Raw policy documents and internal handbooks usually aren’t written for AI to consume directly — too long, too many buried conditions, too much internal jargon. Days 5–7 are for rewriting that content into answers both AI and agents can use as-is.
| Content type | Build method | What to get right |
|---|---|---|
| Policies | Upload documents | State scope and exceptions clearly; mark outdated versions as expired |
| Product info | Upload documents + manual Q&A | Sizing, materials, compatible models — the more complex the product, the more it needs structured Q&A |
| Scenarios | Manual Q&A | Write questions the way customers actually phrase them, not formal policy language |
| Public site content | Website crawl | Confirm pages are current before crawling; set expiry on campaign pages |
Write both the question and the answer close to how a real conversation sounds, not lifted verbatim from a policy PDF. AI support matches on meaning, not keyword density — the closer your entries read to real conversation, the better the retrieval and the answer.
Days 8–9: Mark high-risk boundaries and write handoff triggers
These two days are for the part of the knowledge base that isn’t about teaching AI to act, but teaching it when to stop. Refunds, compensation, price changes, escalated complaints, threats of a bad review, anything with legal language — none of these should be something AI resolves on its own.
Write each high-risk scenario as a chain: trigger phrase or situation, information AI should collect first, then a handoff summary for the human agent. That way, even when a customer is upset or phrasing things indirectly, AI still knows exactly where to stop.
Days 10–11: Load the knowledge base and cross-check for conflicts
With rewritten content ready, start loading it in and do two things simultaneously:
- Check each entry against the conflict list from days 1–2 to confirm only one current version of each policy survives.
- Check whether the same question got captured through multiple sources — website crawl, uploaded documents, manual Q&A — with answers that don’t quite agree. This is one of the most common causes of AI giving wrong answers.
Loading isn’t a one-shot bulk import. Load in batches by topic and spot-check a handful of entries after each batch — catching formatting or semantic drift early is far cheaper than fixing it after everything is in.
Days 12–13: Run a small test and have agents review AI’s answers
For these two days, run AI support against the top 40 questions in a small-traffic or internal test environment, and have agents review each answer:
- Is the answer accurate, and does it cite the current, valid version in the knowledge base?
- Did the scenarios that should route to a human actually get routed, instead of AI answering anyway?
- Does the tone match the brand, and is any required disclosure missing?
When agents spot a problem, the fix isn’t to edit the knowledge base directly. Following the controlled learning loop, an agent’s correction generates a learning suggestion pending review, which goes to the owner’s review queue. Only after the owner approves does it take effect and get saved as a skill or knowledge update — traceable, testable, and reversible with one click. It’s a slightly slower path than a direct edit, but it avoids the chaos of anyone being able to change policy answers unilaterally.
Day 14: Sign off and set the first-week review rhythm
On launch day, run through three checks:
- Confirm every high-risk boundary is active — refunds, compensation, and price changes route to a human without exception
- Confirm no conflicting policy versions remain in the knowledge base
- Set a fixed time each day for the first week to spot-check conversations, rather than waiting for a customer complaint to trigger a review
The first week is the most fragile stretch for a new knowledge base — new customers will keep surfacing edge cases the top 40 didn’t cover. Treat every instance of “AI couldn’t answer, agent stepped in” from day 15 onward as routine input, feeding the same learning-suggestion workflow to fill the gaps. The knowledge base doesn’t finish on day 14 — that’s when it shifts into ongoing maintenance.
Fourteen days isn’t a rigid rule — audit and loading effort scale with store size. But the order matters: surface conflicts first, prioritize by real conversations, write high-risk boundaries as hard rules, and have agents review before launch. For more on how the knowledge base and AI support work together, see how the knowledge base feeds AI and where the AI-first, human-backed boundary sits.