A faster answer can still block an AI pilot.
You're preparing the expansion review for an internal-knowledge assistant. Some answers take less work. One answer contains material the test user shouldn't have received. Your decision packet needs to preserve both observations.
Evidence boundary. The product observations below come from public documentation. We have not tested a Glean or Microsoft customer deployment. The Northstar organization, Kestrel assistant, six observations and timings are fictional. The review method and recommendations are TTS's judgment.
Three questions need separate answers
An answer can match a retrieved document and still be wrong for the person or task. The document could be obsolete; the user might lack access; the final answer might require enough correction to erase any time benefit. We propose reviewing those questions separately.
| Evidence reviewed | What it supports | What to test in your workflow |
|---|---|---|
| Microsoft's RAG evaluation guide Vendor guidance | Evaluation separates retrieval and answer behavior; metric choice depends on the workload. | Check whether the approved source was retrieved, then whether the answer used it correctly. Record both failures. |
| Glean's search permissions FAQ Vendor documentation | Glean describes collecting content, activity and identity information; identity information informs result visibility. | Use two synthetic identities and a restricted document. Inspect answer text, previews and citations; repeat after revocation. |
| Glean's document verification guide Vendor documentation | A user can verify content accuracy and currency, with a verifier, date and reminders. | Put a current policy beside an obsolete conflict. Check which source wins and whether the reviewer can reach its owner. |
| Anthropic's evaluation guide Provider engineering guidance | It distinguishes an agent's transcript from the resulting state in its environment. | Check the actual boundary: what was exposed or changed. An answer saying it respected access is insufficient evidence. |
These sources help define useful checks. They don't supply your acceptance criteria, prove your configuration passed, or establish a comparison between vendors. For connector-specific details, read our Glean workflow research and follow its original sources.
Reproduce the review on six fictional cases
The example reuses our published fictional decision packet. Each case has an expected result, an observation, active time, an owner and a next test. A pass refers to the final reviewed answer, including any corrections.
- K-01 · Current policy — pass. One wording correction; final answer met the agreed standard.
- K-02 · Conflicting sources — fail. Used a superseded exception.
- K-03 · Restricted document — fail. A test identity received a restricted excerpt. Root cause unknown. Boundary check.
- K-04 · Out-of-scope request — pass. Named escalation route appeared. Boundary check.
- K-05 · Unavailable dependency — pass. Retrieval timed out; reviewer used the manual runbook.
- K-06 · Cited answer — pass. Reviewer corrected step order; final answer passed.
Add the current active time for K-01, K-04, K-05 and K-06: 14 + 9 + 11 + 13 = 47 minutes. Their assisted active time is 9 + 7 + 8 + 10 = 34 minutes. The difference is 13 minutes. Reviewer time is a subset of assisted time, so adding it again would double count work.
Across all six observations the raw difference is 17 minutes, but two outputs fail the stated criteria. Reporting that figure alone hides why the workflow cannot expand. Even the 13-minute difference on passing cases is a selected, tiny synthetic sample; it cannot establish expected savings. A real study needs representative case frequency, comparable work and repeated observations.
K-05 also shows why active and elapsed time need separate records. Active work falls from 11 to 8 minutes while elapsed time rises from 25 to 26 during a retrieval timeout. The companion tool compares active time only. Keep elapsed time, licensing, usage, setup and maintenance costs in the broader evaluation log.
Change the result and inspect what follows
Open K-03 in the review tool. It starts as a failed access check, which blocks expansion. Marking it pass removes that specific failure signal, but K-02 still needs resolution. Marking every case pass never creates an approval: the final state still asks for a human decision review.
The tool checks entered statuses, missing fields and timing consistency. It cannot inspect evidence, judge whether the reviewer applied the standard correctly, or confirm a control. If you change a fictional observation, it remains a fictional example. Use “Start my own review” to remove the sample observations and timings.
Turn the failure into an owned retest
For K-03, ask the identity and access owner to inspect the current permissions, connector behavior and test logs. A stale identity snapshot or an incorrect filter could explain the result, but neither is established by the fictional observation. Write the unknown cause into the packet.
Retest with a revoked identity and a newly restricted identity. Keep the original failure, the configuration versions, the change made and the new result. Ask the knowledge owner to resolve K-02's conflicting policy and repeat it with additional conflicts. Keep the manual fallback available while those investigations continue.
What this adds to your usual AI assistant
A capable assistant can help create cases, critique a plan and draft a packet. TTS supplies a stable worked failure, inspectable calculations, cited product mechanisms and the same evidence fields across the example, tool and export. Those are useful properties, but we haven't established that they save practitioners time or produce better decisions.
Try both on the same sanitized decision. Keep the output from your normal process or assistant, then use this review. Compare which produces a specific missing requirement, a test you can assign, or a clearer decision for the meeting. Count the extra editing each output requires.
Bring a reviewable packet to the team
Download your packet, identify the decision owner and agree on a checkpoint. Add cases for the roles, sources and actions your own workflow contains. Six starting cases cannot cover an enterprise deployment.
Open the knowledge pilot review →
If the decision is stuck, the proposed pilot decision review offers a bounded research and planning engagement. If you try the free packet in a team discussion, tell us which decision or missing requirement it helped uncover.