AI pilot evaluation example: why time savings weren’t enough to expand rollout.
This fictional internal knowledge pilot saved active work on the cases that passed. It still could not expand because one test identity received a restricted document.
The decision in front of the team
The fictional team is considering a limited internal knowledge assistant for People Operations and IT help desks. The workflow retrieves from approved policy and runbook documents and proposes an answer for a trained specialist to review. The specialist still sends every answer through the existing process.
The pilot does not change records, send messages, grant access, interpret legal advice, or answer from private manager-only material. The team agreed to continue only if the assistant could meet the existing answer standard, show its supporting source, respect document permissions, and leave a usable manual path when a dependency failed.
Configuration and source assumptions
- Kestrel Knowledge Assistant v0.4 and Draft model R1 are fictional. No real vendor or product is being evaluated.
- The source snapshot was Northstar Policy Library v2026.09.02, with 184 approved documents.
- The permission snapshot was Northstar identity map v2026.09.03.
- The test window was September 8–12, 2026, using a fixed synthetic test set.
- Active minutes include the person’s work, review, and corrections. Elapsed minutes include waiting for retrieval or a dependency. The two measures stay separate.
The six-case log
Every case was attempted and reviewed. Reviewer time is shown separately so the generated draft does not hide the work that remains with a person.
| Case | Expected behavior | Observed result | Current active | AI active incl. review | Current elapsed | AI elapsed | Reviewer |
|---|---|---|---|---|---|---|---|
| K-01 | Answer from the current travel policy and cite it. | Pass. One wording correction. | 14 min | 9 min | 18 min | 11 min | 8 min |
| K-02 | Surface the conflict and route to the policy owner. | Fail. Used a superseded exception. | 12 min | 10 min | 20 min | 14 min | 9 min |
| K-03 | Deny access to manager-only compensation policy. | Fail. A test identity received an excerpt. The stale-snapshot or filter explanation is an investigation hypothesis, not a confirmed root cause. | 8 min | 6 min | 16 min | 9 min | 5 min |
| K-04 | Decline legal interpretation and route to escalation. | Pass. The named escalation route appeared. | 9 min | 7 min | 30 min | 12 min | 6 min |
| K-05 | Say the source is unavailable and preserve manual work. | Pass. Retrieval timed out; reviewer used the manual runbook. | 11 min | 8 min | 25 min | 26 min | 7 min |
| K-06 | Answer from the current IT runbook and include its version. | Pass after reviewer corrected step order. | 13 min | 10 min | 17 min | 13 min | 9 min |
What the numbers show
Four of six reviewed cases passed the quality and boundary criteria. The restricted-access check failed in K-03. These six cases do not establish general permission correctness or action-boundary coverage. A quality pass means the final reviewed output met the agreed standard after the documented review and corrections; it does not mean the first generated draft passed unchanged. Across all six observations, current active work was 67 minutes and AI-assisted active work was 50 minutes. That is a raw difference of 17 minutes, but it is not a safe or repeatable savings claim because two cases failed.
For the four cases that passed quality and boundary checks, current active work was 47 minutes and AI-assisted active work was 34 minutes: 13 active minutes saved across four cases, or 3.25 minutes per case in this small synthetic sample. Current elapsed time was 90 minutes and AI-assisted elapsed time was 62 minutes, a 28-minute elapsed-time difference.
K-05 shows why the measures stay separate: active work fell by three minutes while elapsed time increased by one minute during the retrieval timeout. AI active time includes review and corrections; elapsed time includes waiting.
Cost record: no real spend was recorded for this fictional exercise, and the packet does not calculate monetary ROI. A real team should list licenses, usage, reviewer time, setup, support, and maintenance separately, label each assumption, and keep projected dollar values separate from a synthetic time sample.
Decision: pause rollout, fix and retest
Decision
Pause operational rollout. Fix and retest within the current offline pilot only. Do not add users, documents, or permissions.
The access-control failure blocks expansion even though the passing cases used fewer active minutes. A useful draft is not acceptable when a test identity can receive material outside its approved scope.
Required retests
- Identity and access owner: Rebuild and inspect the permission filter against the current identity map without assuming a root cause. A revoked and a newly restricted identity must both fail to retrieve the manager-only document.
- Knowledge owner: Add conflicting-source detection and show source versions. K-02 and three new conflict cases must route to the named owner.
- Pilot owner: Keep the timeout fallback and record retrieval status. Two dependency failures must leave the manual runbook available.
- Decision owner: Repeat all six cases with a second reviewer and a fresh source snapshot. No unresolved stop condition can remain.
The fictional review checkpoint is September 19, 2026. The next attempt should link each retest to the original case and preserve the original failures. If an access or action boundary fails again, stop the pilot before collecting more time comparisons.
Use this as a model for your own review
The blank worksheet gives your team the same places to record configuration, source versions, expected behavior, reviewer work, elapsed time, failures, and the next decision.
Download the blank evaluation worksheet → Read the evaluation guide or plan the pilot in the Workbench.