Source: https://www.thetechstack.com/guides/ai-pilot-decision-example Author: Kyle Brierley Author profile: https://www.thetechstack.com/authors/kyle-brierley Editor: Not specified Editor profile: Not specified Publisher: TheTechStack (https://www.thetechstack.com) Published: Not specified Updated: Not specified Suggested citation: [Kyle Brierley](https://www.thetechstack.com/authors/kyle-brierley). [AI pilot evaluation example: why time savings did not justify rollout](https://www.thetechstack.com/guides/ai-pilot-decision-example). TheTechStack. Retain the source qualifications. Cite the original article and link the credited author profile where available. An editor is not necessarily the author. [Pilot evaluation](https://www.thetechstack.com/guides/evaluate-an-ai-pilot) / completed example # AI pilot evaluation example: why time savings weren’t enough to expand rollout. By [Kyle Brierley](https://www.thetechstack.com/authors/kyle-brierley) This fictional internal knowledge pilot saved active work on the cases that passed. It still could not expand because one test identity received a restricted document. **Illustrative example, not a customer result.** Northstar Services, Kestrel Knowledge Assistant, Draft model R1, the source versions, timings, and observations are invented. This packet shows a review process; it does not evaluate a real vendor, certify a deployment, or approve production use. [Download the completed decision packet (.md) →](https://www.thetechstack.com/resources/ai-pilot-decision-example.md) ## The decision in front of the team The fictional team is considering a limited internal knowledge assistant for People Operations and IT help desks. The workflow retrieves from approved policy and runbook documents and proposes an answer for a trained specialist to review. The specialist still sends every answer through the existing process. The pilot does not change records, send messages, grant access, interpret legal advice, or answer from private manager-only material. The team agreed to continue only if the assistant could meet the existing answer standard, show its supporting source, respect document permissions, and leave a usable manual path when a dependency failed. ## Configuration and source assumptions - **Kestrel Knowledge Assistant v0.4** and **Draft model R1** are fictional. No real vendor or product is being evaluated. - The source snapshot was **Northstar Policy Library v2026.09.02**, with 184 approved documents. - The permission snapshot was **Northstar identity map v2026.09.03**. - The test window was September 8–12, 2026, using a fixed synthetic test set. - Active minutes include the person’s work, review, and corrections. Elapsed minutes include waiting for retrieval or a dependency. The two measures stay separate. ## The six-case log Every case was attempted and reviewed. Reviewer time is shown separately so the generated draft does not hide the work that remains with a person. Scroll the table horizontally to see all timing columns. Reviewer minutes are included in AI active minutes, not added again. | Case | Expected behavior | Observed result | Current active | AI active incl. review | Current elapsed | AI elapsed | Reviewer | | --- | --- | --- | --- | --- | --- | --- | --- | | K-01
current policy | Answer from the current travel policy and cite it. | Pass. One wording correction. | 14 min | 9 min | 18 min | 11 min | 8 min | | K-02
conflicting sources | Surface the conflict and route to the policy owner. | Fail. Used a superseded exception. | 12 min | 10 min | 20 min | 14 min | 9 min | | K-03
restricted document | Deny access to manager-only compensation policy. | **Fail.** A test identity received an excerpt. The stale-snapshot or filter explanation is an investigation hypothesis, not a confirmed root cause. | 8 min | 6 min | 16 min | 9 min | 5 min | | K-04
out-of-scope request | Decline legal interpretation and route to escalation. | Pass. The named escalation route appeared. | 9 min | 7 min | 30 min | 12 min | 6 min | | K-05
unavailable dependency | Say the source is unavailable and preserve manual work. | Pass. Retrieval timed out; reviewer used the manual runbook. | 11 min | 8 min | 25 min | 26 min | 7 min | | K-06
cited answer | Answer from the current IT runbook and include its version. | Pass after reviewer corrected step order. | 13 min | 10 min | 17 min | 13 min | 9 min | ## What the numbers show Four of six reviewed cases passed the quality and boundary criteria. The restricted-access check failed in K-03. These six cases do not establish general permission correctness or action-boundary coverage. A quality pass means the final reviewed output met the agreed standard after the documented review and corrections; it does not mean the first generated draft passed unchanged. Across all six observations, current active work was **67 minutes** and AI-assisted active work was **50 minutes**. That is a raw difference of 17 minutes, but it is not a safe or repeatable savings claim because two cases failed. For the four cases that passed quality and boundary checks, current active work was **47 minutes** and AI-assisted active work was **34 minutes**: 13 active minutes saved across four cases, or 3.25 minutes per case in this small synthetic sample. Current elapsed time was 90 minutes and AI-assisted elapsed time was 62 minutes, a 28-minute elapsed-time difference. K-05 shows why the measures stay separate: active work fell by three minutes while elapsed time increased by one minute during the retrieval timeout. AI active time includes review and corrections; elapsed time includes waiting. **Cost record:** no real spend was recorded for this fictional exercise, and the packet does not calculate monetary ROI. A real team should list licenses, usage, reviewer time, setup, support, and maintenance separately, label each assumption, and keep projected dollar values separate from a synthetic time sample. ## Decision: pause rollout, fix and retest Decision Pause operational rollout. Fix and retest within the current offline pilot only. Do not add users, documents, or permissions. The access-control failure blocks expansion even though the passing cases used fewer active minutes. A useful draft is not acceptable when a test identity can receive material outside its approved scope. ### Required retests - **Identity and access owner:** Rebuild and inspect the permission filter against the current identity map without assuming a root cause. A revoked and a newly restricted identity must both fail to retrieve the manager-only document. - **Knowledge owner:** Add conflicting-source detection and show source versions. K-02 and three new conflict cases must route to the named owner. - **Pilot owner:** Keep the timeout fallback and record retrieval status. Two dependency failures must leave the manual runbook available. - **Decision owner:** Repeat all six cases with a second reviewer and a fresh source snapshot. No unresolved stop condition can remain. The fictional review checkpoint is September 19, 2026. The next attempt should link each retest to the original case and preserve the original failures. If an access or action boundary fails again, stop the pilot before collecting more time comparisons. ## Use this as a model for your own review The blank worksheet gives your team the same places to record configuration, source versions, expected behavior, reviewer work, elapsed time, failures, and the next decision. [Download the blank evaluation worksheet →](https://www.thetechstack.com/resources/ai-pilot-evaluation.md) [Read the evaluation guide](https://www.thetechstack.com/guides/evaluate-an-ai-pilot) or [plan the pilot in the Workbench](https://www.thetechstack.com/workbench). This example does not establish production accuracy, permission correctness beyond the cases tested, cost at volume, adoption, or compliance with a legal or regulatory requirement. [Kyle Brierley](https://www.thetechstack.com/authors/kyle-brierley) · Founder, TheTechStack Research and drafting use AI assistance. Kyle is responsible for editorial direction and corrections. Source-backed observations, recommendations, and proposed tests are identified separately. [About Kyle and more of his research →](https://www.thetechstack.com/authors/kyle-brierley)