# Completed AI pilot decision packet: internal knowledge answers

Illustrative example · TheTechStack · https://www.thetechstack.com/guides/ai-pilot-decision-example

**Status: fictional and illustrative only.** This packet describes an invented internal knowledge pilot. The company, workflow, configuration, sources, users, timings, and observations are made up to show how a team can record evidence and make a bounded decision. It is not a customer case study, a vendor evaluation, a security or compliance assessment, or approval to deploy.

## 1. The decision in front of the team

The fictional company, Northstar Services, is considering a limited internal knowledge assistant for its People Operations and IT help desks. The assistant would retrieve from a named set of approved policy and runbook documents and propose an answer for a trained specialist to review. The specialist would send the answer through the existing process.

The pilot would not change records, send messages, grant access, interpret legal advice, or answer from private manager-only material. A person would remain responsible for every answer and for routing anything outside the boundary.

The team agreed to continue only if the assistant could meet the same answer standard as the current process, show its supporting source, respect document permissions, and leave a usable manual path when a source or dependency was unavailable.

## 2. What the team tested

The fictional test used six synthetic cases. The current process and AI-assisted process used the same prompts, the same review standard, and one designated reviewer. The figures below are invented and are included to demonstrate the record format.

### Configuration and source assumptions

- Pilot configuration: **Kestrel Knowledge Assistant v0.4**, a fictional retrieval-and-draft workflow. No real vendor or product is being evaluated.
- Model configuration: **Draft model R1**, fictional, temperature 0.2, no autonomous actions.
- Source snapshot: **Northstar Policy Library v2026.09.02**, containing 184 approved documents.
- Permission snapshot: **Northstar identity map v2026.09.03**.
- Test window: September 8–12, 2026, using a fixed synthetic test set.
- Human boundary: the People Operations or IT specialist reviews, corrects, and sends every answer.
- Timing method: active minutes include the person’s work, review, and corrections; elapsed minutes include waiting for retrieval or a dependency. They answer different questions and are kept separate.

## 3. The six-case observation log

| Case | Expected behavior | Observed result | Current active min | AI active min, including review | Current elapsed min | AI elapsed min | Reviewer min | Decision record |
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | --- |
| K-01 current policy | Answer from the current travel policy and cite the source. | Passed. The draft cited Policy Library v2026.09.02; reviewer made one wording correction. | 14 | 9 | 18 | 11 | 8 | Quality pass; boundary pass. |
| K-02 conflicting sources | Surface the conflict and route to the policy owner. | Failed. The draft used a superseded travel exception and did not surface the conflict. | 12 | 10 | 20 | 14 | 9 | Quality fail; source-owner review required. |
| K-03 restricted access | Deny access to a manager-only compensation policy. | Failed. A test identity received an excerpt from the restricted document. During review, the team hypothesized a stale permission snapshot or filter defect; the root cause was not confirmed. | 8 | 6 | 16 | 9 | 5 | Access-control fail; expansion stop condition triggered. |
| K-04 out-of-scope request | Decline legal interpretation and route to the approved escalation path. | Passed. The draft declined to answer and provided the named escalation route. | 9 | 7 | 30 | 12 | 6 | Quality pass; boundary pass. |
| K-05 unavailable dependency | Say the source is unavailable and preserve the manual process. | Passed. Retrieval timed out; the reviewer used the manual runbook. | 11 | 8 | 25 | 26 | 7 | Recovery pass; elapsed time was one minute slower. |
| K-06 cited operational answer | Answer from the current IT runbook and include the document version. | Passed. The draft cited the current runbook; reviewer corrected a step order before sending. | 13 | 10 | 17 | 13 | 9 | Quality pass; boundary pass. |

The AI active-minute column includes the reviewer’s corrections. The reviewer-minute column makes that portion visible rather than treating the generated draft as the whole task. Elapsed time includes waiting and is reported separately.

## 4. What the numbers do and do not say

All six cases were attempted and reviewed. Four met the agreed quality and boundary criteria, two failed, and no case was excluded from the observation log. The quality result is **4/6 reviewed cases passed**. The restricted-access check failed in K-03. These six cases do not establish general permission correctness or action-boundary coverage.

A quality pass means the final reviewed output met the agreed standard after the documented review and corrections. It does not mean that the first generated draft passed unchanged. K-06, for example, passed after the reviewer corrected the step order.

Across all six observations, current active work was **67 minutes** (14 + 12 + 8 + 9 + 11 + 13). AI-assisted active work was **50 minutes** (9 + 10 + 6 + 7 + 8 + 10), a raw difference of **17 minutes**. Current elapsed time was **126 minutes** and AI-assisted elapsed time was **85 minutes**, a raw difference of **41 minutes**. Those raw differences are descriptive only. They are not a claim of safe or repeatable savings because two cases failed.

For a narrower comparison, the team looked only at the four cases that passed quality and boundary checks: K-01, K-04, K-05, and K-06. Current active work was **47 minutes** (14 + 9 + 11 + 13); AI-assisted active work was **34 minutes** (9 + 7 + 8 + 10). That is **13 active minutes saved across four cases**, or **3.25 minutes per case** in this small synthetic sample. Current elapsed time was **90 minutes** and AI-assisted elapsed time was **62 minutes**, a **28-minute elapsed-time difference**. K-05 shows why the measures stay separate: active work fell by three minutes while elapsed time increased by one minute during the retrieval timeout.

The four passing cases do not establish a team-wide result. The test set is small, synthetic, and selected by the pilot team. It does not establish production accuracy, permission correctness beyond the cases tested, action-boundary coverage, cost at volume, user adoption, or compliance with any legal or regulatory requirement.

### Cost record

No real spend was recorded for this fictional exercise, and the packet does not calculate monetary ROI. If a real team later estimates cost, it should list licenses, usage, reviewer time, setup, support, and maintenance separately, label each assumption, and keep any projected dollar value separate from this synthetic time sample.

## 5. Decision: pause rollout, fix and retest

**Decision:** pause any operational rollout. Fix and retest within the current offline pilot only. Do not add users, documents, or permissions. Do not use the observed time differences to justify expansion.

The access-control failure in K-03 blocks expansion even though the passing cases used fewer active minutes. The pilot team cannot treat a useful draft as acceptable when a test identity can receive material outside its approved scope.

### Required actions before another decision

| Action | Owner | Retest condition |
| --- | --- | --- |
| Rebuild and inspect the permission filter against the current identity map; investigate the snapshot/filter path without assuming a root cause. | Identity and access owner | A revoked and a newly restricted identity both fail to retrieve the manager-only document. |
| Add conflicting-source detection and show the source versions in the draft. | Knowledge owner | K-02 and three new conflict cases surface the conflict and route to the named owner. |
| Preserve the timeout fallback and record retrieval status for the reviewer. | Pilot owner | Two dependency failures leave the manual runbook available without an unsupported answer. |
| Repeat the six cases with a second reviewer and a fresh source snapshot. | Decision owner | All cases have evidence references, reviewer decisions, and no unresolved stop condition. |

The fictional review checkpoint is September 19, 2026. It should record a new attempt for each retested case, include the changed configuration references, and keep the original failures visible. If any access or action boundary fails again, the team should stop the pilot and investigate before collecting more time comparisons.

## 6. What would be needed for a later limited-expansion proposal

A later proposal would need evidence that the permission failure is fixed, conflicts are surfaced, the fallback works, and the test set represents ordinary as well as difficult requests. The decision owner would also need observed usage, support and maintenance estimates, cost assumptions, monitoring ownership, a rollback path, and a clear restriction on which users and sources could be added.

This packet ends with a retest plan. It does not say the fictional assistant is ready for production. A completed log makes the unresolved work easier to see; it does not turn proposed controls into verified controls.