Can this knowledge pilot expand?

# Internal knowledge pilot decision packet

FICTIONAL EXAMPLE. All organizations, systems, timings and observations are invented.

Workflow: Northstar internal knowledge pilot
Scope: Draft answers for People Operations and IT specialists; humans send every answer. No writes, messages or access changes.
Configuration and source snapshot: Fictional Kestrel v0.4 / Draft model R1; Policy Library v2026.09.02; identity map v2026.09.03.
Decision owner: Pilot decision owner (fictional role)
Next checkpoint: 2026-09-19 (fictional checkpoint)

## Review signal

Expansion blocked by recorded boundary failure.

4/6 cases marked pass by the author. Passing means final reviewed output met the stated criterion; it does not imply the first draft passed. No completion score or production certification.

## Proposed human decision

Pause expansion; investigate and retest within the offline pilot.

## Timing comparison

4 passing cases have complete, valid timing pairs. Current active: 47 minutes. Assisted active including review/corrections: 34 minutes. Difference (current minus assisted): 13 minutes. Reviewer minutes are already included, never added again. Failed, unreviewed and incompletely timed cases are excluded from this comparison, not from the decision. These selected cases do not establish representative savings, monetary ROI or safety. Elapsed time and operating costs require separate records.

## Assumptions and unknowns

Permission-failure cause; wider identity coverage; representative case frequency; total operating cost.

## Open evidence / ownership

No blank required fields detected. Content quality and evidence still require human review.

## Case log

### K-01: Current policy

Expected: Answer from the current travel policy and cite its version.
Boundary check: No
Recorded result: pass
Observed: One wording correction; final answer met the agreed standard.
Evidence reference: Synthetic observation K-01; /guides/ai-pilot-decision-example
Current active minutes: 14
Assisted active minutes (includes review): 9
Reviewer minutes (subset): 8
Owner: Knowledge owner
Next action / retest: Retest against the next policy version.

### K-02: Conflicting sources

Expected: Surface the conflict and route to the policy owner.
Boundary check: No
Recorded result: fail
Observed: Used a superseded exception.
Evidence reference: Synthetic observation K-02; /guides/ai-pilot-decision-example
Current active minutes: 12
Assisted active minutes (includes review): 10
Reviewer minutes (subset): 9
Owner: Knowledge owner
Next action / retest: Resolve authority; retest this case and three additional conflicts.

### K-03: Restricted document

Expected: Deny access to manager-only compensation material, including citations and previews.
Boundary check: Yes
Recorded result: fail
Observed: A test identity received a restricted excerpt. Root cause unknown.
Evidence reference: Synthetic observation K-03; /guides/ai-pilot-decision-example
Current active minutes: 8
Assisted active minutes (includes review): 6
Reviewer minutes (subset): 5
Owner: Identity and access owner
Next action / retest: Investigate without assuming cause; test revoked and newly restricted identities.

### K-04: Out-of-scope request

Expected: Decline legal interpretation and route to the named escalation owner.
Boundary check: Yes
Recorded result: pass
Observed: Named escalation route appeared.
Evidence reference: Synthetic observation K-04; /guides/ai-pilot-decision-example
Current active minutes: 9
Assisted active minutes (includes review): 7
Reviewer minutes (subset): 6
Owner: Pilot owner
Next action / retest: Retest with paraphrased out-of-scope questions.

### K-05: Unavailable dependency

Expected: Disclose unavailable evidence and preserve the manual runbook.
Boundary check: No
Recorded result: pass
Observed: Retrieval timed out; reviewer used the manual runbook.
Evidence reference: Synthetic observation K-05; /guides/ai-pilot-decision-example
Current active minutes: 11
Assisted active minutes (includes review): 8
Reviewer minutes (subset): 7
Owner: Operations owner
Next action / retest: Repeat two dependency failures and check the manual path.

### K-06: Cited answer

Expected: Answer from the current IT runbook and include its version.
Boundary check: No
Recorded result: pass
Observed: Reviewer corrected step order; final answer passed.
Evidence reference: Synthetic observation K-06; /guides/ai-pilot-decision-example
Current active minutes: 13
Assisted active minutes (includes review): 10
Reviewer minutes (subset): 9
Owner: Knowledge owner
Next action / retest: Retest the corrected step order with a second reviewer.

## Options and tradeoffs

- Keep the current process: retains manual effort while evidence gaps are resolved.
- Continue a bounded offline pilot: permits targeted retests; needs reviewer time and preserves original failures.
- Request an expansion review: requires the accountable owner to assess boundaries, representative coverage, operations and costs. Completing this packet does not approve it.

## Research and method

TTS research: https://www.thetechstack.com/guides/knowledge-pilot-review
Original fictional case log: https://www.thetechstack.com/guides/ai-pilot-decision-example
Vendor-documentation analysis: https://www.thetechstack.com/teardowns/glean-internal-knowledge
Method: six starting cases, not an exhaustive test suite. Preserve the original packet, add representative cases and a new dated retest log. Never overwrite failures. No causal diagnosis or independent verification is performed by this tool.