¶ case file 7 of 7 · policybot — filed 2025-06 · last verified 2026-08

status: concluded — role ended 2026-05

↳ filed at the review — ¶ 10 · 22:23 · receipt: 19/20 cited-source sweep, self-reported

the line stands dark at 22:23 — the archive lamp is lit; a record is read in the light it needs.

PolicyBot

Private proof from institutional policy-support work: a Python RAG assistant that takes questions from the CLI and Slack, routes them through OpenAI File Search, and validates quoted passages before an answer ships.

role: rag systems engineer
timeframe: 2025-06 to 2026-05
stack: Python · OpenAI · Slack API · RAG
repo: private — institutional — Miami University IT

[ problem ] · § as found

Policy lived in three places — documents, pages, and people’s heads. An answer meant knowing which of the three to ask, and the answers didn’t always agree.

constraints —

Keep institutional policy content governed and source-cited.
Support DOCX, PDF, and Markdown policy sources.
Validate quoted passages before presenting answers.
Answer in Slack, where the questions already get asked.

fig. 1 — the guardrail path: no verified quote, no answer. the clay gate is the product — the declines are receipts too.

a drawn plate, at rest — not a screenshot. the path and the four declines are the checked-in validation ledger’s; no policy text or slack message appears.

[ architecture ] · § fig. 2, inked

Policy documents are indexed for retrieval, validated locally for quote grounding, and delivered through Slack Socket Mode with cited responses.

Policy docs

DOCX, PDF, Markdown

indexed sources

File Search

OpenAI retrieval

candidate quotes

Quote validation

Local citation checks

grounded evidence

Cited answer

Grounded policy guidance

cited reply

Slack Socket Mode

Team workflow surface

user question — back to Cited answer

fig. 2 — the system, inked. clay marks the gate: where a check can stop the run.

[ decisions ] · § as filed

d1 — use retrieval with explicit citations · accepted

Policy support is only useful when users can see where guidance came from.¹

¹ tradeoff — Citations make answers more trustworthy but require stricter source handling.

d2 — add local quote validation · accepted

Generated text should not invent or misquote policy language.²

² tradeoff — Validation adds latency and implementation work, but reduces hallucination risk.

d3 — use slack as the delivery layer · accepted

The target workflow already happens in team communication channels.³

³ tradeoff — Slack integration adds event handling to maintain, and puts the answer in the thread where the question was asked.

[ validation ] · § the receipts

at a glance — 4 of 6 terminate in pinned artifacts · 2 described only

validation

01claim: My committed validation summary reports a 19/20 latest structured sweep, a 17/25 keyword sweep, 4 honest fallbacks, and locally rejected answers when quotes could not be verified.

method: committed sweep summary — grading protocol not yet documenteddate: 2026-06
- artifact: checked-in validation ledger — below
[private-safe]
02claim: Answers run through the OpenAI Responses API with File Search, cited filenames, and local quote verification against policy files when available.

method: source and docs audit of the guardrail pathdate: 2026-06
- artifact: checked-in validation ledger — below
[private-safe]
03claim: The repo ships CLI entry points and a Slack Socket Mode bridge; no production usage, workspace adoption, or always-on service claim is made here.

method: source audit — the boundary is part of the claimdate: 2026-06

artifact: no linkable artifact — described only
[private-safe]
04claim: The source repo passes 24 tests in its own .venv-ci on Python 3.12.11, without calling OpenAI or Slack: Slack adapter, Slack formatting, and retrieval helpers. This entry read 3 until 2026-08-03, which was true of a temporary audit virtualenv that could only reach the formatting file. The suite had grown; the claim had not.

method: local pytest run, network-freedate: 2026-06
- artifact: checked-in validation ledger — below
[private-safe]

outcomes

05claim: Answers come back in Slack with the source cited, and topics the sources do not cover are declined rather than answered — 4 such fallbacks in the recorded sweep.

method: design intent, described — no usage metric is claimeddate: date not recorded

artifact: no linkable artifact — described only
[private-safe]
06claim: Responses are framed around cited sources instead of uncited generated advice.

method: guardrail path auditdate: 2026-06
- artifact: checked-in validation ledger — below
[private-safe]

ci rows link the public run · repo pins are the exact commits verified 2026-08.

what i’m NOT claiming —

The sweeps are self-graded counts from my committed validation summary; the grader and per-case pass criteria are not yet documented publicly, so treat 19/20 and 17/25 as disclosed self-reports.
This file claims design and validation shape only — never adoption or uptime. Raw policy text and Slack messages are excluded everywhere.

[ ledger ] · § policybot — validation ledger — checked in 2026-06

entry	value	note
structured sweep	19/20	latest quoted-answer pass
keyword sweep	17/25	validated loose prompts
safe fallbacks	4	unsupported topics declined
local tests	3 passed	Slack adapter, formatting and retrieval helpers, network-free
guardrail path	File Search ⟶ citation extraction ⟶ local quote verifier	unverified answers fall back

download the raw ledger (json) ⟶

sanitized RAG validation summary — no policy text, Slack messages, or raw validation transcripts.

[ corrections ] · § the register

note · 2026-07
The 19/20 structured and 17/25 keyword sweeps are recorded in a committed validation summary, but who graded each case and the exact pass criteria are not yet publicly documented. Until they are, this file labels both numbers as self-reported.

[ appendix ] · § artifacts — none beyond the page

nothing to open here, deliberately — the ledger above is this file’s publishable artifact, shown in full, and the records behind it stay private.