Operational knowledge spans records, policies, runbooks, and postmortems with different keys and levels of structure.
Implemented toolkit / governed knowledge systems
Make GraphRAG prove it deserves the graph.
I built a local-first conversion workbench that profiles source data, gives coding agents a schema-backed ontology task, requires human approval, compiles deterministic graph IR, routes questions through safe recipes, and publishes case-level evaluation evidence.
Local product implemented and tested
Local evaluation · hosted testing planned
- Structured coverage
- 100%
- Held-out cases
- 80
- Incident routing
- 40 / 40
- Seller routing
- 37 / 40
- Decision
- Use GraphRAG when the path is part of the answer. Keep vector RAG as the default for narrative retrieval.
- Local test results
- The local compiler and safe-routing contract work across two domains. Three seller routing failures remain visible.
- Next evaluation
- A graph-versus-vector answer-quality comparison is planned, along with hosted Neo4j testing and latency measurement.
Product problemTurn mixed sources into a governed retrieval contract
Relationship questions need an inspectable path from source data to answer.
Teams need to review entity keys, relationships, field mappings, constraints, and ignored data before ingestion.
Free text must resolve to typed parameters and bounded query recipes without giving the model unrestricted Cypher access.
Held-out cases must show where graph, vector, hybrid, and unsupported routes succeed or fail.
The workbench stores these decisions in one GraphSpec and carries source provenance through compilation, retrieval, and evaluation.
Reusable workflowOne contract, several tools
AI proposes. A person approves. The compiler decides what can run.
01 / Profile
Inspect the evidence
Hash files, infer fields, sample values, discover text sections, and warn on potential sensitive data.
02 / Propose
Use any coding agent
Codex, Claude Code, Cursor, or another repository-aware agent proposes one schema-backed GraphSpec.
03 / Review
Refine visibly
Edit node types, relationships, mappings, and recipes through forms with a live ontology canvas.
04 / Approve
Bind meaning to sources
Semantic and source hashes form the build gate. Dragging layout does not alter approval.
05 / Compile
Create neutral graph IR
Nodes, relationships, chunks, and exact provenance exist before any provider or Neo4j write.
06 / Evaluate
Keep losses visible
Graph, vector, hybrid, unsupported, safety, and paraphrase cases produce a versioned report.
ArchitectureTrust boundaries first
The model never receives an unrestricted database capability.
Inputs
Local sources
CSV, JSON, Markdown, and text remain local until a cloud provider is explicitly chosen.
Contract
GraphSpec
Pydantic emits the same JSON Schema for the CLI, editor, coding agents, and structured model outputs.
Compiler
Deterministic IR
Every structured field is mapped or ignored with a reason. Text facts need an exact source quote.
Storage
Neo4j + vectors
Tenant-scoped idempotent merges and model/version-specific index namespaces prevent silent reuse.
Runtime
Approved recipes
AI selects a route and typed parameters. It cannot invent executable Cypher.
Evidence
Case-level artifacts
Provider, model, ontology, data, commit, route, parameters, citations, failures, and latency travel together.
Worked examplesGraph, vector, hybrid, unsupported
Choose retrieval to match the question.
“Which customers are exposed to INC-104?” traverses incident, service dependencies, customer usage, and ownership. The path is bounded at three hops.
“Summarize the postmortem mitigation” retrieves narrative text. Modeling every mitigation sentence as graph structure would add cost without obvious value.
Ownership comes from exact graph relationships; response guidance comes from cited runbook chunks.
INC-777 does not exist. The system returns no recipe, no Cypher, and no score instead of guessing.
Measured resultsNo predetermined winner
Current evidence covers source conversion and safe recipe routing.
22 mapped fields, 17 nodes, 17 relationships, 36 chunks, and zero unresolved structured fields.
20 mapped fields, 33 nodes, 53 relationships, 87 chunks, and zero unresolved structured fields.
40 of 40 reviewed cases passed the deterministic recipe-selection and parameter suite.
37 of 40 passed. Three cases failed; the linked report includes each result.
These tests measure source conversion and query routing. The next evaluation will compare answer quality, citations, and latency using the same synthesis model for graph and vector retrieval.
Security and governanceFail closed by design
The ontology is governed before it becomes executable.
Semantic or source changes make approval stale. Draft history is retained locally.
Writes, procedures, unbounded traversal, missing tenant predicates, missing numeric limits, and undeclared parameters are rejected.
Loopback binding, same-site HttpOnly session cookie, per-launch CSRF token, and no telemetry.
Read-only routes only, synthetic data, rate limits, no source uploads, and no local editing API.
Decision and roadmapWhere complexity earns its keep
Use the graph where relationships matter.
Use it for dependency impact, ownership, entitlement, supply-chain, configuration, and policy questions where relationships and paths must remain inspectable. Use vector RAG for summary and explanation. Use hybrid retrieval only when both are necessary.
Next, I would test OpenAI and Ollama answer generation, compare Neo4j and vector retrieval, and measure hosted performance using synthetic samples.