Decisions from building curatorial.app, an AI platform for curators, and the Curatorial OS, the fleet of agents that runs the company behind it. Each shows the choice, what it was weighed against, and the risk it accepted.
THE ENGINEERING EVIDENCE
is in the appendix and in the full 32-slide case study.
Serve one small, specific profession (curators) whose work is source-heavy and badly served by general chat tools. Enter through students and independents, then move to institutions on proof.
A broad “AI for knowledge workers” product, competing head-on with general assistants on distribution and model quality.
A low revenue ceiling (roughly $0.8–2.3M a year at 10–30% penetration) in exchange for a market a small team can actually win and learn from.
The order of segments is a risk decision, not a size decision: start where acquisition is cheap and feedback is fast, add human sales only where deal size pays for it.
Bottom-up sizing from public data (e.g. BLS curator counts). Planning estimates from November 2025, pre-launch: not traction.
| Commercial choice | Why | What enforces it |
|---|---|---|
| Permanent free tier | Earn the first win at no cost to the user | Open-weights model, per-tier caps, atomic token accounting |
| Viewers free, editors paid | Every shared project becomes an invitation | Read-only role enforced by row-level security in the database |
| Model class follows the tier | Cost per user rises only where value does | One model wrapper with per-tier and per-feature routing |
| Heavy users bring their own key | Power users don't erode margin | Vaulted keys injected per request; they pay their provider directly |
| Teams share research | A natural step up from one seat to Team | Shared pools under team-scoped security |
| Risk | Choice | Cost of the choice |
|---|---|---|
| Users seeing each other's data | The database enforces access (row-level security on every table); application code is the second line | More policy work up front, and every new table needs one |
| Leaked customer API keys | Keys in a database vault, decrypted per request, never in environment variables or logs | Slightly more latency and complexity per call |
| Documents leaving our control | Self-hosted OCR, so document text isn't sent to a third party for extraction | We run and patch the service ourselves |
| Internal details leaking in errors | Clients get an error code only; detail stays in server logs, checked by a lint rule | Harder debugging from the client side |
| Agents acting as users | Bots authenticate as their own identities with signed tokens | Separate identities to manage |
One function prices every model call and feeds both billing and monitoring. Reconciliation becomes “did we miss a call?”, never “do two sums agree?”.
Every call carries a feature name, so the question “what does this capability cost?” has an answer, not just “what did this vendor cost?”.
A response that reports zero tokens is flagged, not billed as free. An unknown model falls back to a default rate and raises a warning.
Swapping a model is a configuration change, tested before release. Heavy users can move their cost to their own key.
| What the evals caught | How | Failure avoided |
|---|---|---|
| A notebook generator ignoring its sources | Groundedness scored 0–10 out of 100. Retrieval was reading an empty column. After the fix: 0.96. | Confident, ungrounded summaries shipped to users, probably for every earlier output |
| A permissions bug that wasn't | Every count test failed with “permission denied”. Isolating the layer showed a paid-tier gate the test user lacked. | Loosening security to fix a test problem |
| Errors silently masked | A route caught the wrong exception type and hid recoverable model output behind a generic error. | Users seeing failures the system could have handled |
| Normalising a failure | A route failed its schema on 6 of 6 live calls. It was excluded and labelled, not baselined. | “Broken” quietly becoming the expected score |
It was adopted later. Retrofitting secrets handling is always harder than starting with it.
Decide how quality will be judged before anything is built, not after.
Business rules in one tested place stop features drifting apart.
AI coding agents work with little context. A standing map of the system keeps them on course.
Where is AI already used, with what data, at what cost and risk? Most organisations find more shadow use than they expect.
→ use-case map + risk register
Treat each tool or model choice as a decision: criteria, a test, a decision and a review date. Build, buy or configure.
→ decision records
A small evaluation set per use case and a baseline before rollout, with human review designed in.
→ evaluation set + baseline
Cost attribution, guardrails, adoption tracking and narrow agents for routine work, as in the Curatorial OS, so gains compound.
→ run-book + dashboard
LLM-backed endpoints in five families
retrieval recall@3 on a seeded set
evaluation suites · test definitions
database tables with row-level security
containers in the agent fleet, plus scheduled timers
bounded contexts in a framework-free domain layer
The full 32-slide case study covers architecture, retrieval, model routing, evaluation, monitoring, security and the agent fleet: portfolio.curatorial.app