A to-do list the agent can see

So in May I built one. The design rule was a boring local tool with a smart interface for agents. Then in October I gave it a map, an AI organiser behind the map, and a way for Codex to see what I’m looking at. That’s the bit worth writing about.

The boring part

The to-do system itself is a SQLite database, a small API and a plain web UI on my Mac. It does dumb CRUD and nothing clever. All the intelligence is in the agent, and the agent learns how to use it by calling one tool that returns the playbook: how to split a dump of notes into todos, how to guess priority and due dates, when to go and search GitHub for the matching PR, how to link it, and when to ask me to disambiguate rather than guess.

In practice I paste raw meeting notes into Codex and say "add these". It reads the playbook, extracts the items, finds the PRs, drafts entries, checks the ambiguous ones with me, and writes them. A merged PR on a todo changes the card and asks if it’s done.

Built on top of Work Memory

The map doesn’t only show tasks. It sits on top of my Work Memory, the Git-backed wiki my agents maintain, and pulls in the ideas, open questions and improvement follow-ups that live in there as well as whichever stream ledgers I point it at. So a card might be a task from the to-do database, or an idea I dictated into Codex three weeks ago that got filed under notes, or a lesson an agent captured after a bad session. They all end up on the same canvas, each one quoting its source rather than summarising it. There are four kinds of card: an action, an investigation, a question and an idea. An investigation is allowed to be real work without a chosen solution or a deadline, and a question I dictated stays a question rather than being tidied into a conclusion. The point is that I shouldn’t have to translate every rough thought into a project-management form before it’s allowed to exist. I say it to Codex, Codex saves it, the map picks it up.

It writes back the same way. If I add a note to a Work Memory item on the map, it goes into that file’s own section and through Work Memory’s normal closeout, with a request id so a retry can’t double-post. The map owns its presentation and nothing else: positions, grouping, Today, what I’ve rejected. Tasks belong to the to-do daemon and are read through a read-only handle. Everything derived can be deleted and rebuilt.

The AI underneath

The connections on the map are made by a model, but the model is kept on a very short lead.

The daemon collects without a model. Every minute, and whenever a task or a memory file changes, it recomputes an input revision for each item from five things: its source, my notes, my explicit links, the related workstream context from the work graph, and the evidence behind its existing links. If that revision has changed, the item is queued once, with the reason.

A separate worker interprets the queue. It runs under launchd every five minutes, uses Gemini Flash on my work Vertex account, and is capped at a dollar a day and twenty-four calls. A quiet check costs nothing and calls nothing. When there is work it takes up to five items, proposes a title, a short summary, which area the item belongs in, what kind of thing it is, and relationships to other items, and every seven days it revisits the oldest understandings in case they’ve drifted.

What it can’t do matters more than what it can. It cannot change a task’s status, priority, deadline, Today or reminders. It cannot touch a confirmed relationship or a raw capture. Every connection on the map carries a label that says where it came from: proposed by the model, backed by a source, confirmed by me, rejected, or outdated. Those are kinds of evidence, not confidence scores. A new relationship the model proposes stays proposed until I confirm it, and two items sharing words or a stream doesn’t make one depend on the other; the model is told not to manufacture a connection for every item. When it cites evidence it has to cite the actual source text, because the daemon re-reads the source before applying anything and refuses a quote that isn’t there. Progress wording comes from dated source text, never from the model. A stale review is refused. A suggestion I’ve rejected doesn’t come back unless the evidence is genuinely new. When I picked the model I ran four of them over the same twenty cases, and Flash was the one that returned everything inside the contract.

Automations watching automations

I’ve said elsewhere that if something breaks I want to know, and this is where I put that into practice. An independent checker, with no model and no credentials, polls the map, the daemon and the worker every five minutes and raises a Nudge alert if the organiser pauses, the queue stalls, or an item has been isolated. A Codex heartbeat runs every thirty minutes with bounded authority to diagnose and repair the internal faults it can verify against source, and a daily nine o’clock audit checks availability, backlog, budgets, backups and a rotating sample of the organiser’s actual quality, and checks the repairer too. None of them is a second organiser. The organiser organises; the others make sure it’s still doing so.

Nudge

Nudge is my own reminder app, a small native Mac app with an MCP adapter that both Codex and Claude share. The map reads every active reminder straight from Nudge’s own store and shows it on the right card, so a reminder is never a copy. The rule between them is that origin decides the return path. A reminder made on the map is self-contained: a note, my wording, where I left it, what’s next. A reminder asked for in a chat is created by that chat with its own tool, so it carries the way back to that conversation, and the map just displays it. Snoozing, dismissing or cancelling a reminder never touches the task. The health alerts above arrive the same way, with the full report inside the notification.

Talking and pointing at the same time

The part I like most is WebMCP. The map page registers a handful of tools with the browser, so when Codex’s built-in browser is open on it, Codex can ask the page what I have selected right now, search for an item by a loose reference, inspect one by its exact id, highlight a set of cards, add a note in my words, change what’s in Today, or record that one thing depends on another. If a browser doesn’t support WebMCP the page is just a page.

What that gives me is chat and UI at once. I click a card and say "what’s this waiting on", and Codex answers about the thing I’m looking at rather than the thing it guessed I meant. I command-click two cards and say "these are related", and it records that, with my wording as the evidence. Selection belongs to the tab and the agent reads it when I ask, not by watching my clicks, so if I change my mind before it reads, it sees the newer selection. Putting two cards in a selection doesn’t make a relationship by itself; I have to say so. The search tool never picks a result for me; it returns ranked candidates and the agent has to choose or ask, which removed most of the wrong-item mistakes. The agent sees my selection, my viewport and the page instance, and the write tools carry request ids so a replayed call is idempotent.

It’s the first to-do system I’ve kept using, and I think that’s because it isn’t really a to-do app. It’s a shared surface. I look at it, the agent looks at it, and we’re pointing at the same thing when we talk.