Agents are neurodivergent toasters

Why neurospicy people are surprisingly good at handling agents

Half the world is busy building a god. But this god is going to lose its keys.

Let me explain.

18 months ago I got an official ADHD diagnosis after a full neuropsych workup. The output was five pages of test scores and a clinical opinion.

Reading it made me feel…some type of way.

They got me on meds and a re-evaluation of my relationship to myself.

A few weeks ago I had to look it up again, but this time I didn’t see a raw, gut wrenching writeup of my failings.

I saw a system architecture document.

Specifically, architecture for an agent that I’ve spent the last few years building for my clients. I’ll show you and once you see it you can’t unsee it. Because this one thing explained what I haven’t been able to before: why some of us seem to get these machines on a visceral level regardless of how much math we know.

Sports car with a missing gearbox

On the raw cognitive tests I did well. Above average on most of them. These were seemingly trivial stuff. I held 8 digits in my head (forwards, then backwards), redrew a complex figure from memory, 36 out of 36.

Visual memory, verbal memory, working memory: all pretty damn strong. Next to result after result the examiner wrote the same note:

Work tempo faster than normal, accelerating towards machine-like speed.

So if you only looked at the raw numbers you’d wonder why I was in the room. Better yet, why does my wife nod in agreement when someone says I’m forgetful.

Then you read the rest and it flips. My memory is just fine, it has more than enough horsepower. But the part that decides where to point the engine and whether to hold it there is broken.

Aim the attention. Keep it on target. Push it through something boring. That’s the weak, spiky part. It runs feast or famine: either I’m locked in so hard I lose a whole afternoon or I cannot get the thing to turn over at all.

So I have a sports car equivalent of a brain and missing a gearbox.

I’ve seen this one before

We measure the context window of models like it was a holy grail. Github is full of complaints where devs plead for OpenAI to increase Codex GPT 5.5 context window to 1M as if that would magically solve their problems.

But it isn’t really the problem. My “context window” is big. 8 digits is a lot (for a human). The problem is allocation.

There’s a paper I keep coming back to called Lost in the Middle (Liu et al.). The finding: you can hand a model a big context, but its attention over that context sags in the middle. Stuff that is technically right there stops getting used. The window is full. The model just isn’t looking at the right part of it.

That’s my entire neuropsych report in one line.

I don’t lose my keys because my memory failed. I lose them because at the moment I set them down, the spotlight wasn’t on my hands. The data was available. Nothing wrote it to disk.

(I’ve written before that when a model hallucinates, it’s usually a sign of overwhelm, not a bad prompt. Turns out I was describing myself and didn’t know it.)

Huberman wrote that ADHD is not a deficit of attention, it’s a deficit of attention regulation. “How do we fix it?” is one of the most important questions of my life.

But if this deficit is also apparent in agents, then how do we fix it for them? This is a tech blog after all and nobody cares about my self indulgent ramblings about how my brain works.

Well, here’s what I can borrow from my real life though: We stop trusting the model to “just remember”. We make memory an explicit tool call.

Write it down now, at the moment of action. Because anything you don’t deliberately persist is gone the second the context rolls.

That’s not just a metaphor for how I live it’s literally how I live. Everything I do on my computer, every system, every meeting, every data point I’ve made Alfred hold a piece of context so I don’t have to.

I’ve been screwing a keey hook into the wall by the door so I stop losing the keys.

Turns out I’ve been writing memory tools for my own brain since long before I knew that’s what they were.

5 things that make agents neurodivergent

Once I started pulling the thread it didn’t stop. Point for point, the report and the agent rhyme.

1. Allocation, not capacity. Big window, weak attention across it. Lost in the middle. We just covered this one.

2. Memory has to be an explicit write. Trust “I’ll remember later” and it’s gone. Force the write in the moment or lose it. The vault, the notebook, the hook by the door.

3. Neither one self-starts. A base model sits inert until something prompts it, no inner drive to go do the tedious background task. Neither have I. So I supply my own trigger. (What the trigger actually is gets dark.)

4. The capability map is jagged. My scores aren’t a smooth line. Excellent at memory, mediocre at naming-interference, dead average at impulse control. Agents are the same. Superhuman at one thing, faceplanting on the trivial task sitting right next to it. A normal engineer keeps getting blindsided by that. I don’t. My own head breaks the smooth-gradient assumption every single day. I live on the jagged frontier.

5. Even the broken self-awareness matches. This is my favorite. On the self-report form, I rated my own working memory as severely impaired. The actual performance test then measured it as above average. I was confidently, measurably wrong about my own internals, in precisely the confabulating way a model is wrong when you ask it to explain itself.

So even on unreliable introspection, I score like the machine.

But agents are different than humans

I’d be doing the exact thing that makes me cringe if I just went “lol my brain is an LLM”.

I have a body. I have drives. I have stakes, and continuity, and a daughter, and a marriage that the report itself lists as a stabilizing force.

And that “self-starting trigger” from earlier? In a clean system that’s a cron job.

In me, it’s anxiety. The report says it outright: high anxiety is what I use to compensate, and the by-product is a burned-out, enervated wreck on the far side. That’s the loop my wife described to the examiner: plan carefully, overload, overwork, collapse, replan. The activation function fires. It also bills me in cortisol and dead weekends. It isn’t a function. It’s suffering.

A lot has happened since last January. I wonder what would a new test say, but I’d wager anxiety got replaced by a different driver, because now I constantly optimize for maximum presence with my family. This is a lot more powerful because anxiety required me to wait until things start to fall apart (procrastinate) so I can get into action mode. Presence does not wait, because every moment I miss now I’m never getting back.

A base model is frozen. It learns nothing from its own loop. I am not frozen. On one of the memory tests they read me a list of 15 words: I got 7, then 11, then all 15. I learn in real time. I got sober. I rebuilt my life. The report goes out of its way to note how much insight I’ve built about my own wiring.

On the single axis the labs care about most right now (continual learning) I beat the agent. (For now.)

So, no. My brain is not an agent.

Engineered for drift

I frown where people tell me to “can’t you just tell Alfred/Hermes/Claude/etc to verify its sources”? It’s the AI equivalent of “have you tried setting reminders?”

No Karen, I have not. What a wonderful idea. I’ve been destroying opportunities, relationships, getting myself fired and setting fire to my own potential for three decades while all I had to do was to use a fucking calendar. You got any more genius where that came from?

My human brain drifts all the time. It’s involuntary, even if you treat it as a personality trait. Agents drift too.

The difference between an agent that passes as great and actually being great at offloading tasks is how much you’ve engineered it for drift.

Adding more lines to a CLAUDE.md or AGENTS.md file is not “creating a harness”. You’re just creating a Custom GPT with a worse UX.

Your agent deterministically being triggered to compute the right context upon different events and then cascading into actions is what’s going to create a harness useful for you.

Here’s an example for you:

Every night Alfred goes through the same routine, triggered by time:

1. Look through chat sessions from the last 24 hours and extract key activities

2. Cross reference and match them with existing Matters, Tasks and Decisions I have.

3. Update the Tasks and Decisions accordingly

4. Find all meeting transcripts from the last 24 hours and process them using steps 2 and 3.

5. Then based on this, prepare a morning briefing for me.

Only steps 2 and 5 are done by AI. All else is Python code the AI wrote. But it does not stop there.

6. At 5am Alfred re-reads the briefing and outputs a list of apps and URLs I will need to have opened when I sit down for work.

7. It then triggers a script that closes ALL apps on my Mac Mini (except for cmux and Claude) and opens all the apps and URLs from the previous step.

8. Then at 6am I get a Slack message telling me exactly what to do.

Steps 6 and the message writing from step 8 is done by the AI. everything else is…you guessed it. Python.

I could have just asked Alfred to create a cron job to do this and let it figure out on its own. This is exactly what I did at first in February. It worked for a week and then it started failing randomly because I didn’t put in the work to engineer for drift.

---

I have spent my whole life as the systems engineer for one specific, unreliable, context-limited, badly-regulated process.

My own cognition.

I didn’t pick it as a career. I started as a loud six-year-old who couldn’t sit still and kept losing things. Decades before the industry needed anyone to solve the same class of problem for a machine.

The job of harnessing an agent (getting reliable output out of a fast, capable, poorly-regulated process you can’t fix at the core, only build scaffolding around) is the job I’ve been doing on myself since before I could read a clock.

That’s not a deficit story. It’s a transfer-of-expertise story. The disability and the expertise are the same skill. I just learned it on the hardest possible test case: me.

And I think that’s the real reason it clicked. Not because I’m sharp about transformers. Because I’ve been running the harness from the inside my entire life.

So the first time I saw one written down in code, it read like my own handwriting.