I Used 2.5 Billion AI Tokens in 7 Months. We Are Barely Using These Systems.

People sometimes tell me I am overly optimistic about AI.

That I have fallen for the hype. That LLMs still hallucinate. That agents are unreliable. That the big AI companies have convinced everyone that chatbots are more capable than they really are.

This is always slightly funny to me.

I work on LLM interpretability for my PhD. I am very familiar with how strange, brittle, and unreliable these systems can be.

And after using roughly 2.5 billion tokens in the last seven months, with my current usage often around 10 million tokens per day, my conclusion is almost the opposite:

Most people have barely seen what current AI systems can do.

And I'm not talking about some future AGI models, but the systems we have right now.

Chat Is a Terrible Way to See the Frontier

Imagine asking an AI:

Who was the fifth US president?

A decent model answers James Monroe.

Make the model twice as capable.

James Monroe.

Ten times as capable.

Still James Monroe.

You have built a test with essentially zero remaining dynamic range.

I think something similar has happened with how most people experience AI.

They open a browser, ask ChatGPT a question, read the answer, perhaps notice a hallucination, and form an opinion about the technology.

But chat is increasingly just one tiny interface to these models.

The interesting systems now look more like OpenClaw, coding agents, or a Hermes-based system I am working on: give an agent a computer, terminal, browser, files, tools, memory, and an objective, then let it work.

OpenClaw itself is explicitly built around this idea: an always-running agent on your own computer that can interact with email, GitHub, browsers, messaging, and other software.

That is a very different experiment from asking ChatGPT twenty questions.

And you do not even need some enormous swarm of 500 agents.

Often one competent agent, occasionally spawning a few subagents or making a hundred parallel API calls, is enough for the experience to become qualitatively different.

The 100-Metre Problem

Benchmarks have the same problem.

Imagine we measured human athletic ability only with a 100-metre sprint.

At first, improvements would look huge. Eventually, progress would appear to flatten: each new athlete beats the previous record by fractions of a second.

If you extrapolated from that metric alone, you might conclude that humans are approaching their fundamental limit.

But the metric would completely fail to reveal that a human can run 42 kilometres.

A marathon is not a 100-metre sprint with a score above 100%.

It is a capability that becomes visible only when you change the task.

This is increasingly how I think about AI benchmarks. Most are bounded at 100%. Once models become good at them, enormous improvements elsewhere can become almost invisible.

A model going from 97% to 99% looks incremental.

But perhaps the meaningful change is that it can now work autonomously for two hours instead of ten minutes, recover from failures, navigate unfamiliar software, write and test its own tools, or complete a research workflow containing hundreds of dependent steps.

We keep asking:

How much faster can it run 100 metres?

When the interesting question is becoming:

How far can it run?

This is also why autonomous-agent evaluations are much more interesting to me than another saturated QA benchmark. The capability axis itself is changing.

Once you start looking along that axis, some of the things already happening in 2026 look much less like “ChatGPT got 3% better” and much more like discovering that the runner can suddenly do a marathon.

Things I Have Seen This Year

Some public examples already sound slightly absurd.

Anthropic recently let Claude attack a mathematical research problem. The model used 31 million output tokens, coordinated roughly 60 subagents, generated hundreds of scripts, and ran thousands of numerical checks. The human contribution was largely messages like “keep going” and “believe in yourself.” The system produced a new mathematical result.

Earlier this year, former Google PM Bilawal Sidhu built WorldView in three days with AI agents: a Palantir-like 3D interface combining live satellite tracking, aircraft data, CCTV feeds, traffic, and seismic information from public sources.

Then there is AdderBoard. The original challenge was simple: build the smallest transformer that can add two 10-digit numbers with at least 99% accuracy. Claude Code produced a 6,080-parameter model. Codex found 1,644 parameters. The resulting search then pushed the problem down into transformers with only dozens of parameters, including constructions where weights were set analytically rather than learned conventionally.

These are not chatbot tricks.

They are examples of models operating inside environments where they can search, test, fail, write code, inspect results, and keep going.

My personal examples are sometimes even more convincing to me because I know exactly how little I did.

I have watched an AI scientist carry out competent research and arrive at genuinely novel ideas.

I once asked an agent to collect vouchers associated with scientific papers. It opened the website, figured out a bizarre interface, wrote a scraper, tested it, manually checked whether its own scraper was correct, and finished the collection. When the website later changed, it noticed the scraper had broken and fixed it without my telling it to.

For another project, an agentic system created a self-reflective technical diagram, turned the material into a roughly 40-slide presentation, and then produced a 30-minute recorded presentation with generated narration.

I have had agents recreate research papers and reproduce their experimental results within hours.

For quick research prototypes, my interaction sometimes approaches:

Here is the hypothesis. Here is roughly how I would test it. Go investigate.

Then I come back and inspect the plots.

That is a completely different relationship with computing.

Ten Million Tokens a Day Changes How You Work

Ten million tokens is roughly on the order of a hundred books of text per day.

Obviously, I am not reading a hundred books every day.

That is exactly the point.

At this scale, the old interaction model breaks:

AI produces text → human reads all of it

You instead need:

AI explores → AI filters and checks → human reviews the important parts

The scarce resource becomes human attention.

• What deserves my attention?
• What could be wrong in a way that matters?
• What should I inspect personally?
• What can I test automatically?
• What can I simply delegate and trust?

This is a skill.

And I think this is partly why heavy AI users experience much less frustration than people who repeatedly complain that their coding agent did something stupid.

You learn when to hold its hand and when to let it rip.

AI Engineering Is Becoming More About Architecture

As implementation becomes cheaper, I find myself caring much more about things that used to feel one level above coding—and dusting off old books on software architecture:

• Information flows.
• Interfaces between components.
• Design patterns.
• File hierarchy.
• Where state lives.
• Which component is allowed to modify what.
• How failures propagate.
• What the agent needs to know at each stage.

The actual software architecture.

If an agent can generate thousands of lines of competent code quickly, then specifying the wrong system extremely efficiently is not much of a victory.

This is why techniques like spec-driven development and tools such as grill-with-docs are interesting to me. Grill-with-docs essentially forces the agent to interrogate your design before implementation and records the important vocabulary and architectural decisions as it goes.

The human job increasingly moves upstream.

• Think clearly about what should exist.
• Design the environment.
• Then let the machine instantiate it.

Learn How the AI Can Trick You

The other important skill is verification.

Not every task deserves the same amount of supervision.

If I am generating a throwaway visualization, I may barely inspect the code. I care about the resulting figure.

If I am changing an important research pipeline, I might inspect the architecture, the tests, and the critical implementation.

For some tasks, I deliberately construct hidden tests: questions the model cannot bluff past.

This is the point I would emphasise most. Ask “did you test this?” and you will get “yes.” Instead, keep checks where you already know the answer, cases you verified by hand, or questions whose correct response you can recognise instantly.

The goal is to frame the task so the agent cannot simply produce something that looks correct. Verify its work against those tests, never against the model’s own account of its work.

In practice, that means designing checks like these:

• If it claims a scraper found every paper, can I independently know how many should exist?
• If it implements an algorithm, can I test properties of the output that were never included in its prompt?
• If it reproduces a paper, can I compare against known numbers?

You start thinking less like somebody pair-programming with autocomplete and more like somebody designing an evaluation environment for another engineer.

This dramatically reduces frustration.

The goal is not to stare over the agent’s shoulder while it writes every line.

The goal is to know where staring over its shoulder is actually worth your time.

Cheap Cognition Changes the Questions

There is one more ingredient: cost.

Some of the most capable Chinese models have made inference incredibly cheap. DeepSeek V4 Flash, for example, has recently been priced in the range of cents per million tokens for some classes of input, with cached input dramatically cheaper still. Even after its August pricing changes, the economics remain radically different from only a short time ago.

A year ago I often thought:

What is the best possible use of these tokens?

Now I increasingly think:

What problems become possible if computation is cheap enough that I can afford to waste most of it?

• Generate 100 hypotheses.
• Try all of them.
• Throw away 95.
• Have another agent attack the remaining five.
• Run the experiment again from scratch.
• Explore several implementations in parallel.

This is not just the old workflow becoming cheaper.

Once cognition becomes sufficiently inexpensive, different algorithms for doing knowledge work become rational.

The Wave

This is why I find the pessimism around the current AI moment so strange.

I understand why someone can look at it and see uncertainty.

Jobs changing. Skills becoming obsolete. Models improving faster than institutions can adapt. Systems that are powerful but imperfect.

I see the same wave.

I just have a very different reaction to it.

A normal person looks at a huge wave and thinks:

That looks dangerous.

A surfer looks at exactly the same wave and thinks:

Holy shit. Look at that.

The surfer is not unaware that waves are dangerous.

They are excited because they learned how to ride them.

That is roughly how I feel right now.

Every few weeks I see another glimpse of something that makes me rearrange my mental model of what one person can build.

And after 2.5 billion tokens, I do not feel like I have mapped this territory.

I feel like I have barely started exploring it.

Years from now, some people may remember this period mainly as stressful and uncertain.

I suspect I will remember it as a golden era of blooming flowers.

Not because every idea works.

Because suddenly there are possibilities everywhere.