The AI Capacity We Already Own

By William Paul “Bill” Herald

I want the computer I already own to do useful work. With AI, that means more than getting a large model to start. I want an answer worth reading, tools that report what actually happened, and enough responsiveness that I can stay involved.

That is why I keep contributing to Colibri. It explores how much useful AI we can run without requiring everyone to buy a bank of expensive accelerators or send every task to a remote service.

I also want to explore whether making better use of existing computers could reduce some of the resources needed for everyday AI. That is my research hypothesis, not a result I have proved. Gathering more evidence is part of the work I want to do.

A CPU can do the work; a GPU is an option

Colibri supports running large models on the computer’s main processor—the CPU—with model data spread across memory and storage. A graphics processor, or GPU, is optional on supported paths. The practical requirements still depend on the model: avoiding a required GPU does not remove the need for enough memory, storage or patience. Colibri’s project documentation.

GPU acceleration can help, but I would not promise that every GPU configuration makes every task faster. Moving model data between devices and coordinating the work also takes time. The useful question is what the complete task does on a particular machine.

This work includes Kimi K3, a model from Moonshot AI. Moonshot describes 2.8 trillion total parameters, with 104 billion activated per token. Its routed layers select 16 of 896 experts, alongside shared experts. Colibri includes a Kimi K3 engine. Those are substantial capabilities to investigate, not a promise of quick answers on an ordinary computer. Moonshot’s model description.

“Only some experts” needs a little explanation

A mixture-of-experts model does not use every expert for every token, the pieces of text it processes or generates. A router chooses a subset at relevant layers as the computation proceeds. It is not simply choosing a few subject specialists once for the entire question; other parts of the model still do work.

That selective computation is interesting because increasing the model’s total capacity need not increase computation in the same proportion. But fewer active parameters do not mean the electricity bill falls by that same fraction. The selected weights still have to be available and moved, and the rest of the computer remains involved. This architecture can also be used in a data center; it is not an efficiency advantage exclusive to running locally. DeepSeek’s mixture-of-experts report.

Training is a different job from answering

Running an already trained model is called inference. Training creates or changes the model’s learned parameters. Moving an answer to my computer does not undo the energy already spent training its model.

The scale of that training is real. Meta reports more than 16,000 H100 GPUs for Llama 3.1 405B, and its model card lists 30.84 million GPU-hours for that model. GPU-hours describe accelerator use over time; they are not a direct meter reading of whole-facility electricity. Meta’s release explanation, model card.

AI can help develop AI. Meta also describes using synthetic data and its larger model to improve smaller models. That can change the work and improve its efficiency. It does not make generating training material or updating a model computationally free. That conclusion follows from the process, not from a measurement I made of a training facility.

What my own work establishes

A fresh September 13 check confirms 16 merged Colibri pull requests authored by me, with none open at the time of that check. The new v1.11.0 release explicitly credits my recent fixes for dashboard memory cleanup, preservation of Windows expert-memory budgets, and planner tests that avoid writing large zero-filled payloads. Those are inspectable contributions, not a claim that I solved large-model latency. Contribution record, v1.11.0 release.

One completed GLM-5.2 weather workflow took about 26 minutes, even though retrieving the observation took 224 milliseconds. A CPU-only candidate also produced a short readiness reply, but the longer probe reached its deadline without output. Those configurations and deadlines differed, so they do not establish a CPU/GPU winner. They show why I want better comparisons. Existing measurement notes.

The energy possibility is still a question

Could selective local computation use less electricity for a useful result? That is the possibility I want to investigate. I do not yet have measurements showing that it does.

It could also use more. Lower power over a much longer task can mean more total energy. Repeatedly reading experts from storage can add another cost. Published research analyzing SSD expert offloading found an energy penalty in the systems it studied. That is a reason to measure carefully, not a measurement of Colibri or my computer. SSD-offloading energy analysis.

I want to compare the same useful result: time to finish, whole-computer electricity in kilowatt-hours, retries and corrections. First use and reuse should be visible. A smaller model belongs in the comparison. Any comparison with a remote service needs a comparable measurement boundary and an honest account of what we cannot observe.

If I try locally and then repeat the task in the cloud, I may have added work. If a suitable local model reliably replaces a remote task, that is a different outcome worth studying. Neither outcome, by itself, proves a reduction in data-center construction or total demand.

I want to keep helping Colibri because this is worthwhile engineering and research. Better access, privacy choices and human control matter to me even before an energy advantage is established. Privacy still depends on what the surrounding tools read and send.

The theory gives me a reason to investigate. The evidence will tell me how much of it holds up.

I’ll share follow-up measurements here, including what works, what doesn’t, and what remains uncertain.

Follow the independent work at wphc.us.

Sources and further reading

• Implementation and contributions: Colibri documentation and v1.11.0 release notes.
• Model architecture: Moonshot’s Kimi K3 and DeepSeek-V3 technical report.
• Training scale: Meta’s Llama 3.1 model card.
• Energy research: SSD expert-offloading analysis and Power Hungry Processing, which studies how inference energy varies across models and tasks.