A Mac built for crowds of agents
Most tools for running models locally assume one person is typing to one model. Agent systems don't work that way. A coordinator might start a dozen or more workers, and they'll all hit the same model at once, often with the same long document in front of them. On NVIDIA hardware, vLLM already copes with that kind of load. When Incept5's CTO John Davies launched inferd live on the ThursdAI podcast on 5 November 2026, the hosts said "nothing did it well on the Mac."
inferd is Incept5's answer. It's an inference engine for Apple Silicon, released under the Apache 2.0 licence, and the current release is v0.4.1. According to ThursdAI, Incept5 deploys private models for clients who can't send data to the cloud, using DGX Sparks, Mac Studios and Mac minis. That explains why the firm wanted one Mac to serve a whole team of agents. The repository calls the project "young, experimental software", so we treat the vendor's numbers below as claims, and we compare them with the one independent test published so far.
Under the hood
Incept5 says it wrote the engine from scratch in October and November 2026 in C++, Objective-C++ and Metal, Apple's GPU programming layer. Some MLX kernels are vendored under the MIT licence, but no MLX runtime sits in the token path. The engine combines four standard server techniques:
- Continuous batching. New requests join the batch that's already running at the next decoding step. They don't have to wait for the whole batch to finish.
- A paged KV cache stored in 8 bits. The KV cache holds the attention keys and values the model has already worked out for each conversation. Splitting it into 8-bit pages lets more agents' contexts fit in unified memory at the same time.
- Prefix sharing with an SSD overflow tier. Shared opening text is processed once and reused. Anything that won't fit in memory spills to the internal drive. On Macs with less than 256 GB this tier is switched on by default and sized at up to 64 to 96 GiB. It stays on disk after the server stops until you clear it with
./run.sh clean. - Speculative decoding for every agent. A small drafter model, incoai's DFlash2 (built on z-lab's DFlash block-diffusion approach), guesses several tokens ahead, and the main model checks the guesses in a single pass.
The agent-contexts API
What sets inferd apart from a general server is an extension to the familiar OpenAI chat format. A client can save and pin a big shared context, such as a long report, with POST /v1/contexts, and every agent then starts from that saved state instead of reading the text again. An optional inferd field on each request sets the context, a session ID, effort level, maximum context and priority. Setting "cache": "drop" runs one-off jobs without pushing other agents' work out of the cache. GET /v1/cache shows what's cached, pinned, parked or free, and every reply reports cached_tokens.
Sessions are the other half. Between turns, an agent's latest state stays in memory or gets parked on the SSD. Incept5's site says a parked agent resumes in 0.2 to 0.7 seconds, where re-reading its material would take 15 to 18 seconds.
Everything else is plain OpenAI API on port 8300, so most frameworks only need a new base URL (http://localhost:8300/v1), key and model name. The repository also lists completions, logprobs, guided output and a /metrics endpoint.
Models, Macs and setup
The default pairing is Qwen3.8-27B at 4-bit with the DFlash2 drafter, and it runs on Macs with 32 GB of memory or more. Qwen's Hugging Face card lists the model under Apache 2.0 with a native context of 262,144 tokens, extendable to one million. Owners of 64 GB machines can step up to an 8-bit version, and a 256 GB Mac Studio can run Qwen3.8-Flash-Next. Your own checkpoints load through --model and --draft flags.
You'll need macOS 26 or 27 and Python 3.12. Each macOS version has its own build, and the README warns that running the macOS 26 build on macOS 27 produces wrong logits on longer prefills. M5-class chips and newer are the main target, M3 and M4 have had light testing, and nobody's tried M1 or M2 yet. The prebuilt release doesn't need Xcode: unpack it, run ./install.sh --model 27b, then ./run.sh start --model 27b. The first install pulls about 17 GB of weights, though copies already sitting in ~/models, the Hugging Face cache or LM Studio get reused.
What Incept5 claims
Incept5 measured its headline figures on a 256 GB Mac Studio M5 Ultra between 2 and 5 November 2026. With 128 agents generating at once, the machine produced 1,487 tokens per second in total while drawing 293 to 300 watts at the wall, which works out to 4.4 to 5.1 tokens per joule. The repository's tables show 722 tokens per second for 32 Qwen3.8-27B agents on 16k contexts, against roughly 369 for Splash 1.5 on the same machine, and 849 tokens per second at 64 agents. A 32 GB Mac mini M6 managed 13.3 to 14.0 agent turns per minute on 8k contexts. On a 173-question slice of Humanity's Last Exam, the 4-bit 27B model scored 32.4%.
Incept5 is open about the trade-off. Its own pages say Splash, TensorFold and mlx-serve decode a single stream faster. If you're one developer chatting with one model, inferd won't help much. The gains only appear once many agents share the machine.
The first outside test
On 9 November a GitHub user posted issue #3 with results from a 128 GB Mac Studio M5 Max running the v0.4.1 prebuilt on macOS 27.2. One agent ran at 90.4 tokens per second. As concurrency rose to 16, total output climbed to 314.7 tokens per second, but each agent's median speed fell from 93.7 to 22.5 tokens per second, and median time to first token grew from 0.21 to 2.86 seconds. Against Incept5's published M5 Ultra numbers for the same model, that's roughly 55 to 62%. Agent tests at 8 and 16 agents answered every check question correctly. The release gate passed overall, though one of four lossless speculative-decoding checks diverged. The tester traced that to an exact tie between two candidate tokens. When we read the issue, no maintainer had replied.
Costs and caveats
The real cost is the Mac. Incept5's site gives a range of about $1,000 for a Mac mini up to $20,000 for a Mac Studio, without naming a currency. After that you won't get a per-token bill. Model support is narrow, since everything is tuned and benchmarked on Qwen. The project is small, with 33 stars and 85 commits when we checked, and its security model assumes local, trusted use. Keep port 8300 on localhost or behind a properly secured gateway.
Where builders could start
The best fit is a pipeline where lots of workers read one large input. Incept5 describes 32 auditors reviewing a 40,000-token report in 8 minutes 19 seconds, with 99% of input tokens served from shared memory. Contract review teams, alert triage and evaluation runs follow the same pattern. For New Zealand firms that don't want client files leaving the building, a Mac in the office keeps documents on local hardware and swaps API spend for a one-off purchase.
A cheap first experiment is to point an existing OpenAI-client agent at port 8300 on a Mac you already own, then run the repository's test harness (docs/TESTING.md). Track total throughput for batch jobs, and per-agent speed and time to first token for interactive ones. Results from M3, M4 and smaller machines would be especially useful, since there's little published data for them. You can share them with other Auckland builders on the society's projects page.
Sources
- GitHub: Incept5-AI/inferd
- inferd: Agents on a Mac (Incept5)
- External result: Mac Studio M5 Max 128 GB, v0.4.1 prebuilt (inferd issue #3)
- Qwen/Qwen3.8-27B model card (Hugging Face)
- ThursdAI episode, 5 November 2026
- ThursdAI: Incept5 AI Releases: inferd
Build with people working on the same things
Join the Auckland Society for AI: NZ$599 a year with a ForgeBox and the members' projects area. Our Auckland meetups are open to everyone.