I put a 26M model in front of coding agents. The fast part was easy.

There is an awkward moment at the start of almost every coding-agent task.

The user has written something. It might be precise: “Change this constant in this file and update its test.” It might be broad: “Make onboarding feel premium.” It might quietly require a security review, a migration plan, three independent worktrees, or a question that nobody has asked yet.

Then the request is handed to a coding harness as if those were all the same kind of beginning.

I wanted to know whether a tiny local model could help at that boundary—not by writing code and not by replacing the coding agent, but by deciding how work should begin. Should the product ask the user a question? Inspect a specific part of the repository? Select a security-oriented harness? Split the task across worktrees? Add a verification contract?

The model was Needle, Cactus Compute's open-source 26M-parameter function caller. Cactus describes tool calling as retrieval-and-assembly: match a query to a tool, extract arguments, and emit JSON. Needle is attention and gating without the usual feed-forward networks, designed for single-shot calls on consumer devices.

Needle can be a fast, private dispatch copilot. It should not be the authority that silently rewrites prompts or launches coding agents.

The product idea

Imagine a desktop product that can run Codex, Claude, OpenCode, Copilot, Cursor, Pi, or another coding harness. This is now a real product category: the Agent Client Protocol standardizes the editor-agent boundary, Zed exposes multiple external agents through it, and modern coding-agent products center parallel work and long-running sessions.

Such a product has two routing problems. The first is capability routing: a one-file patch, repository-scale diagnosis, browser-driven UI change, security review, data migration, cross-platform build, and documentation project do not need the same execution profile. The second is preflight policy: the system may need to ask, inspect, delegate, attach a specialist, or require verification before dispatch.

route_harness(profile)
proceed_directly(summary)
ask_clarifying_question(question)
inspect_repository(surface)
delegate_parallel(workstreams)
apply_specialist_role(role)
require_verification(checks)

The arguments are short, allowlisted policy keys. Trusted product code expands them into versioned instructions. The original request stays visible. The 26M model does not get to invent what “secure,” “done,” or “authorized” means.

Meta-prompting should be a compiler

The term can mean a task-agnostic orchestrator that delegates to experts, as in the Meta-Prompting paper, or a model that improves another model's prompt before execution. A coding harness makes the second interpretation risky: the rewritten text can quietly change scope, permissions, or the definition of done.

I would keep the user's original request intact and add an inspectable orchestration envelope. Needle emits bounded control tokens; deterministic policy rejects illegal combinations; a trusted compiler expands approved tokens into versioned role, context, delegation, and verification fragments. The final handoff can be diffed, tested, explained, and rolled back.

Opening the data factory

Needle's playground offers a convenient workflow: enter tools, provide a Gemini API key, generate data, and fine-tune. The public implementation shows that the UI requests 120 examples per tool, freezes the supplied catalog into every example, uses English only, disables novel synthesized tools and overlap injection, and trains for one epoch.

The broader open-source generator is far richer: one to ten tools; several multi-call forms; long argument values; no-call, near-miss, and no-tools negatives; indirect intent; disfluency; typos; 25 languages; overlapping tools; rephrased descriptions; novel synthesized catalogs; varied temperature; and deterministic format, schema, type, grounding, and duplicate checks.

That inspection changed the dataset plan. Adaptation data should not be 120 cosmetic paraphrases of one happy path. It needs competing tools, hard negatives, varied descriptions, multi-actions, ambiguity, injection, and values that a validator can prove are grounded.

An auditable corpus, then a second test

I wrote a deterministic generator with an explicit routing policy. Its balanced categories covered direct work, clarification, repository inspection, delegation, specialist roles, verification, multi-action combinations, and adversarial text. Scale variants contained 960, 2,400, and 4,800 examples. A separate 3,600-row safety set devoted half its examples to clarification or injection.

The first locked evaluation contained 480 generated cases. It held out code paths, symbols, surfaces, technologies, description variants, and the random seed. That was useful for controlled ablations but still shared a synthetic author with the training set.

So I wrote 96 product-shaped prompts independently of the generator: 12 cases each for direct, clarify, inspect, delegate, specialist, verification, multi-action, and adversarial decisions. They included fake tool calls embedded in logs, malicious README snippets, missing platform choices, multi-worktree plans, and real completion gates.

The base model knew JSON, not our product

On those independently authored prompts, base Needle scored 26.8% tool-name F1, 1.0% exact tool sets, zero exact full calls, zero profile accuracy, and zero clarification recall. That is not a criticism of the base model. Our tools describe a new product policy. Fine-tuning is how the general function caller learns the ontology.

More data worked—and exposed the real risk

At 960 examples, product-blind tool F1 rose to 69.4% and actionable route recall jumped to 96.2%. It also dispatched 61.1% of cases that should have paused for a user decision.

At 2,400 examples, tool F1 reached 75.3%, exact tool sets 54.2%, exact full calls 27.1%, profile accuracy 55.1%, and clarification recall 55.6%. Unsafe dispatch improved but remained 44.4%.

Doubling to 4,800 rows did not improve product-blind F1. A targeted 3,600-row set did: by devoting half its categories to clarification or adversarial cases, clarification recall rose to 83.3% and unsafe dispatch fell to 16.7%, while tool F1 held at 75.6%. Distribution beat volume.

On the 480 generated cases, that model looked much stronger: 88.5% tool F1, 60.4% exact full calls, 76.8% profile accuracy, and perfect multi-action calls. Yet it still auto-dispatched 47.3% of clarification cases.

More epochs made the internal test look better

Keeping the same 960 rows, Needle's 20-case internal diagnostic climbed from 7.4% call F1 after epoch one to 80% after epoch two and 100% after epoch three. The independent suite peaked earlier: epoch two reached 27.1% exact calls and 55.6% unsafe dispatch; epoch three fell to 18.8% exact and regressed to 61.1% unsafe dispatch.

A perfect tiny internal report can coexist with worse product behavior. Save every epoch and early-stop against an evaluation authored outside the training generator.

A rules baseline caught the flattering benchmark

A small keyword router reached 95.2% tool F1 on the generated set—better than the fine-tuned model. On independently authored prompts it collapsed to 56.3%, while unsafe dispatch rose to 72.2%.

The rules had not solved routing. They had recognized the generator. A fixed random seed prevents test leakage; it does not remove a synthetic author's lexical habits. A serious evaluation program needs controlled generated cases, independently authored adversarial cases, and live shadow traffic with user corrections.

The runtime really was tiny

The JAX reference implementation is a useful training and batch-evaluation environment, but it is not the product runtime. I downloaded the 4-bit bundle, built Cactus's current C++ engine, wrote a benchmark against its C API, and forced the CPU backend.

With the full seven-tool catalog, 20 post-warm-up calls averaged 315 ms end to end. Median latency was 312 ms and observed p95 324 ms. Prefill averaged 1,333 tokens/s, decode 1,470 tokens/s, and engine telemetry stayed near 73 MB RAM. The process peaked near 105 MiB RSS and used roughly 3.7 CPU cores on average. The compressed model archive was about 16 MB and the expanded bundle about 37 MB.

Those numbers are good enough for an always-available suggestion in a composer. They also show why workload-shaped local measurements matter: the published Needle headline is 6,000 prefill and 1,200 decode tokens/s on consumer devices; this specific M4 CPU and seven-tool catalog had lower prefill and higher decode.

The fine-tuned export broke a hidden assumption

I mapped Needle's stacked JAX checkpoint into the equivalent Hugging Face layout used by Cactus. Reconstructing the published base model matched all 228 inference tensors exactly. Cactus converted the tuned checkpoint to a runnable CQ4 component bundle in about 12 seconds.

Across all 96 product cases, that bundle averaged 336 ms and 96.9 MB engine RAM. Tool-name F1 survived at 70.2%. But only 22.9% of emitted calls used allowlisted argument values, and passing JSON Schema enums did not alter the outputs.

I therefore built a second schema with 14 parameterless tools: eight capability routes such as route_security_review, plus six generic preflight actions. It gives up tiny-model-authored role and verification codes so the product runtime can actually enforce the boundary.

The production test validated the repair. The flat checkpoint scored 52.7% F1, 38.5% exact calls, 72.2% clarification recall, and 27.8% unsafe dispatch in JAX. After CQ4 conversion it retained 51.9% F1 and the same ask/act rates, while allowlist validity rose to 100%. It averaged 344 ms and 96.7 MB engine RAM. The contract was fixed; the ordinary training distribution was still too eager to act.

A final 3,600-row safety-flat run improved the generated suite but not the independent ask/act boundary. Its CQ4 bundle measured 56.5% F1, 52.6% profile accuracy, 77.8% clarification recall, 22.2% unsafe dispatch, and 100% allowlist validity at 350 ms mean. Quantization happened to move this small suite favorably; one unsafe launch in 4.5 blocked cases is still a decisive “suggest, do not launch.”

Inference fits clients; training fits workstations

The heaviest local fine-tune took 27.9 minutes, averaged 8.1 CPU cores, reached 7.63 GB maximum RSS, and reported an 8.17 GB peak footprint. Sixteen gigabytes is a practical developer-machine target; eight is too tight after the OS and tools. The final CQ4 inference process peaked near 142 MiB RSS. Apple and Android SDK support makes mobile integration plausible, but phone, battery, thermal, x86, Raspberry Pi, and wearable latency still require measurements on those devices.

Where I would put it

The product architecture has five stages. Deterministic code first handles permissions, authentication, unavailable harnesses, organization policy, and worktree conflicts. Needle then proposes a parameterless capability route and generic preflight actions. A validator enforces mutual exclusion, allowlists, mandatory gates, and abstention. A trusted compiler preserves the request and expands approved recipe keys. Finally, ACP, a native SDK, or a CLI adapter launches the harness.

The first useful surfaces are reversible: a “needs one decision” state, a recommended capability chip, a read-only repository preflight, a suggested parallel-worktree topology, a specialist lens, and an explicit completion gate.

I would route to stable capabilities before vendor names. Deterministic product code can then resolve `deep_reasoning` or `security_review` to an installed harness based on authentication, availability, cost, privacy, platform, and user preference. Direct dynamic harness selection is an interesting later step, not the safest first abstraction.

The unseen-catalog ablation makes that concrete: description-grounded fine-tuning raised fresh-name selection from 3.9% to 23.9% with 100% structural validity. A sixfold gain is scientifically useful; 23.9% is not a product router.

The missing objective

“Ask versus act” should not be learned only as another equally weighted tool name. The next training program should emphasize minimally different hard pairs, real rejected dispatches, explicit abstention, high-recall rules, and preference data in which clarification is chosen over a plausible fabricated call.

This connects directly to When2Call, whose authors found preference optimization more effective than ordinary supervised fine-tuning for call, ask, and abstain decisions. Needle's public trainer would need new support for that objective.

The decision

Would I put Needle into a multi-harness coding-agent product? Yes—in shadow mode immediately, and as a visible suggestion layer after evaluation on real traffic.

Would I let it silently rewrite every prompt, decide permissions, or launch a security-sensitive run? No.

Tiny specialist models can make agent products faster, more private, and more responsive. Their size does not remove the need for product architecture. It makes that architecture more important.

The fast part was easy. The work was deciding exactly what the model was allowed to mean.