Before the AgentOPENAI BUILD WEEK · DEV TOOLS

OPENAI BUILD WEEK · CODEX · LOCAL AI

A local preflight decides how Codex should begin.

A fine-tuned 26M model proposes a bounded route. Deterministic code validates it, visibly compiles the handoff, and waits for you before Codex starts.

Try the product See the five-stage flow
USER PROMPT“Make the API
fast enough.”
LOCAL26MNEEDLE
ASKINSPECTDELEGATEROLEVERIFY
STRUCTURED DIRECTIVESNOT FREE-FORM REWRITING
45.8 MBtuned CQ4 bundle
350 msmeasured M4 mean
14bounded control tools
0 cloudrequests during routing
00

THE WORKING PRODUCT

REAL MODEL · REAL VALIDATOR · EXPLICIT HANDOFF

The architecture is no longer just a diagram.

The repository now includes the tuned model, a one-command local browser demo, the complete five-stage pipeline, safety tests, and an inert-by-default Codex adapter.

Open the judge demo ↗
01DETERMINISTICHard gates

Repository state, conflicts, harness availability, and high-impact operations.

02LOCAL CQ4Needle proposes

One capability plus parameterless preflight actions on CPU.

03POLICYValidate

Mutual exclusion, allowlists, budgets, and mandatory safety gates.

04TRUSTEDCompile

Versioned recipes around the verbatim original request.

05HUMAN-GATEDCodex handoff

A visible command that remains inert until explicit confirmation.

RUN LOCALLY./scripts/install-demo.sh && ./scripts/run-local-demo.shinstructions ↗
01

THE VERDICT

Yes—
with a hard constraint.

Use Needle to select trusted prompt recipes. Do not ask it to silently author the final prompt.

Needle is a specialist function caller. Its native move is to match a request to a tool, copy short values, and emit JSON. That is a good fit for routing; it is a poor reason to trust 26M parameters with open-ended prose, policy, or permissions.

+

THE USEFUL FORM

Decision model

“Inspect tests.” “Ask for scope.” “Use the security recipe.” “Delegate these independent tracks.”

×

THE RISKY FORM

Prompt ghostwriter

Unbounded rewriting can drift from user intent, invent authority, and mix untrusted repository text into the control plane.

02

WHAT IT CAN DECIDE

Six moves before
the first tool call.

Each result is a short, allowlisted directive. The harness owns the wording, policy, and permissions that follow.

01

Clarify

Detect a missing product, scope, platform, or acceptance decision before code is touched.

ask_clarifying_question
02

Inspect

Point the harness toward tests, API paths, schema, dependencies, configuration, or security boundaries.

inspect_repository
03

Delegate

Recommend parallel work only when tracks are genuinely independent and have an integration contract.

delegate_parallel
04

Role

Select a trusted security, testing, performance, migration, accessibility, or database recipe.

apply_specialist_role
05

Verify

Attach tests, builds, benchmarks, migration dry runs, accessibility checks, or security regressions.

require_verification
06

Proceed

Preserve an already actionable prompt instead of decorating every request with extra process.

proceed_directly
03

ROUTER LAB

Replay the measured decisions.

These are real outputs from the locked 60-case external suite—not a browser simulation of the model.

Choose a decision

LOCAL EVAL REPLAY
USER REQUEST

“Make the API fast enough.”

26M router
FINE-TUNED OUTPUT
[
  {
    "name": "ask_clarifying_question",
    "arguments": {
      "question": "acceptance"
    }
  }
]
TRUSTED COMPILER

Ask for the missing performance target and workload before implementation.

ObservedCorrect on this case—but clarification was still the weakest learned route overall.
04

HARNESS BLUEPRINT

Small model.
Strong frame.

The router stays read-only. Deterministic code validates its output and compiles the final handoff.

01TRUSTEDUser request

Original intent stays intact.

02DETERMINISTICRepo inventory

Tests, routes, schema, config.

03LOCAL MODELNeedle router

One or more directives.

04POLICYValidate + compile

Enums, scope, permissions.

05FRONTIER AGENTExecute + verify

Code, tests, evidence.

A

Inventory, don’t ingest

Needle has a 1,024-token encoder. Feed it trusted metadata—not a whole repository or arbitrary instructions copied from comments.

B

Compile, don’t paraphrase

security_engineer indexes a reviewed prompt fragment. The model does not invent what “secure” should mean today.

C

Abstain, then escalate

The released CLI exposes no calibrated confidence. Instrument margins, calibrate on holdouts, and default uncertain paths to rules or user input.

05

LOCAL EXPERIMENT

APPLE M4 · 10 CPU CORES · 16 GB MEMORY

It trained.
It learned.
It did not generalize enough.

One 158-second CPU epoch turned an unusable new tool contract into a promising prototype. A locked suite with novel wording exposed the remaining gap.

Base Fine-tuned
Tool-name F1+23.5 pp
34.9%
58.4%
Exact tool set+33.3 pp
16.7%
50.0%
Exact full call+31.7 pp
0.0%
31.7%
Argument schema validity+58.2 pp
41.8%
100.0%
780training rows
140calls per tool
41optimizer steps
60external cases
12multi-action cases

External evaluation: 48 novel single-action prompts + 12 two-action prompts. Grammar-constrained decoding enabled. Percentages are from project artifacts; no claim is a Cactus production benchmark.

06

THE DEEPER STUDY

4,800-ROW SCALE SET · 3,600-ROW SAFETY ABLATION · 96 PRODUCT-BLIND CASES

More data helped.
Then “ask vs act” became the whole story.

PRODUCT DECISIONADVISORY

Fast enough to run locally. Useful enough for suggestions. Not safe enough to own autonomous dispatch.

DATA-SCALE RESULTS

Product-blind metrics use independently authored prompts—not another sample from the training generator.

CheckpointRowsLocked F1Blind F1Blind exactAsk recallUnsafe act
Base028.2%26.8%0.0%0.0%16.7%
Small96071.9%69.4%17.7%38.9%61.1%
Medium2,40088.5%75.3%27.1%55.6%44.4%
Full4,80089.2%75.1%27.1%50.0%50.0%
Safety3,60090.7%75.6%27.1%83.3%16.7%

The synthetic trap: a keyword baseline scored 95.2% F1 on the generated locked set and only 56.3% on product-blind prose. Same-generator benchmarks can flatter both rules and models.

The ablations: a second pass over the same 960 rows helped, but a third regressed exact calls and unsafe dispatch. Description-grounded tuning lifted unseen-harness selection from 3.9% to 23.9%—still far below a launch threshold. Distribution and architecture mattered more than repetition.

CACTUS CQ4 · CPU BACKEND · 20 TIMED CALLS

Production-path speed,
measured on this M4.

The full seven-action catalog ran through a locally built Cactus C++ engine. Model initialization was 1.21 s; post-warm-up latency stayed between 307 and 327 ms.

315 msmean end-to-end
1,333prefill tok/s
1,470decode tok/s
~73 MBengine RAM
~3.7CPU cores average
37 MBexpanded CQ4 bundle
FINE-TUNED EXPORT FINDING

The argumented CQ4 checkpoint kept 70.2% tool F1, but only 22.9% of calls used allowlisted values. Moving bounded choices into parameterless tool names raised validity to 100%. The final safety-flat CQ4 bundle reached 56.5% F1 and 77.8% ask recall at 350 ms—but still dispatched 22.2% of blocked cases. The output contract is fixed; autonomous policy is not.

WHERE IT FITS IN A MULTI-HARNESS ORCHESTRATOR

Put the model between two deterministic layers.

Hard gates remove decisions the model must never make. A validator then checks the proposal before a trusted compiler hands work to ACP, a native SDK, or a CLI adapter.

NOW

Composer suggestions

Surface a recommended capability, missing decision, preflight inspection, or completion gate before submission.

NOW

Shadow router

Compare a local proposal with the user’s actual harness choice and turn corrections into evaluation data.

NEXT

Capability resolver

Choose a stable profile first; resolve it to an installed harness with deterministic auth, cost, privacy, and platform policy.

LATER

Guarded defaults

Auto-select only low-risk routes when hard rules and the calibrated model agree. Keep ambiguity and high-risk work confirmed.

Read the complete experiment ↗Open the research trail ↓
07

FIRST-PROTOTYPE FAILURE MODES

The tiny model
needs adult supervision.

These numbers preserve the original 780-row prototype. The deeper study above improves several of them without changing the core safety conclusion.

01

Clarification misses

The first prototype recalled only 25% of external clarification cases. “Make it better” could still be routed as execution.

02

Composite collapse

The first prototype's two-action exact full-call accuracy was 8.3%. Diverse multi-call data later fixed the template without fixing safe abstention.

03

Valid ≠ correct

Fine-tuning reached 100% argument-schema validity while exact full-call accuracy remained 31.7%.

04

Untrusted context

Repository comments and docs can contain prompt injection. Metadata collection must not turn data into authority.

05

No calibrated confidence

The public JAX generator uses greedy decoding and does not return an abstention-ready confidence score.

06

Narrow by design

Needle emits calls; it is not a conversational planner, code reasoner, or substitute for the downstream agent.

NON-NEGOTIABLE GUARDRAILS

Validate the control plane.

  • ✓ allowlisted tool names + enum values
  • ✓ dedupe + cardinality limits
  • ✓ no authority expansion
  • ✓ original/final prompt diff
  • ✓ shadow-mode evaluation
  • ✓ user handoff for uncertainty
08

FROM PROTOTYPE TO PRODUCT

Earn the right
to automate.

Measure each layer separately: route quality, prompt preservation, and downstream coding outcome.

  1. 01

    Observe

    Collect human route labels. Do not change prompts.

    NO AUTOMATION
  2. 02

    Shadow

    Run Needle locally, validate, and compare.

    LOG ONLY
  3. 03

    Assist

    Show recommendations to the user or operator.

    HUMAN CHOICE
  4. 04

    Gate

    Auto-apply only directives above their own precision bar.

    HIGH PRECISION
  5. 05

    Expand

    Add clarification and composite routing last.

    MEASURE DRIFT
09

RESEARCH TRAIL

Primary sources,
not vibes.

Model and company claims link to Cactus, Hugging Face, GitHub, and Y Combinator. Architecture choices draw from the original papers and official agent-safety guidance.

Model

Needle announcement

Cactus Compute

Code

Needle source + architecture

Cactus Compute

Weights

Needle model card

Hugging Face

Company

Cactus — Summer 2025

Y Combinator

Paper

Meta-Prompting

Suzgun & Kalai

Paper

SWE-agent

Princeton NLP

Paper

CodePlan

Microsoft Research

Paper

Repo-Level Prompt Generation

Google Research

Paper

DSPy

Stanford NLP

Guide

Building effective agents

Anthropic

Safety

Understanding prompt injection

OpenAI

Safety

Prompt Injection Prevention

OWASP

Eval

BFCL V4

UC Berkeley

Paper

When2Call

NVIDIA

Data

APIGen

Salesforce Research

Protocol

Agent Client Protocol

ACP

Product

External agents

Zed

Runtime

Cactus engine

Cactus Compute

Sources checked July 17, 2026. Company benchmark claims are attributed to Cactus. Local measurements, datasets, manifests, and case-level results come from the reproducible artifacts in this owned local project.