Two Random Vectors and Qwen3-4B: What 32% to 52% Actually Measured

Qwen3-4B, one of the better small open-weight models you can run on a consumer GPU, received a correct last-integer grade on 8 of our 25 multi-step arithmetic tasks, 32%, in one greedy run in thinking mode. In the stored traces of those same runs, the expected integer appears somewhere in at least 20 of the 25. Prepend two vectors of random noise to the embedding sequence, hold the decoding configuration fixed, and the stated-accuracy figure moves to 51.6%, 129 correct grades across 250 generations, ten seeds by 25 tasks.

We published that result in an essay, and we built a nearly three-hour video on it. Then Igor Rivin (@igorrivin) independently ran the direct control we had preregistered and never executed, on an NVIDIA GH200, and submitted the results as pull requests #4 and #5. The repository verified the artifacts against its stored result files, merged both pull requests, and published a correction on 2026-08-27 that withdraws the capability reading of the number. That attribution records his repository contribution; it does not imply his endorsement of Irys or of this article.

This article is the result with its receipts, including the receipt that withdrew the headline. It is a piece of Irys research, not a product claim, and we keep that line visible throughout: everything below ran on open models, on our own synthetic task set, in a pipeline you can inspect. Where our published essay got a number wrong, we say so. Where a claim died, we show the control that killed it and who ran it.

Repository status as of 2026-08-30. The perturbation program is archived unmodified under legacy/ in the public repository (HEAD af90eb5). The canonical correction is legacy/docs/CORRECTION_NESTED_ARITHMETIC_2026_08.md. Its opening line says that external articles which promoted the withdrawn claims link there. This is one of those articles. The video production built on the original reading was nearly complete, with a script, a 59-scene plan, a timing lock, narration audio, draft renders, and an upload package; it was stopped on 2026-08-30 before the final render and before any version was published.

Every number wears a label

Because this is first-party evidence, we use a fixed grammar of exactly four evidence classes: first-party measured (our experiment or a verified repository fact, with n and denominator), external (Igor Rivin's contributed controls, third-party papers, vendor figures), modeled (arithmetic from stated assumptions), and withdrawn (a previously public interpretation the repository has formally retracted). A measured number can stay measured while its interpretation is withdrawn. A result that was never run is omitted, not labeled as pending; a public result produced under a known defect is withdrawn. Claims also carry a reasoning role: an observation is what we saw; an interpretation is the narrowest reading consistent with it; a hypothesis is our leading explanation; an open test is the experiment that would kill it.

If an article is going to argue that the industry mislabels its numbers, it had better label its own.

The finishing signature

The test is deliberately friendly: 25 nested arithmetic expressions with three to six sequential operations, generated by our own code from a fixed seed. Not a public benchmark, and 25 tasks give little resolution near 100%. The model, quantized with bitsandbytes NF4 on an unpinned software stack, decodes greedily with a 1.2 repetition penalty on every arm, in Qwen thinking mode, under a 1,024-token budget. Those last two settings turned out to be the load-bearing ones. Grading is one line: extract the integers in the graded string, and the task is solved if the last one equals the expected value.

Two measurements reframe the 32%.

Answer-anywhere (first-party measured, a lower bound). Text search finds the expected integer in at least 20 of the 25 stored baseline traces. This is a search over output text, not a probe of hidden states, and the traces were stored truncated at 2,000 characters, which makes 80% a floor. The narrow sentence: the model writes down the right value in at least four runs of five, and receives a correct last-integer grade in 8 of 25.

Natural finish (first-party measured, conditional on finishing). Every generation ends either with an end-of-sequence token or by hitting the cap. Across 500 perturbation-arm generations (250 on Qwen3-4B, 250 on Qwen3-8B, 25 tasks by 10 seeds each), 126 finished naturally: 94 of 250 on the 4B, 32 of 250 on the 8B. All 126 were correct. The conditional claim is 126/126, not 500/500; the other 374 hit the cap, and 75 of those (35 on the 4B, 40 on the 8B) contained the expected integer somewhere in the trace without ending on it. The repository's own reanalysis, which adds the baseline arms, counts 100/100 on the 4B (94 perturbation plus 6 baseline) and 34/34 on the 8B (32 plus 2). The 126 naturally finished perturbation-arm generations averaged about 718 tokens; truncated generations reached the 1,024-token cap. Among the 4B's 156 truncated perturbation-arm generations, 35 received a correct grade, 22.4%, which the correction identifies as the base rate for the last integer of a severed trace. The 126/126 figure is retrospective: EOS was conditioned on after examining stored outcomes, not selected in advance as a predictor. It is a descriptive association, not a preregistered test, a deployable correctness signal, or a general estimate. That conditional-correctness result holds on the analyzed Qwen3-4B and 8B arms; it does not extend to every model.

These are correlations on one task family, measured through text. At the time we read them as a capability-expression gap: the model surfaces the right value far more often than it delivers it. The correction re-reads the same two counts as a completion-under-cap signature, and they were, in fact, the clue. That the failures sit in the opening tokens, where the model commits to a template of headers and enumerated steps and then rides it into the wall, is an interpretation drawn from reading traces, not something these counts establish. Our explanation for the template habit, preference tuning rewarding organized-looking openings, is an untested hypothesis about termination behavior.

The reviewer-side version of this failure is familiar to anyone who supervises legal drafting: a memo with immaculate headings and an unfinished analysis steers scrutiny away from its own gap. Polished completion is not legal correctness. Irys One is designed to keep source verification distinct from formatting; that is a design boundary, not a claim that formatting failures of this kind are detected automatically. Its research workflow checks citations against primary sources rather than trusting a well-formed footnote.

The intervention, and what honest reproduction has to handle

Draw two vectors of standard-normal noise, 2,560 coordinates each (the model's embedding width), from one generator seeded at 2024 and drawn sequentially. Renormalize each so its RMS exactly equals the embedding matrix's scale, measured at 0.02195 for the Qwen3-4B weights we loaded. Prepend them at sequence positions 0 and 1, attention mask on, holding the model-side decoding configuration fixed. Two things still change besides the prefix: every real token's position index shifts by two, and the two arms later pass through asymmetric post-processing before the same last-integer predicate is applied.

The repository audit we ran before drafting narrows the reproduction promise in five ways:

• No model revision was pinned anywhere in the codebase at the time these results were produced. The pipeline is inspectable; the exact weights are not certified. The correction adds pinning of model revisions and chat-template versions as a process rule.
• Two RMS conventions exist. Arithmetic renormalizes each vector exactly. Planning, legal, and text-generation runs use randn * rms, which matches the target RMS in expectation only, and draw from per-seed generators numbered 42 through 46 instead of the shared 2024 stream.
• The n=3 scouts are not separate runs. They are the first three prefixes of the n=10 sequential stream from the generator seeded at 2024.
• The two arms were not graded on identical strings. The baseline path strips think blocks through post-processing the perturbed path does not share. How often that changed a verdict was not counted.
• Generated-token counts on the perturbed arm were under-reported by the prompt length, about 60 tokens per generation, a bug our internal audit identified and pull request #4 fixed. That is why no per-arm token-count comparison appears in this article, and why the essay's worked contrast of a 400-token messy solution against an 800-token formatted failure is withdrawn rather than repeated.

The two-vector dose was also chosen because the sweep peaked at two on these same 25 tasks, so its effect size was never replicated on held-out tasks.

The evidence table

Every first-party figure in this article, with its evidence class, task denominator, seed count, and selector. Repository links go to the archived artifact at HEAD; where the exact artifact for a number could not be located, the row says so.

The scout caveat, in words: the first three-seed scout said 60%; ten seeds said 51.6%. Small samples flatter. The essay also reported a no-chain-of-thought arm at +0 points and a comparison of optimized noise directions against random draws; neither artifact could be located at HEAD, the direction comparison's computation path and arm sizes were not recovered, and this article draws no conclusion from either.

One note on the headline p-value, because it deserves scrutiny. The essay reported p = 0.000015 as a McNemar test. No script in the repository computes McNemar. The number reverse-engineers to a two-sided exact sign test with tasks as its units: 17 tasks where at least one of ten seeds beat the greedy baseline against zero where the baseline beat all ten. The 8B's p = 0.000177 is likewise a task-level test, a continuity-corrected chi-square on 16 oracle-versus-baseline gains against 0 losses. Neither has a seed-as-independent-sample defect; their limitation is that both compare oracle-over-ten against one greedy baseline and therefore test neither the 51.6% nor the 28.8% single-seed mean. The nesting problem shows up elsewhere: in the GH200 diversity study, an unpaired Welch test over ten seed-level accuracy rows gave p = 0.037, and the same data paired at task level gave p = 0.33 by paired t-test and p = 0.29 by Wilcoxon. The correction's process rules now require paired or clustered analysis on task-nested samples, and note that the repository's own published McNemar values had this flaw.

The control we did not run

Every arithmetic run above used Qwen thinking mode under a 1,024-token cap. The direct control, the cheapest remedy for a model that does not finish, is to turn thinking mode off with one chat-template flag. Our internal audit preregistered that control in the spring of 2026. We did not run it. Igor Rivin did, on an NVIDIA GH200 at bf16, and the validity assessment he contributed reports the result.

With thinking disabled, on the same 25 tasks under the same cap, one greedy run per model: Qwen3-1.7B 24/25, 4B 25/25, 8B 24/25, 14B 25/25, 32B 25/25, finishing in roughly 220 to 300 tokens, with zero truncations at every size. That is a separate experiment on a separate stack, not a same-run flip from 32% to 100%. The within-GH200 comparison that isolates the flag on the 4B is 5/25 with thinking on (21/25 truncated) against 25/25 with thinking off (0/25 truncated, 221-token mean). The 32B's 0% on our stack was 25 of 25 generations hitting the cap; at bf16 with thinking on it scored 1/25 with 25/25 at the cap, and 25/25 with thinking off, so quantization did not explain the failure, and the essay's "likely a quantization artifact" is withdrawn. The assessment puts it plainly: the benchmark has no dynamic range. It measures whether a model stops within 1,024 tokens. Twenty-five tasks that every size solves also cannot distinguish sizes, and the ladder proves nothing about arithmetic competence beyond this task family.

Read against that control, here is where the 19.6 points came from on Qwen3-4B. Baseline stated accuracy was 8/25: 6/25 naturally finished and correct, plus 2/25 cap-hit runs that received a correct grade (the 2 is derived by subtraction and passes through the baseline's asymmetric post-processing; it is not a directly tabulated raw-trace count). The perturbation mean was 129/250: 94/250 naturally finished and correct, plus 35/250 cap-hit runs that received a correct grade. The finish-rate contribution rose 13.6 points, from 24.0% to 37.6%; the cap-hit-grade contribution rose 6.0 points, from 8.0% to 14.0%. The score moved through two buckets, natural finishes and whichever integer survived in severed traces, so the termination shift does not account for every decimal on its own. No confidence intervals accompany these point estimates; with ten seeds nested within 25 tasks, generation-level binomial intervals would misstate uncertainty by ignoring the clustering, and valid task-clustered intervals were not computed. The no-think control removes truncation from the picture entirely, which is what kills the capability reading.

The correction withdraws five claims from the README: that perturbation reaches capabilities scaling cannot; that it beats temperature sampling at equal cost (72% versus 64% plurality); the cost-per-capability comparison against the 32B; the reading of per-model deltas as a model-dependent computation mechanism; and the 32B quantization explanation. The withdrawn readings stay in the evidence table with their labels because the numbers were measured; what died was what they were taken to mean. Also withdrawn by extension: "diverse prefixes plus room to reason, both required," "scaling bought four points and interrogation bought forty," the per-precision mechanism story for the 8B, the 800-versus-400-token footprint contrast, and the claim that candidate generation had become cheap enough to make selection the research program.

What the correction says survives: "the bottleneck is convergence, not knowledge" is supported and strengthened. Under a binding cap in thinking mode, the two vectors reliably shift trajectories toward completion. On arithmetic this is moot, because disabling thinking is cheaper and better. It is not evidence of improved reasoning.

Four mechanisms still fit the evidence

We do not know why the vectors change termination. Four candidate accounts, each with a prediction, support, pushback, and a decisive test, and each now a hypothesis about completion under a cap rather than about capability:

• Stochastic resonance. A thresholded system, a sub-threshold signal, and noise that lifts the signal's peaks over the line. Greedy decoding is aggressively thresholded; if completing trajectories sit just below the argmax boundary, noise is activation energy. Predicts content-indifference, an inverted-U dose curve, and a signed boundary: help for stuck models, harm for cruising ones.
• Position shift. The +2 index shift alone perturbs the computation. The zero-vector control shows the occupancy-plus-shift bundle is worth +4 points over the 32% baseline; the same shift with diverse content is worth +19.6, with occupancy and shift bundled rather than factored. Decisive test: shift indices with nothing inserted, at full scale.
• Attention redistribution. Content-free inputs re-spread attention and shift feed-forward activations, the account Shi et al. give for meaningless discrete tokens. We have not measured our own activation statistics.
• Sink disruption. Early positions become attention sinks (Xiao et al.); noise in the sink seats scrambles degenerate early patterns. The observation it would explain, uncontrolled and from one planning task: the baseline halted after fourteen incoherent words, while every perturbed seed produced a 650, 710-word plan. We never imaged the attention maps.

Whether the diversity is even ours. On the GH200, where greedy decoding is itself nondeterministic, a fixed-prefix numerical-noise null matched randomized prefixes on every ensemble metric: on 25 wide_mult tasks at k=10, each arm produced plurality@10 of 17/25, oracle@10 of 22/25, and rescued 8 of 10 baseline failures, with one discordant task in either direction on each paired endpoint, McNemar p = 1.0. On our RTX 5090 / Windows stack, the same probe decodes byte-identical inputs identically across three processes (a zero noise floor) while perturbed inputs diverge, on the two of five tasks completed before a hardware power fault stopped the run. Perturbation is a causal intervention on a deterministic stack; whether it is useful on a task the model cannot already solve is unresolved on either stack. The diversity study holds the details. The old sentence "diversity, not presence, carries the effect" is therefore too strong; the licensed version is that diverse prefixes raised termination in our setup.

Three outside groups report related observations at different levels of the stack: random Gaussian soft prompts matching trained ones (Kim et al.), meaningless tokens changing reasoning through activation shifts (Shi et al.), and filler tokens carrying hidden computation (Pfau, Merrill & Bowman). They make the general phenomenon less surprising. They do not replicate this intervention, task set, or grading path, so they do not rule out a local confound, and this article is the record of one.

The falsifier we ran, and what it now means. The essay listed a temperature comparison as future work: if plain temperature sampling at n=10 matched perturbation, we had rediscovered sampling. We ran it, 25 tasks and ten runs per arm, selectors reported as mean stated accuracy / plurality@10 / oracle@10. Perturbation led every tested temperature on mean (51.6 vs 37.6 / 41.2 / 39.2) and on plurality (72 vs 64 / 60 / 48), and temperature 0.6 tied the oracle at 100%. At the time that killed only the claim that perturbation alone reaches every answer. After the no-think control, the whole comparison inherits the cap confound: both arms were measuring termination, so "beats temperature at equal cost" is withdrawn, and the voting advantage did not survive either.

What is still open. A preregistered probe on a sink-free architecture is frozen behind a hardware blocker, with its own written rule: no article-level mechanism claim until it runs. The position-shift ablation exists only as a two-task smoke test on a 0.6B model, which showed position shift alone sending the model into "step by step by step" loops. That licenses no conclusion, and it means the confound is capable of large effects on its own. Stochastic resonance leads as the organizing hypothesis because it arrived with the full pattern pre-written, not because the others are refuted, and no contender explains a capability improvement, because none was measured.

Selection is a worked example, not a program

Ten seeds hand you ten answers. Every way of choosing is a selector, and a result quoted without its selector is a lie. On the arithmetic set the selector question is moot, since a template flag delivers 25/25 with no pool at all; the mechanics still matter for anyone reading a best-of-k number elsewhere.

• Oracle@10 (any seed correct, answer key in hand): 25/25 on the 4B, 20/25 on the 8B. A retrospective ceiling. Never deployable, and never a system's latent capability.
• Plurality@10 (most common answer; ties go to the first seed): 18/25 on the 4B, 14/25 on the 8B. Automatic, but it needs comparable answers.
• Strict majority@10 (more than half agree): 10/25 on the 4B, below the 129/250 single-seed mean though above the 8/25 baseline. On the 8B: 3/25 against a 4/25 greedy baseline.

Correct final integers coincide because the task defines one correct value. Wrong answers scattered in this pool because each severed trace ended somewhere different; that is a property of truncation, not a law of ensembles. In this retrospective, cap-confounded 25-task pool, plurality outscored strict majority because majority demanded consensus from a pool that was mostly wrong. That is a worked selector example, not evidence of model capability and not a deployment rule.

We built a learned scorer to close the distance between the single-seed mean and the oracle: a small network of about 1.45 million parameters (our essay said 300K; the audit of our own code corrected it), scoring candidate latents at the cost of a matrix multiply. It was nondeterministic. The encoder produces 2,560-dimensional representations; the scorer consumes 1,024-dimensional latents; the projection bridging them was initialized from the global random number generator every session, so the scorer rated candidates through a randomly different lens each run, and 9 of 12 legal tasks collapsed under it. The fix (a fixed seed) is in the code; the clean rerun was never performed. Every scorer-selected and evolution-selected result the essay published is withdrawn, including a striking incident-response contrast (from "rotate credentials, check the logs" to honeypot deployment and MITRE ATT&CK lateral-movement tracking) that survives only as an illustration of the defect. The defect is first-party measured; the results it produced are not results.

A selector fit to use must be deterministic, validated on held-out tasks, inspectable, and stable under rerun. None of those is about intelligence. Because stored configurations and outputs preserved enough lineage, the affected claims could be traced to specific artifacts once the control existed; that record did not prevent the skipped control. The Irys Swarm API uses source-and-worker lineage in a different document-reasoning system, one whose findings Irys describes as traceable to the document they came from, the step that created them, and the evidence they rest on; this experiment did not use it, and it does not use perturbation.

Where the effect exists, and where it does not

• Qwen3-4B: stated accuracy 32 → 51.6 (n=10); the answer-anywhere floor barely moves, at least 20/25 to at least 205/250, under text search over 2,000-character stored traces. Licensed reading: a termination aid under the cap.
• Qwen3-8B (8-bit): 16 → 28.8 (n=10); the floor moves from at least 8/25 to at least 125/250; natural finishes from 2/25 to 32/250. The identical termination coupling.
• DeepSeek-R1-Distill-1.5B: 76 → 74.4 (n=10). The only model that finished every baseline task, and therefore the only capability measurement in the set; perturbation reduced its termination from 25/25 to 91% and its stated score by 1.6 points. A termination observation, not a confirmation of any mechanism.
• phi-2: 12 → 18.7 (n=3, 4-bit), with a sampled oracle of only 28% and no termination rate reported, so the "low ceiling" reading is uncontrolled.
• 14B and 32B: single-seed anecdotes (n=1), not comparable; the 32B's 0% is 25 of 25 generations at the cap, not a quantization artifact.

The essay's readings of these rows as a "convergence aid" on the 4B and an "exploration aid" on the 8B, and the usage rule built on them ("measure the surfaced-versus-delivered gap; a wide gap says perturb"), are withdrawn with the per-model mechanism claim.

The quantization switch (first-party measured, one model pair). Qwen3-8B at 4-bit: +1.3 points on 25 tasks across ten seeds, no detectable effect in this setup, with the baseline at the cap on 24 of 25 tasks. The identical experiment at 8-bit: +12.8 points mean effect (n=10). Both cells are termination-confounded. The essay's mechanism for the difference, four-bit rounding leaving 16 representable weight levels and eight bits leaving 256, is withdrawn as a per-precision reading of a cap-confounded delta.

Reproduction heuristics for this pipeline, not deployment rules: run the no-think control first, because on our arithmetic set it ended the experiment; report termination beside every accuracy figure; if you rerun the 8B arms, expect the 8-bit and 4-bit cells to differ; and check the sampled oracle before spending on diversity.

Applied tasks (first-party measured; the correction labels all of them suggestive and uncontrolled). On five planning tasks, an LLM judge scoring coherence, correctness, completeness, specificity, and actionability compared one output per arm, the best-scoring seed of five standing for the perturbation arm; it preferred that arm on 3 of 5 and greedy on none. The other 2 of 5 were evolution-selected under the defective scorer and are withdrawn; they are not part of the surviving hypothesis. On twelve legal tasks (FTC unfairness, GDPR classification, contractor misclassification, veil piercing, and others), a blind five-dimension rubric scored the judge-selected best of five seeds above the single greedy baseline on 11 of 12, mean lift +1.6 on a ten-point scale. Most individual seeds produced lateral moves or mild regressions against the baseline; the wins concentrate in the best seed of each pool. These are open-model research results under an LLM-judge protocol, not an Irys One product-accuracy benchmark, and no lawyer verified the law in them. The best seed on contractor misclassification applied California's ABC test, New York's economic-reality standard, and Texas common law with per-jurisdiction exposure estimates; on veil piercing, one seed named the "real operating business, not a sham shell" element every other output missed. Blinding removes condition bias. It does not make the judge legally omniscient, and these tasks have no no-think baseline, no temperature-matched best-of-five, and no null arm. Those controls are the difference between a hypothesis and a result, and they have not been run. The one legal loss, GDPR controller-versus-processor classification, is another termination signature: the seeds burned their full 2,048-token budgets in thinking loops.

On 15 open-ended prompts (first-party measured, same missing controls), pairwise LLM-judge results were 7-1-7 against the 4B baseline and 5-7-3 against a 14B, with disclosures: the stored artifact keeps arm-name keys, so blinding is not verifiable from the files; the perturbation side used a completion-biased best-of-5 selector (finish with EOS, then prefer length) against single-run baselines; and perturbation never won on 7 of the 15 prompts. That diversity competes with scale only inside the small model's knowledge is a hypothesis.

The hard border. On a data-breach notification task, all eleven outputs, baseline, seeds, and evolution, cited statutes that do not exist, with fictional agencies and wrong deadlines, fluently. The rubric scored them fine. No selector inside the model, oracle included, can detect absent knowledge, because every candidate is wrong in a different plausible way; detection requires an outside reference. Perturbation changes how a model finishes what it contains. It is powerless, and dangerously fluent, about what a model lacks. In legal work, missing knowledge is met with source-backed retrieval, the job Irys One's citation checks against connected legal databases exist to do, not with more candidates.

The economics, in three labeled panels

Measured. Single greedy run, thinking mode, last-integer grading, same 25 tasks: Qwen3-4B 8/25, Qwen3-14B 9/25, both termination-limited (the 14B hit the cap on 19 of 25). The essay's like-for-like framing, scaling bought four points while the prefix bought 19.6, is withdrawn: both deltas are termination deltas under a defective setup, and the like-for-like comparison is with the chat-template flag that took the separate GH200 bf16 ladder to 96, 100% at roughly a fifth of the tokens.

Vendor-reported (checked 2026-08-30). Qwen's official speed benchmark reports two separate tables. Under SGLang with AWQ-INT4 on an H20, at a one-token context: Qwen3-4B 199.7 tokens/s, 14B 96.5, 32B 47.7; that table reports no memory because SGLang preallocates it. Under Transformers with AWQ-INT4, a separate run: 2.9 GB, 10.0 GB, and 19.1 GB of GPU memory, at 51.6, 44.7, and 41.8 tokens/s. The memory figures and the fast throughput figures come from different serving stacks and are not one measurement, and neither was taken on a consumer card. NVIDIA's launch pricing: GeForce RTX 5060 Ti 16 GB at $429 (April 2025) and GeForce RTX 5090 at $1,999 (January 2025), a 32 GB GDDR7 card; launch MSRPs, not street prices. Runpod serverless, as listed on 2026-08-30: $0.58 per hour for 16 GB workers; $1.15 (RTX PRO 4500) to $1.58 (RTX 5090) for 32 GB. These are vendor facts; they no longer support any thesis about the intervention.

Modeled (the essay's scenario, dated to its assumptions, and now a dead model). Assume ten runs at a 2,048-token budget, 20,480 tokens, at a commodity rate of $0.40 per million tokens; the essay did not tie that rate to a named provider, so it stands as an assumption. Generation-only: $0.008 per query. The essay added about $0.001 for the learned scorer, for $0.009 all-in. Recomputed: a 1.45M-parameter scorer costs roughly 2.9 million floating-point operations per candidate, about 29 million for ten, against roughly 1.6 × 10¹⁴ for the ten 4B generations, a ratio under one in a million, so the scorer line is effectively zero and the essay's $0.001 was a placeholder, not a calculation. Both essay figures are kept because they were published. Assume a frontier reasoning trace of 15,000 output tokens at the essay's $30 per million: $0.45. Generation-only, 56x; all-in, about 50x. That ratio held task quality equal by assumption; the assumption is withdrawn, because on this task set the 4B did not deliver frontier-equivalent arithmetic through perturbation, and the flag was cheaper than either. The arithmetic stands; the decision it was built to support does not.

Current-price sensitivity (modeled; the commodity denominator is the essay's unnamed $0.40/M assumption, not a current named provider rate): OpenAI's GPT-5.6 Sol model page, as of 2026-08-30, lists promotional pricing of $20 per million output tokens through at least 2026-11-21. At $20/M the same 15,000-token trace costs $0.30: 37.5x generation-only and 33.3x all-in at the assumed commodity rate. At 10,000 queries a day for 30 days, the modeled comparison is $2,700 against $90,000 at today's listed price, against $135,000 at the essay's $30/M assumption. The essay's broader sensitivity box, frontier traces between 8k and 25k tokens and commodity rates between $0.30 and $0.60 per million, spans roughly 18x to 105x all-in at $30/M; scale each end by two thirds at $20/M. Every empirical benchmark number in this article came from a stored run in the public repository; these cost figures were modeled from assumptions and are labeled separately.

Keep that modeled ratio separate from the Irys-reported 39x on the Harvey Legal Agent Benchmark, $1.30 versus $50.90 per task across 1,251 public tasks in 24 practice areas, with the caveats in the same breath: Irys ran the public task set while Harvey's published numbers came from a private holdout that Harvey describes as mirroring the public distribution, and Irys used a different scoring judge because of rate limits, with over 90% agreement checked against the recommended one. It is Irys's own report of a different system on a different benchmark with a different denominator. It is not a substitute for the withdrawn cost story, and it does not rescue it.

The research contract

Twenty-five arithmetic tasks, five planning scenarios, twelve legal tasks, fifteen open-ended prompts: small. The chronology, with the verbs as neutral as we can make them:

• 2026-03. The nested-arithmetic perturbation results are published in the repository README and in our essay.
• 2026-04 to 2026-06. Our internal audits identify the EOS/extraction confound and the token-accounting bug, correct the scorer size, find the verbosity-analysis field-ordering defect (the post-mortem is in CRITICAL_ANALYSIS.md; the earlier verbosity claim reversed once the bug was fixed), and find the unseeded scorer projection. Budget and no-think controls are preregistered and not run. The README is unchanged. The correction's own words: "That process failure is ours."
• 2026-08-02. Igor Rivin submits PR #4 (four measurement bugs, Linux and multi-model portability, 33 tests) and PR #5 (the thinking-mode control, an un-truncated scaling ladder, cross-family evidence, and the diversity study with a numerical-noise control).
• 2026-08-27. The findings are verified against the stored result files; both pull requests are merged; the correction is published; the perturbation work moves under legacy/.
• 2026-08-30. The video production built on the original reading, nearly complete, halts before the final render and before any version is published.

In-house, we found the verbosity-analysis defect and the scorer nondeterminism, and identified the token-count bug without fixing it. Igor Rivin found four additional measurement bugs and ran the missing no-think control. The direct-control omission is what invalidated the headline.

Still open: the sink-free probe, the position-shift ablation at scale, the scorer rerun, the diversity study's remaining three of five tasks on stable hardware, and no-think, temperature-matched, and null-arm controls for every applied result. The ledger is append-only. The perturbation line is archived; we are not asking anyone to continue it.

The correction also changed the rules the repository runs under: no accuracy claim on a generation task without its termination rate beside it, and no capability claim from a benchmark whose baseline finishes fewer than 95% of tasks; the cheapest direct remedy (template flags, budget, prompting) is tested before any scaling or cost comparison; diversity claims need a fixed-input null on the same hardware with the stack's noise floor reported; per-generation significance on task-nested samples requires paired or clustered analysis; any audit that downgrades a public claim updates the README in the same session; and model revisions and chat-template versions are pinned in every result file.

Everything is at github.com/dl1683/Latent-Space-Reasoning. Satellite pieces on the finishing analysis, the voting math, the mechanism tournament, and the economics are planned; none is published yet.

What the measurements say

A benchmark result is not evidence of capability until the evaluation rules out cheaper changes to the conditions that determine whether the model can finish. On our 25 tasks, one greedy thinking-mode baseline against ten perturbation seeds under last-integer grading, two random vectors moved the finish rate by 13.6 points and the severed-trace grade rate by 6.0; in a separate GH200 bf16 experiment, one chat-template flag took every tested Qwen3 size to 96, 100% with zero truncations. So 32% to 51.6% was a completion-under-cap effect, not evidence of better arithmetic reasoning. The mechanism behind the prefix effect is open, and it is a mechanism of termination. None of the applied results has the controls to show whether any task exists where a flag is not the answer and the vectors still are. These perturbation results remain Irys research, not an Irys One product-performance claim. The fabricated statutes are the permanent warning label.

Related: Reinforcement Learning Is the Wrong Bet (the original latent-space reasoning pipeline with trained judges) · Auditability Is the Product · Citations Are Not Audit Trails.

Irys Swarm API: irys.ai/irys-api · irys.ai · Partners · Book a demo · LinkedIn