THE DALTARIS RADAR · October 30, 2026

The week reliability got its own column.

Microsoft's ThinkingBox benchmark ran 507 agent tasks 20 times each, and the model that solves 93.9% at least once solves 13.4% every time. Also this week: memory becomes infrastructure, agent-written code is 91% working and 25% secure, transactions land on the lakehouse, and H100s fall to $2.64/hr.

#0

Featured: headline accuracy is not production reliability

What happened: Microsoft and Hugging Face's ThinkingBox-Training (October 22) turns its own benchmark into a training method, and publishes the month's most uncomfortable eval table along the way. ThinkingBox-Bench runs 507 agent tasks twenty times each, grading terminal database state instead of transcripts. Kimi-K3 solved 93.89% of tasks at least once but only 68 of 507 (13.41%) on all twenty attempts. Claude Opus 5 inverted it: narrower coverage, far more consistency. Claude Opus 5.5 gained 0.66 percentage points of pass@1 over Opus 5 yet passed exactly the same 241 tasks on all twenty attempts. Then the training result: 50 steps of GRPO with executable database-state checks as the reward took Qwen3.8-27B from 38 to 107 tasks at 20/20, cutting the estimated cost per dependable task from $24.36 to $8.00.

Why it matters: The flagship's mechanism in one line: pass@1 is a leaderboard metric; dependability is the shipping metric. Every frontier claim a vendor makes this quarter is a pass@1 claim until they show you the twentieth attempt. The Daltaris read, same as the permission gap last week: the industry measures what is easy to score, and production fails where nobody looks. Note the paper's own honest caveat: benchmark tasks were held out of training, but the shared framework limits how far the transfer claim travels. Visual: twentieth-attempt-2026-10-30-v1.png, scheduled as the Friday X post (draft 11440174).

What to do: Add a 20/20 column to your agent evals: run every task twenty times and report the share that passes every attempt. Reward what production grades.

#1

Memory had its infrastructure week

What happened: Databricks unveiled Agent Memory (beta, built on the MemCurrent architecture) alongside AI Runtime for serverless GPUs (October 22). In Databricks' own testing, not independently verified, agents with memory completed three times as many enterprise tasks on about a third fewer tokens. The same week, Salesforce positioned per-platform memory as a dead end: its Agent Context Engine argues memory must live outside any single platform, with entitlements, masking, and provenance as the enterprise unlock, because scheduled agents in n8n, Make, or Zapier each wake up blind and re-brief from scratch.

Why it matters: Two vendors, one admission: the stateless agent is a cost center. Every morning of re-explaining yesterday is tokens burned just to reach baseline. The Daltaris read: memory is crossing from prompt trick to governed infrastructure, and the governance half (who may read which memory, with what masking, on what expiry) is the data-contracts problem wearing new clothes.

What to do: Ask where your agents' memory lives, who can read it, and what expires it. If the answer is 'in the prompt,' you are paying the amnesia tax without the receipt.

#2

91% of agent-written code works. 25% is also secure.

What happened: The Last Secure Code Benchmark (LSCBench, arXiv October 10, from NUS and Alibaba researchers) grades six frontier coding agents on 450 tasks built from 150 real vulnerabilities. Every task asks only for functionality and never mentions security, which is how most people actually prompt. Six frontier agents pass 73.6% to 91.3% functionally; the best, Claude Fable 5, writes code that is both working and secure 25.1% of the time. Kimi-K3 has the highest functional rate (91.3%) and the lowest secure one (16.4%). Across 2,700 runs, 66% ended with code that works and is still vulnerable. Appending 'while also being secure' lifted scores 16 to 22 points for three models, but most tasks stayed insecure anyway.

Why it matters: The eval-design detail is the story: ask for the feature without mentioning security and you measure the industry's actual prompting practice, honestly. A green test suite told the authors nothing about which agent was safer. This is the flagship's mechanism in a second costume: the metric you score (functional pass) is not the property you ship (secure).

What to do: Replay the attack, not just the test suite. If your agent builds anything that touches a path, a URL, or a shell, the security test is the eval that matters.

#3

Multi-statement transactions land on the lakehouse

What happened: Databricks made multi-statement transactions generally available on Delta Lake and Apache Iceberg (October 16): stored procedures, temp tables, SQL scripting, and streaming tables with atomic multi-table transactions and row-level concurrency instead of locks. Read the vendor framing with care: Databricks claims first-lakehouse status and stronger performance, with no independent benchmarks attached.

Why it matters: One of the last practical reasons teams kept a separate warehouse, ETL consistency and audit-publish workflows that need atomicity, now runs on open table formats. The Daltaris read: the moat keeps moving from the engine to the format to the catalog. Treat the performance claims as marketing until someone reproduces them.

What to do: If atomicity is the reason you still dual-write to a warehouse, re-run that decision against Iceberg tables this quarter.

#4

H100s at $2.64/hr: the buy-vs-rent math just moved again

What happened: Neo-cloud H100 pricing keeps falling as Blackwell supply ramps: $2.64/hr on-demand and $2.20/hr spot on Spheron (live October 23) versus about $6.88/hr on AWS. Secondary-market cards trade at $25,000 to $35,000. The breakeven math: a $35,000 card needs 17,412 rental hours to pay off, about 6 years at 8 hours a day, and resale values keep sliding as B200/B300 become the production standard.

Why it matters: Last year's flagship is this year's mid-tier. Unless utilization is genuinely round-the-clock and predictable over a year or more, depreciation alone kills the purchase case. The Daltaris read for the CFO side of the audience: compute is a flow cost now, and the 6-to-12-month lag in hyperscaler list prices is a tax on whoever never shops the neo-clouds.

What to do: Redo buy-vs-rent with today's rental rates and a realistic utilization profile (40 to 70%, not 100%). For steady state, price reserved commitments before capex.

#5

Signal: the best coding agent scores 26.7% on inference-stack work

What happened: Apple's RSI-Inference Bench (public October 16) hands agents 37 real vLLM and SGLang tasks, day-0 model support, inference optimization, and bug fixes on H100/B200 GPUs, with eight hours each. The best agent passes 26.7% of day-0 model-support tasks.

Why it matters: The first public scoreboard for the exact job infra teams are being asked to automate: letting an agent touch the serving path. The scores say not yet.

What to do: Keep agents off the serving path until a benchmark like this one says otherwise.

#6

Signal: CircleCI's second outage came from deletions it already knew were scheduled

What happened: CircleCI's Postgres disk filled twice in three days (October 6 and 9): the first from maintenance logs outpacing archiving, the second from Temporal's scheduled workflow-history deletion firing on schedule, deletions the team already knew were coming.

Why it matters: The postmortem fixed the first cause and missed the second known trigger. Scheduled work is invisible work: it never pages you until it does.

What to do: Inventory your scheduled deletions and compactions with the same seriousness as your deploys.

#7

Signal: frontier agentic scores at a tenth of the price

What happened: GLM-5.3-Flash (320B total, 18B active) is priced at $0.15/$0.50 per 1M tokens versus $1.40/$4.40 for GLM-5.3, and beats Claude Opus 4.8 on AutomationBench (48.8 vs 41.0), the benchmark that measures the multi-step tool-use loop a production agent actually runs.

Why it matters: A routing decision disguised as a model release: the intelligence-per-dollar curve keeps bending, and the eval that matters is the agentic loop, not the chat benchmark.

What to do: Price your agent's loop per task, not per token. The cheap model that completes the loop beats the dear one that doesn't.

SOURCES

✦ The Daltaris Radar lands weekly. Follow @Daltaris on X to get each issue first.