1. Routing becomes the cost lever: NeMo Switchyard, AI gateways, and the tokens you never needed to send

The Register · Datadog · The New Stack · July 23 – August 13, 2026

Nvidia’s NeMo Switchyard is a software router that sits as a proxy between inference servers and models, directing each request to a model sized for the task rather than sending everything to a frontier model. Nvidia’s headline number is a 74 percent reduction in job completion cost versus running Claude Opus 4.8 for everything, at a cost of roughly six points of accuracy. Joey Conway, senior director of AI software and models, described the design in organisational terms — specialists doing specialist work, with an orchestrator judging complexity. The launch came with Nemotron 3.5-30B-A3B-Lightning, a 30-billion-parameter mixture-of-experts model for low-latency general tasks, alongside task-specific small models such as the 1-billion-parameter Nemotron Parse for PDF work; routing targets can be open-weight or proprietary. The Register notes AT&T reports 80–90 percent savings in some applications using the same smart-routing pattern. The most portable idea in the piece is the metric: completion cost, not per-token price, because a cheaper model that burns ten times the tokens is not actually cheaper.

Datadog’s AI gateway guide is the vendor-neutral architecture for the same idea, and it is the more useful document if you are building rather than buying. A gateway exposes one API across provider models, infrastructure platforms and self-hosted models, and centralises four things worth centralising: routing (task-based assignment moved from hard-coded model names into configuration, plus classifier-based adaptive routers like OpenRouter’s auto-router or LiteLLM’s Auto Router); reliability (fallback and retry configured once, circuit breakers that stop routing to a failing provider before timeouts cascade, load balancing across multi-region deployments — with the caveat that fallback models need evaluating before they are trusted); budget controls (virtual keys with per-team spend ceilings that return 429 before requests reach a provider, TPM/RPM caps that matter most for agent loops that fan out in parallel); and observability (per-provider error and latency rates, retry and fallback frequency, and cost/latency/quality broken out per agent task, plus the ratio of lightweight to frontier calls across the fleet).

The New Stack’s token-optimisation piece is the sprint-sized version. Its diagnosis of why costs multiply after the demo is mundane and familiar: system prompts resent on every request, unbounded conversation history, oversized RAG chunks, unfiltered tool output, and sequential model calls that could have been one. Because consumption scales O(N²) with traffic, demo-era waste compounds. The five fixes it recommends are explicit prompt caching (roughly 90 percent cheaper for the cached prefix), embedding-based semantic caching at a ≥0.92 similarity threshold, rolling summarisation of older turns under a context budget, trimming JSON fields out of tool payloads (70–90 percent overhead reduction), and intent-based model routing (50–80 percent spend reduction). Distillation and fine-tuning can collapse a 1,000-token system prompt to 20. Its closing advice is the operational one: audit tokens across input, state and routing layers, target a >60 percent cache hit ratio on repeated tasks, and track cost-per-resolution per workflow release — the same shift from unit price to completion cost that Nvidia is selling.

Sources: The Register (NeMo Switchyard) · Datadog (AI gateway best practices) · The New Stack (AI pipeline token optimization)

2. Alibaba Cloud’s DualLane: using AI to decide when not to use AI

The Register · August 11, 2026

The cheapest inference is the inference you do not run, and Alibaba Cloud has put that principle into production. DualLane — presented at SIGKDD 2026 in a paper titled “DualLane: Fast and Reliable LLM Agents for Interactive AIOps via Dual-Path Planning” and credited to eleven Alibaba Cloud employees — classifies incoming technical support tickets into high-frequency routine scenarios and low-frequency long-tail ones. Rather than classifying first and then acting, DualLane runs both paths concurrently: the fast path spends only a couple of tokens deciding whether the ticket is routine, and if it is, it terminates the slow path and answers from a template with no LLM reasoning at all. Only genuine long-tail tickets pay the full slow-path cost, which runs up to 3,000 tokens.

The economics are small per ticket and large in aggregate: roughly $0.001 to process 3,000 tokens, against a couple of tokens for the fast-path assessment, at a reported 96.5 percent offline benchmark accuracy and better latency than comparison systems including LLMCompiler and React. Two things make this worth copying beyond the specific numbers. First, the concurrency choice avoids the usual failure mode of routing systems — a classifier that adds latency to every request in order to save cost on some of them. Second, the destination for routine tickets is not a smaller model but no model, which is the option most routing discussions skip past. It is a useful counterweight to the rest of this week’s cost coverage: NeMo Switchyard and AI gateways optimise which model answers, while DualLane asks first whether a model needs to answer at all. For AIOps teams with a heavily-skewed ticket distribution — which is most of them — that question is likely to be worth more than the routing table.

Sources: The Register (Alibaba Cloud DualLane)

3. Open weights set the price floor — and Writer, Capital One and xAI are all standing on it

The New Stack · VentureBeat · August 13–15, 2026

The New Stack’s comparison of Grok 4.6 and Fable 5 Max is the clearest statement of the week’s underlying economics. The two models are close to indistinguishable on capability — Grok 4.6 scores 61 on the Artificial Analysis Intelligence Index against Fable 5 Max’s 56, leads GDPVal-AA at 1,753 Elo to 1,741, and trails slightly on CursorBench v3.2 at 69.9 versus 70.5 percent — and both run on 1.5 trillion parameters, with Grok’s gains coming from post-training rather than scale. The prices are not close: $2 input / $6 output per million tokens against $10 / $50, roughly 80 percent cheaper on input and 88 percent on output, with DeepSeek V4-Pro further down at $0.66/$1.98 off-peak. The article’s thesis is that the frontier is converging on pricing power rather than intelligence, and the mechanism is downloadable open weights: when a customer can self-host something adequate for free, no closed lab sustains a premium indefinitely.

Writer productised exactly that dynamic. Palmyra X6 is post-trained on GLM 5.2, Z.ai’s open mixture-of-experts model, priced at $2 per million input and $8 per million output tokens, and scores 0.87 across nine evaluations — ahead of Claude Opus 4.8 at 0.86, GPT-5.5 at 0.80 and Gemini 3.1 at 0.77. Paired with an upgraded harness, Writer reports its Agent platform running at 52 percent lower cost, 48 percent faster, with a 10 percent quality gain, averaging 26 seconds per task at 82 tokens per second and up to eight hours of unattended operation. The platform work around the model is the operationally interesting part: governance and reporting dashboards, reusable “Playbooks and Skills” workflows, and analytics tracking consumption cost with configurable alert thresholds. CTO Waseem AlShikh framed the customer requirement precisely: enterprises want token consumption to explode because it means adoption is happening, but they need cost to flatten.

Capital One arrives at open weights from a different direction. Milind Naphade, SVP of AI Foundations, argues they are the only route to the depth of customisation the bank needs — not fine-tuning but starting from an open model and customising it “to the point where it’s almost unrecognisable,” including dissecting architecture and tracing model lineage to satisfy regulators. His point about regulated environments is worth sitting with: making AI operate inside a hard set of musts and cannots is much harder than bolting on guardrails, and that constraint, not cost, is what rules out an opaque API. Capital One debuted its first agent tool in January 2026 after more than two years of experimentation. Three organisations, three motives — price pressure, margin, and regulatory control — converging on the same substrate.

Sources: The New Stack (Grok 4.6 vs Fable 5 Max) · VentureBeat (Writer Palmyra X6) · VentureBeat (Capital One open-weight platform)

4. Rubrik’s month with Mythos: the harness matters more than the model, and less automation was the right answer

SiliconANGLE · August 13, 2026

Field reports on autonomous remediation are usually either vendor case studies or cautionary tales, and Rubrik’s is neither. The company got access to Anthropic’s Mythos Preview through Project Glasswing in June 2026 and put a dedicated team of engineers and infosec staff on scanning its own codebase for vulnerabilities — first broad passes across entire repositories with no targeted assumptions, then narrower passes guided by patterns the first round surfaced. Co-founder and CTO Arvind Nithrakashyap wrote it up, and the notable thing is what Rubrik did not do: it did not hire more code reviewers to absorb the output, and it did not push for maximum automation.

Instead the team rebuilt the vulnerability review pipeline around the harness — the software layer wrapping the model that manages tool calls, maintains checkpoints, and injects business and security context. That is where the engineering effort went, and it is the most transferable lesson here: the differentiated work in agentic remediation is not model selection, it is the scaffolding that gives the model the right context and constrains what it can do with it. The second decision was scope. Rubrik deliberately limited machine remediation to specific vulnerability classes where automation is reliable; everything outside those categories produced a structured plan handed to a human engineer who owns it. Nithrakashyap’s framing — in security, trustworthy automation and maximum automation pull in different directions, and they chose the former — is the sentence to bring to your own remediation roadmap review.

On numbers, be appropriately careful: Rubrik reports a significant reduction from raw findings to validated priority issues but publishes no figure of its own. The quantified claims in the coverage belong to Anthropic’s broader Glasswing programme — more than 10,000 high- or critical-severity vulnerabilities found, with a 90.6 percent true-positive rate across assessed flaws — not to Rubrik’s deployment. Read alongside this week’s practice literature, the shape is consistent: DevOps.com’s incident-correlation guide puts automated remediation deliberately last, after unified telemetry, tuned correlation policies and validation against historical incidents; and Chronosphere’s Sam Farid and Nate Heinrich argue the value of building your own AI SRE lies largely in the documentation exercise it forces, since agents need written context about how systems work before root-cause analysis is tractable. Everyone credible is saying the same thing: invest in context and constraint, then automate the narrow band you can defend.

Sources: SiliconANGLE (Rubrik and Anthropic Mythos) · DevOps.com (correlating incidents with AIOps) · The New Stack (build your own AI SRE)

5. Observability grows LLM-native primitives: entity events, cardinality limits, and agent trust platforms

OpenTelemetry · Grafana · July 30 – August 14, 2026

OpenTelemetry’s guide to consuming entity events addresses a gap that has been obvious for years: metrics, logs and traces describe how systems behave, but nothing in the standard set describes what exists. Entity events carry infrastructure inventory and topology as OTLP log records — otel.entity.type, an otel.entity.id map of identifying attributes, descriptive otel.entity.attributes, an entity.relationships array, and an event type of entity_state or entity_delete. As of spec version 1.58.0 in June 2026, relationships are embedded in state events with a type such as depends_on or contains and a target reference. The consumer design the post recommends is the substantial part: append events to a durable log and treat current state as a replay projection rather than mutating records; keep bi-temporal event time and recorded time so you can answer both “how was this configured Tuesday” and “what did we know at 9am”; treat identity keys as strict contracts and keep volatile attributes like IP addresses descriptive so distinct entities never silently merge; and expose the graph through GraphQL for humans and an MCP server so agents can query topology conversationally. Author Matthieu Noirbusson of Sensor Factory also flags the operational grit — clock skew across producers, and coalescing heartbeats without losing structural change. That MCP surface is the quiet bridge to the rest of this week: an agent doing root-cause analysis needs the inventory, not just the signals.

The companion foundational read is the cardinality limits guide, and it belongs in the cost conversation as much as the memory one. The SDK caps unique attribute combinations per metric stream at a deliberately conservative 2,000 by default. What matters is the overflow semantics: excess measurements are not dropped, they are folded into a single data point tagged otel.metric.overflow=true, so totals stay correct while anything that filters or groups by attributes undercounts. The sharp edge is that the entire original attribute combination is discarded, not just the offending high-cardinality attribute — so a perfectly reasonable dimension like success=true/false becomes unreliable for the dashboards, SLOs and alerts built on it the moment its combination overflows. The recommended posture is to fix instrumentation or drop attributes in views before raising the limit, to remember that an SDK-side cap does not bound backend series (a thousand pods at 2,000 combinations each still produces a very large number), and to audit overflow in production with a Prometheus query over otel_metric_overflow. The goal is not zero overflow but rare, visible and actionable overflow.

Grafana’s piece covers the agent-specific layer, drawn from building Grafana Assistant from hackathon project to GA in a little over six months. A “trust platform” here means the apparatus for gaining confidence in a non-deterministic system: live traffic monitoring with conversation transcripts and tool calls alongside conventional engineering metrics (latency, cost, token usage, error rates); quality evaluation combining deterministic checks with LLM-judge evaluators scoring transcripts pass/fail or on a Likert scale, with PII and toxicity detectors out of the box; test suites built by converting failing conversations into regression tests; experiment tracking for offline evaluation wired into CI/CD; and version control over agent configuration with rollback. The components are the agento11y SDK, the gcx CLI, Grafana Cloud Metrics as the store for evaluation scores, and Grafana Alerting on top. Read with DevOps.com’s piece — where Honeycomb’s Liz Fong-Jones separates telemetry, the raw data, from observability, the property of combining it with human knowledge to understand the system — the throughline is that instrumentation has to keep pace with autonomy, and evaluation is now part of instrumentation.

Sources: OpenTelemetry (consuming entity events) · OpenTelemetry (cardinality limits) · Grafana (agent trust platform) · DevOps.com (AI raises the stakes for observability engineering)

6. The serving side: a 2.4T open model on one rack, attention redesigned for long context, and $240M of inference capacity

NVIDIA · Reuters · July 31 – August 12, 2026

If open weights are setting the price floor, someone has to make serving them economic, and Nvidia’s writeup on Qwen3.8-2.4T-A95B is a concrete look at what that takes. Alibaba’s largest open-weight model runs 2.4 trillion total parameters with 95 billion activated per token, using a fine-grained mixture-of-experts design that combines full and linear attention, a one-million-token context window and up to 128K output. Serving it on a GB300 NVL72 — 72 Blackwell Ultra GPUs in a single rack with 130 TB/s of NVLink bandwidth — matters specifically because MoE expert routing generates heavy all-to-all traffic that a rack-scale interconnect absorbs without becoming the bottleneck. Day-one numbers in FP8 were over 4K tokens per second per GPU and over 350 tokens per second per user, with more expected from NVFP4. The feature most relevant to cost control is configurable reasoning: low, high and xhigh settings let a caller trade compute for reasoning quality per request — the same routing logic as NeMo Switchyard, but inside a single model rather than across a fleet. Serving paths include SGLang, vLLM and Nvidia Dynamo, with NIM containers and NeMo AutoModel for fine-tuning from Hugging Face checkpoints.

Nvidia’s foundational piece on co-designing model attention for fast, interactive long-context inference is the deeper version of the same argument: long-context serving economics are set by attention design, and treating the model architecture and the serving stack as one problem is how interactive latency at large context becomes affordable rather than aspirational. Together these two pieces explain why the configurable-reasoning knob exists at all — the compute cost of a request is a design parameter now, not a fixed property of the model you picked.

The capital behind all of this kept moving. Reuters reported IBM and Together AI signing a $240 million deal for an Nvidia-powered AI inference cluster — notable mainly for what it is: an inference deal, not a training deal. The spend that used to chase training runs is increasingly buying serving capacity, which is the demand-side mirror of everything else in this issue. Routing, caching, two-lane agents and cheaper open weights are all attempts to get more work out of a fixed inference budget; deals like this one are what happens when the budget grows instead. Both trends point at the same organisational conclusion: cost-per-resolution on inference is becoming a first-class operating metric, tracked and owned the way latency and error budgets already are.

Sources: NVIDIA (serving Qwen3.8-2.4T on GB300 NVL72) · NVIDIA (co-designing attention for long-context inference) · Reuters (IBM and Together AI inference cluster)