This week at a glance
The clearest signal this week was a step back on autonomy. In VB Intelligence’s August survey, 56% of organisations deploying agents allowed, or were building toward, production changes without human review, down from 75% in July. The share expecting to keep human review for the foreseeable future rose from 20% to 42%. 61% said an agent had caused at least one customer-facing failure after passing internal tests, and only 9% trust automated evaluation today. A Riverbed survey reported by ITPro points the same way: 90% want autonomous IT operations within two years, but 77% will not let AI make operational decisions without human approval.
Vendors are shipping to match that mood. Palo Alto Networks’ Cortex XCOR runs its investigation before the on-call engineer logs in, but still pages them. New Relic used its New Relic Now event to announce AI Evaluation, an LLM-as-a-judge service tied to traces, in public preview from November. Gremlin made Foresight AI generally available, and OllyGarden added a check for missing instrumentation. Under the AI stack, Clockwork raised $31 million for GPU-cluster fault tolerance, and CoreWeave filled in details of its Forge platform. Lava Security found more than 2,000 GPU servers exposing Nvidia’s DCGM Exporter without authentication, with a CVSS 8.2 crash bug fixed in version 4.8.2. On spend, VB Intelligence found Google named as the primary AI platform by 48% of production users, and FICO found only 8.2% of organisations on a single AI deployment platform. Firebase’s iOS crash postmortem and drone strikes on Yandex data centres rounded out a week of outage lessons.
On our watch list
- VB Intelligence’s next agentic reliability wave. The drop from 75% to 56% came between July and August, and VB says several shifts were not statistically significant. A third reading in the same direction would confirm the retreat from unreviewed production changes. A rebound would mark August as noise.
- New Relic AI Evaluation in public preview in November. Watch whether the LLM-as-a-judge scores for hallucination, prompt injection and data leakage come with published accuracy figures, and what the preview costs. New Relic has given no pricing for AI Evaluation, Ground Truth CLI or Infrastructure 360.
- How far Cortex XCOR lets agents act without paging. Martin Mao says XCOR defaults to human-in-the-loop and that leaders can widen permissions over time. Watch for the first customer who lets it close incidents unattended, and for any independent check of the 75% root-cause figure, which still has no published method.
- DCGM Exporter patch uptake. Lava Security counted more than 2,000 exposed servers and 12,000 GPUs. A repeat scan after the 4.8.2 fix will show whether operators are closing the profiling endpoints or only upgrading. Watch also for exploitation of CVE-2026-47483 against shared AI clusters.
- Cloudflare’s December 1 pricing switch. Traces and Workers Tracing move from per-span billing to $0.25 per GB ingested and $0.10 per GB-month stored. Watch for one-year retention, which Cloudflare lists as “coming soon,” and for Traces leaving beta.
- Whether Clockwork’s customer list grows past LinkedIn and Together AI. Clockwork sells fault tolerance as a “goodput multiplier.” Watch for a neocloud or hyperscaler announcing it as a default layer, and for published goodput numbers from a named customer.
- Yandex’s recovery plan. Yandex Cloud said its Sasovo availability zone could not be restored “in the near future,” and further sites were hit later in the week. Watch whether it writes the facility off, as AWS did with its Bahrain site earlier this year, and how customers without multi-zone designs recovered.
- Firebase’s follow-through. Google’s postmortem says “backend data anomalies should not cause app-side crashes,” but the status page stayed green throughout. Watch for an SDK release that validates config payloads and for a change to how Firebase incidents are reported.
- Evaluation platform purchases. 62% of VB’s respondents plan to adopt, add or replace an evaluation platform within 12 months, 35% within three. Watch for funding rounds and acquisitions among Confident AI, Braintrust and LangSmith as observability vendors move into the same space.
- Accelerator utilisation as a reported metric. 80% of VB’s production respondents who gave a figure run accelerators at 50% or less. Watch whether neoclouds and AI platform vendors start publishing utilisation or goodput alongside price.
This week’s topic map. Agent autonomy and AI SRE sit together at the top: VB Intelligence and Riverbed on trust, Cortex XCOR, Gremlin and Netflix on tooling. New Relic and Cloudflare connect observability for AI to OpenTelemetry on the left. The infrastructure cluster below links Clockwork and CoreWeave to Nvidia’s DCGM Exporter and its CVE. Platform spend and the Firebase and Yandex outages sit to the right.
View interactive topic map →
Article index
25 articles, grouped by sub-theme. Nineteen are from this week’s coverage window (October 4 to October 11). Six are longer foundational reads on the beat. Sponsored coverage, vendor blogs and press releases are labelled within each group.
Agent autonomy and human review
Two surveys and one practitioner essay on how much control teams are handing to agents. Both surveys use self-selected or vendor samples. VB Intelligence surveys its own readers and panel. The ITPro story reports Riverbed’s research, and Riverbed sells into this market. The tabletop essay is sponsored by Webflow, its author’s employer.
AI SRE and reliability engineering
Three launches and four practitioner reads. The Cortex XCOR figures are still Palo Alto’s own, with no method. The Gremlin item reads as a press release. Netflix’s talk and the three engineering posts describe systems their authors built or ran.
| Article |
Source |
Published |
| 4. XCOR launches to trace outages in minutes. It still pages engineers. |
The New Stack |
Oct 5, 2026 |
| 5. Gremlin Launches Foresight AI to Proactively Fix Reliability Risks (press release) |
DevOps.com |
Oct 7, 2026 |
| 6. OllyGarden Extends AI Agent to Identify Instrumentation Gaps |
DevOps.com |
Oct 9, 2026 |
| 7. Ontology-Driven Observability: Building the E2E Knowledge Graph at Netflix Scale (foundational, conference talk) |
InfoQ |
Oct 9, 2026 |
| 8. Attaching Evidence to Alerts: an Enrichment Sidecar for Alertmanager (foundational) |
DevOps.com |
Oct 8, 2026 |
| 9. Kubelet watches inodes. Just not until it’s an emergency. (foundational) |
CNCF Blog |
Oct 5, 2026 |
| 10. Availability Should Follow the Workload, Not the Infrastructure (foundational, contributed) |
DevOps.com |
Oct 6, 2026 |
Observability for AI, built on OpenTelemetry
New Relic’s event announcements, Cloudflare’s tracing beta and two OpenTelemetry project posts. The New Relic blog is the vendor’s own product marketing. IT Brief UK is the neutral write-up of the same event. The OTel-native post is co-written by SigNoz and New Relic engineers.
Keeping AI infrastructure running
A funding round, three interviews from CoreWeave’s Fully Connected event and one exposure study. The three Fully Connected pieces are theCUBE coverage paid for by CoreWeave. The Clockwork round and the Lava Security research are reported independently.
Platforms, spend and sprawl
Where production AI runs, how many platforms it runs on, and what happens when billing rules are unclear. The VentureBeat figures come from a self-selected reader sample of 68 production users. The CIO Dive story reports FICO’s survey of more than 1,000 executives.
Outages and physical risk
A postmortem on how a config change crashed iOS apps worldwide, and a cloud availability zone lost to a drone strike. The Firebase incident itself happened on September 28–29; the newsletter’s free edition ran this week.
Detailed write-ups
1. Fewer teams let agents change production unreviewed
VentureBeat / ITPro / The New Stack · October 6–8, 2026
VB Intelligence compared its July and August survey waves. Among organisations deploying autonomous agents, the share that allow or are building toward production changes without human review fell from 75% to 56%. Among final AI purchasing decision-makers, it fell from 88% to 61%. The share expecting to keep human review for the foreseeable future rose from 20% to 42%. The failures explain why. 61% reported at least one customer-facing failure in the past 12 months from an agent that had passed internal tests, and 22% reported more than one. Only 9% say automated evaluation is trustworthy today.
Monitoring has not caught up. Transaction trace logging is the main production check for 36%, and real-time automated quality checks for 29%. Of the 38 organisations already letting agents make unreviewed changes, only about a quarter rely mainly on live output-quality checks. 62% plan to adopt, add or replace an evaluation platform within 12 months. The sample is 140 self-selected VentureBeat readers and panel members, and VB notes that several of the shifts are not statistically significant.
A Riverbed survey, reported by ITPro, shows the same gap between intent and practice. 90% of IT leaders want agentic AI running IT operations within two years, but 77% will not let AI make operational decisions without human approval. One in five run automated IT operations today, and only 17% say they have unified visibility across their environment. Riverbed CMO Jim Gargan: “The journey to autonomous IT is well underway.” In a Webflow-sponsored essay, Webflow’s Andy Gombar argues that incident tabletops should now test the AI as well as the responders, with injects for confidently wrong summaries, stale playbooks and an AI that is simply unavailable.
Sources: VentureBeat (trust in agent production changes falls to 56%) — https://venturebeat.com/orchestration/the-share-of-enterprises-trusting-ai-agents-to-make-production-changes-on-their-own-has-fallen-from-75-to-56-vb-intelligence-finds · ITPro (enterprises aren’t quite sold on autonomous IT operations) — https://www.itpro.com/technology/artificial-intelligence/enterprises-arent-quite-sold-on-autonomous-it-operations · The New Stack (incident tabletops, sponsored by Webflow) — https://thenewstack.io/ai-incident-response-tabletops/
2. AI SRE tools ship with a human still on call
The New Stack / DevOps.com / InfoQ · October 5–9, 2026
A week after Palo Alto Networks launched Cortex XCOR, The New Stack’s coverage explains how it handles the engineer. Its AI SRE agents start when an alert fires and usually finish investigating before the on-call engineer can log in, but XCOR still pages that engineer. Martin Mao, now SVP and GM of observability at Palo Alto, says the human-in-the-loop default is deliberate and that leaders can widen the agents’ permissions over time: “We don’t want to keep waking engineers up in the middle of the night.” The headline figures are unchanged, and still Palo Alto’s own: a 75% root-cause success rate and an average investigation under three minutes, with no published method.
Gremlin made Foresight AI generally available on October 7 after a beta. It looks for reliability risks before they cause incidents, proposes fixes as configuration patches or infrastructure-as-code changes, then re-runs the original test to confirm each fix. It draws on what Gremlin calls its Failure Atlas, more than a decade of data on how systems fail. No price was given. CEO Kolton Andrus: “AI-driven development means shipping code at 10X velocity; it also means 10x the opportunity for bugs, risks, and failures.” OllyGarden raised a further $4 million and added Minimum Viable Instrumentation to its Rose AI agent, which flags services with no instrumentation and recommends the OpenTelemetry data to add.
For what it takes to build an AI SRE on real data, Netflix’s QCon talk is the week’s best long read. Netflix handles more than 38 million logging events a second. It is building an ontology of its services and turning Slack incident discussions into graph data, which its AutoSRE agent queries alongside metrics, logs, traces and topology. Renzo Sanchez-Silva: “The ontology is a contract between chaos and understanding.”
Sources: The New Stack (XCOR still pages engineers) — https://thenewstack.io/cortex-xcor-ai-observability/ · DevOps.com (Gremlin Foresight AI, press release) — https://devops.com/gremlin-launches-foresight-ai-to-proactively-fix-reliability-risks/ · DevOps.com (OllyGarden instrumentation gaps) — https://devops.com/ollygarden-extends-ai-agent-to-identify-instrumentation-gaps/ · InfoQ (Netflix ontology-driven observability, talk) — https://www.infoq.com/presentations/netflix-observability-aiops-ontology-scale/
3. New Relic puts an LLM judge next to the trace
IT Brief UK / New Relic · October 6–7, 2026
At New Relic Now, New Relic announced AI Evaluation, part of its AI Observability platform. It runs an asynchronous LLM-as-a-judge service that scores AI application behaviour for hallucinations, prompt injection and data leakage, and links each score to the distributed trace. It enters public preview in November. The company also announced Ground Truth CLI, a command-line tool that developers and AI agents can use to investigate incidents and run recovery checks, and Infrastructure 360, a single view of cloud resources, dependencies and configuration changes. No prices or availability dates were given for those two. Chief product officer Brian Emerson: “Generative AI has redefined application health.”
New Relic’s own blog describes the OpenTelemetry side. Its AI Observability for OpenTelemetry detects AI workloads by scanning incoming spans for OpenTelemetry’s gen_ai.* attributes, which is also how it finds unmanaged “shadow” AI services. The prompt and the response sit on one span, and the data is stored without translation at ingest. The post lists model inventories, cost tracking and model comparison, but does not say whether the feature is generally available.
Sources: IT Brief UK (New Relic launches AI Evaluation) — https://itbrief.co.uk/story/new-relic-launches-ai-evaluation-for-production-safety · New Relic (AI Observability for OpenTelemetry, vendor blog) — https://newrelic.com/blog/news/introducing-ai-observability-for-opentelemetry
4. Cloudflare traces the proxy, and OpenTelemetry closes the log gap
InfoQ / OpenTelemetry Blog · October 6–10, 2026
Cloudflare Traces is in open beta. It turns security rules, transformations, cache decisions, routing, Worker execution and origin handling into OpenTelemetry spans on a single request timeline, with no instrumentation to write. It accepts W3C traceparent headers, so Cloudflare’s spans join upstream traces, and it exports over OTLP to any compatible backend. Trace Rules allow targeted sampling, for example a 1% baseline with overrides by path, header, IP or geography. InfoQ’s example shows why this matters: a 539 ms cache miss in which 527 ms was spent waiting on the origin. From December 1, 2026, Traces and Workers Tracing move from the per-span billing announced earlier to volume pricing: 50 GB of ingestion and 10 GB-month of storage included on paid plans, then $0.25 per GB and $0.10 per GB-month.
On the OpenTelemetry side, OBI (OpenTelemetry eBPF Instrumentation) can now add trace_id and span_id to log lines that services already write to stdout or stderr, with no SDK or code change. It handles JSON, NDJSON and plain text. The caveats matter. It needs CAP_SYS_ADMIN and Linux 6.0 or later, Node.js output can miss context under backpressure, Python needs PYTHONUNBUFFERED=1, and Java virtual threads are not yet covered. A companion post from SigNoz and New Relic engineers sets out what “OTel-native” should mean for any product: OTLP export for every signal, standard environment variables and semantic conventions, and push rather than polling. Their summary: “default to push, stay vendor-neutral, and document the contract.”
Sources: InfoQ (Cloudflare Traces open beta) — https://www.infoq.com/news/2026/10/cloudflare-traces-open-beta/ · OpenTelemetry Blog (zero-code trace-log correlation with OBI) — https://opentelemetry.io/blog/2026/obi-trace-log-correlation/ · OpenTelemetry Blog (OTel-native by design) — https://opentelemetry.io/blog/2026/otel-native-by-design/
5. Money for GPU fault tolerance, and CoreWeave fills in Forge
SiliconANGLE · October 5–8, 2026
Clockwork raised $31 million, led by Seligman Ventures, Wing Ventures and Premji Invest, bringing its total to $73 million. It sells a software layer that sits between GPUs and AI workloads, with nanosecond-accurate telemetry for spotting failures early. LinkPass handles network failover, TorchPass moves work off failing GPUs, and the new TorchSnap takes multinode snapshots of distributed inference jobs without code changes so they can restart where they stopped. LinkedIn uses LinkPass to cut thousands of GPU-hours of downtime a month, and Together AI offers TorchPass as a service. The article recalls that Meta’s Llama 3 training run on 16,384 GPUs hit a hardware problem about every three hours. CEO Suresh Vasudevan: “Fault tolerance is a goodput multiplier.”
Three interviews from CoreWeave’s Fully Connected event, all in theCUBE coverage paid for by CoreWeave, add detail. CoreWeave Forge, first announced at the event last week, is free to start with paid tiers. Its RL Rollouts preview, built on Nvidia Dynamo, loads new checkpoints into live deployments and reloaded models 15x faster than a baseline in CoreWeave’s testing. Dell and CoreWeave said they now test racks at the factory with CoreWeave’s Racky manager, which combines power, cooling and environmental telemetry. Dell’s Sarat Krishnan called it “pushing your risk to the left.” Vast Data’s CTO Alon Horev argued that long agent sessions should move their memory out of GPUs into tiered storage.
Sources: SiliconANGLE (Clockwork raises $31M) — https://siliconangle.com/2026/10/05/clockwork-io-bags-31m-in-funding-to-keep-ai-inference-and-training-workloads-running-like-clockwork/ · SiliconANGLE (CoreWeave Forge, CoreWeave-sponsored theCUBE coverage) — https://siliconangle.com/2026/10/08/ai-inference-gets-full-stack-coreweave-launches-forge-fullyconnected/ · SiliconANGLE (Dell and CoreWeave on reliability, CoreWeave-sponsored theCUBE coverage) — https://siliconangle.com/2026/10/05/dell-coreweave-build-ai-infrastructure-reliability-factory-fullyconnected/ · SiliconANGLE (Vast Data on agent memory, CoreWeave-sponsored theCUBE coverage) — https://siliconangle.com/2026/10/06/ai-agent-memory-belongs-storage-says-vast-data-cto-fullyconnected/
6. GPU telemetry left open on the internet
CSO Online · October 9, 2026
Researchers at Lava Security found more than 2,000 GPU servers exposing Nvidia’s DCGM Exporter, the agent that feeds GPU metrics to Prometheus, to the internet with no authentication. Across four scans they counted more than 12,000 unique GPUs, which they value at about $100 million of hardware, including B300, H200 and H100 systems. The endpoints reveal GPU utilisation, memory, power use and error events. Lava’s Michael Katchinskiy: “Anyone who could reach these endpoints could see what hardware organizations were running.”
About a quarter of the exposed servers also exposed the profiling endpoints behind CVE-2026-47483 (CVSS 8.2). An unauthenticated request to /debug/pprof/ can exhaust resources and crash the exporter, which can affect AI workloads on the same host. Nvidia fixed it in DCGM Exporter 4.8.2. The recommended steps are to upgrade, bind exporters to loopback or private interfaces, keep the exporter and Prometheus off the public internet, and confirm that pprof is disabled, since unpatched versions may enable it by default.
Sources: CSO Online (exposed Nvidia GPU monitors) — https://www.csoonline.com/article/4233081/exposed-nvidia-gpu-monitors-can-reveal-ai-infrastructure-secrets.html
7. A config flag crashed iOS apps, and a drone took out an availability zone
The Pragmatic Engineer / The Register · October 8–9, 2026
The Pragmatic Engineer’s Gergely Orosz took apart the Firebase outage of September 28–29. iOS apps using Google Analytics for Firebase crashed on launch; Android was not affected. According to Google’s postmortem, a routine cleanup of a stale legacy configuration flag produced a malformed payload that rolled out globally, and the iOS SDK did not check that a flag’s name was not nil. Google took about 70 minutes to acknowledge the problem and about 1.5 hours to start a rollback. The fix took about 2 hours 11 minutes to roll out, and cached bad responses kept some apps crashing for up to four hours more. The status page stayed green, and the postmortem arrived four days later. Its key line: “Backend data anomalies should not cause app-side crashes.”
The Register reported that Yandex Cloud’s ru-central1-b availability zone went down after a drone attack caused a fire at its Sasovo data centre, about 300 km southeast of Moscow. Yandex said operations there had “completely halted,” pointed customers to alternative restoration plans, and said service could not be restored in the near future. The Register noted that a second Yandex data centre was struck soon after, and that AWS wrote off its Bahrain facility after a drone attack earlier this year. Physical loss of a zone is no longer a hypothetical scenario for disaster-recovery planning.
Sources: The Pragmatic Engineer (Firebase’s global outage) — https://newsletter.pragmaticengineer.com/p/the-pulse-firebases-global-outage-213 · The Register (drone attack takes out Russian datacenter) — https://www.theregister.com/off-prem/2026/10/09/ukrainian-drone-attack-takes-out-russian-datacenter/5302137
8. Google leads production AI, and nobody runs just one platform
VentureBeat / CIO Dive / The Register · October 5–9, 2026
Among 68 VentureBeat respondents running AI in production, 48% named a Google platform as their primary one (Gemini 28%, Google Cloud 20%), against 22% for OpenAI and 13% for Azure. Use is spread wider: 65% use a Google platform and 56% use OpenAI. The operations numbers are more telling. 80% of those who gave a figure run their accelerators at 50% utilisation or less, uptime is the top success metric at 56%, and only 48% track return on AI infrastructure rigorously. FICO’s survey of more than 1,000 executives, reported by CIO Dive, found only 8.2% of organisations on a single shared AI deployment platform. FICO’s Scott Zoldi: “Once that shared standard is in place, that’s the flywheel.”
The Register showed what sprawl costs at small scale. Norwegian startup Vegalabs ran Claude through Microsoft’s platform expecting its $25,060 in Azure startup credits to cover it. Anthropic models bought through Azure Marketplace are excluded from those credits, so the charges went to its card, reaching $17,600, while $21,168 of credits expired unused. Vegalabs admits it set no budget alert. Microsoft told it that Azure Support cannot waive Marketplace charges, and Anthropic sent it back to Microsoft.
Sources: VentureBeat (48% name a Google platform as primary) — https://venturebeat.com/resources/ai-infrastructure-and-compute-48-of-organizations-running-ai-in-production-name-a-google-platform-as-their-primary-ai-platform-more-than-twice-openais-share · CIO Dive (AI platform sprawl undermines ROI) — https://www.ciodive.com/news/single-ai-deployment-platforms-rare-app-sprawl-grows/832443/ · The Register (invoice tennis over a $17,600 Claude bill) — https://www.theregister.com/paas-and-iaas/2026/10/09/microsoft-and-anthropic-play-invoice-tennis-with-startups-17600-claude-bill/5302041
Calls to action
- Find and lock down your DCGM Exporters. Upgrade to DCGM Exporter 4.8.2 or later, bind it to loopback or a private interface, keep it and Prometheus off the public internet, and confirm
/debug/pprof/ is not reachable.
- Write down which production changes agents may make unreviewed. VB’s respondents are pulling back after agents that passed tests failed in front of customers. List the change types allowed without review, and add a live output-quality check for each, not just trace logging.
- Add AI failure injects to your next incident tabletop. Include a confidently wrong AI summary, a stale playbook retrieval and an AI that is down, and check whether responders can still triage by hand.
- Set budget alerts on every marketplace model deployment. Check which models your cloud credits actually cover before deploying them. Vegalabs lost $21,168 in credits while separate Marketplace charges reached $17,600.
- Validate config payloads on the client. Firebase crashed iOS apps on a malformed flag. Make SDKs and apps reject or ignore bad remote config rather than crash, and alert on crash spikes yourself instead of waiting for a vendor status page.
- Re-cost Cloudflare tracing before December 1. Estimate your trace volume in GB against the $0.25 per GB and $0.10 per GB-month rates, and set Trace Rules sampling before the new pricing starts.
- Check that your multi-zone design survives losing a zone. Yandex Cloud lost an availability zone to a fire with no near-term restore. Test failover for your most important AI and data services with one zone removed.
|