{"id":5868,"date":"2026-09-13T13:43:34","date_gmt":"2026-09-13T18:43:34","guid":{"rendered":"https:\/\/www.cybersecurityinstitute.com\/blog\/?p=5868"},"modified":"2026-09-13T13:43:34","modified_gmt":"2026-09-13T18:43:34","slug":"ai-ops-weekly-september-13-2026","status":"publish","type":"post","link":"https:\/\/www.cybersecurityinstitute.com\/blog\/?p=5868","title":{"rendered":"AI Ops Weekly &mdash; September 13, 2026"},"content":{"rendered":"<style>\n.single .entry-title,\n.single .entry-header .entry-title,\n.single .post-title,\n.single header.entry-header h1,\n.single h1.entry-title,\n.single .page-title,\n.post-template-default h1.entry-title,\n.post-template-default .entry-header,\narticle .entry-header,\narticle .entry-title { display: none !important; }\n.single .entry-header { margin: 0 !important; padding: 0 !important; }\n.single .entry-content { margin-top: 0 !important; padding-top: 0 !important; }\n<\/style>\n<table role=\"presentation\" class=\"wrapper\" cellpadding=\"0\" cellspacing=\"0\" border=\"0\" width=\"100%\">\n<tr>\n<td align=\"center\">\n<table role=\"presentation\" class=\"container\" cellpadding=\"0\" cellspacing=\"0\" border=\"0\" width=\"680\">\n<p>        <!-- Banner --><\/p>\n<tr>\n<td class=\"banner\" style=\"background-color:#0e7490;background:linear-gradient(135deg,#0e7490 0%,#0891b2 100%);padding:36px 32px;color:#ffffff;\">\n<p class=\"date\" style=\"color:#ffffff !important;\">September 13, 2026 &middot; Weekly Edition<\/p>\n<h1 style=\"color:#ffffff !important;\">AI Ops<\/h1>\n<p class=\"tagline\" style=\"color:#ffffff !important;\">A thin week for evidence and a heavy one for marketing. Five items in this issue carry numbers anyone measured: AWS cut P50 time-to-first-token 71&ndash;77% with prefix-aware routing on SageMaker (its own benchmark, on its own hardware); Netflix rebuilt Conductor to 420 million workflow executions a month; two Adobe engineers cut a tenant&rsquo;s GPU metric cardinality from 10,000+ series to about 300; a Salesforce engineer described a pipeline that ran green for 11 days while its numbers were 40% wrong; and OpenAI&rsquo;s own research-acceleration data shows humans intervening on more than half of <em>successful<\/em> long-running agent tasks. Everything else is a vendor blog, a contributed column or a commissioned survey &mdash; including two sponsored posts, three separate Dynatrace items and a Honeycomb sampling processor that is not actually in OpenTelemetry yet. Twenty-one stories, labelled.<\/p>\n<\/td>\n<\/tr>\n<p>        <!-- At a glance --><\/p>\n<tr>\n<td class=\"content\">\n<h2>This week at a glance<\/h2>\n<p>The honest summary of this week is that the AI-operations beat produced a great deal of argument and very little measurement, and the two are easy to confuse when both arrive as blog posts. <strong>Five items<\/strong> carry numbers that someone actually measured. <strong>AWS<\/strong> shipped <code>PREFIX_AWARE<\/code> routing to general availability on <strong>SageMaker Inference<\/strong> and published a benchmark: on 8,000-token shared prefixes, P50 time-to-first-token fell <strong>71&ndash;77%<\/strong> and KV cache hit rate went from roughly <strong>25% to 82%<\/strong> &mdash; but on short contexts throughput improved only <strong>1.7&ndash;2.0%<\/strong>, and every figure is AWS&rsquo;s own, run on seven <code>ml.p5.48xlarge<\/code> instances with no third-party verification. <strong>Netflix<\/strong> took Conductor from a memory-bound ceiling to <strong>420 million workflow executions a month<\/strong> across <strong>200,000 workflow definitions<\/strong>, with maximum workflow size up roughly <strong>10x<\/strong> to 30,000 tasks and failed lock acquisitions down from about <strong>2,700 per interval to essentially zero<\/strong>. Two <strong>Adobe<\/strong> engineers writing for the CNCF showed a per-tenant GPU metrics design that cut a tenant view from <strong>10,000+ time series to about 300<\/strong> &mdash; a <strong>~97%<\/strong> cardinality reduction &mdash; after finding a GPU sitting idle at <strong>0% utilisation for 11 straight days<\/strong>. A Salesforce engineer, writing about incidents at a previous employer, described a pipeline that passed every job-level check for <strong>11 days<\/strong> while producing audience counts that were <strong>~40% wrong<\/strong>. And <strong>OpenAI<\/strong>&rsquo;s own August publication reports <strong>3.1 agent-workdays per human workday<\/strong>, inference cost per researcher ranging from a <strong>$600\/day median to $7,000\/day at the 90th percentile<\/strong>, and human intervention required on more than <strong>50% of successful<\/strong> four-to-eight-hour tasks.<\/p>\n<p>The rest of the issue needs labels more than it needs summarising. Two posts on The New Stack are paid placements: <strong>&ldquo;It passed CI. It passed your evals.&rdquo; is sponsored by Dynatrace and written by a Dynatrace employee<\/strong>, and its trace timings are an illustrative worked example rather than production data; <strong>&ldquo;Observability has a data problem&rdquo; is sponsored by Bronto<\/strong>, and its headline claims &mdash; 100x more data retained, 12 months of full-fidelity retention, sub-second search across petabytes &mdash; are Bronto product claims. Two APMdigest pieces are vendor columns with no disclosure on the page: <strong>&ldquo;The SIEM Isn&rsquo;t the Problem&rdquo; is by Bindplane&rsquo;s CEO and co-founder, Mike Kelly<\/strong>, whose &ldquo;40%+ telemetry volume reduction&rdquo; is a claim for Bindplane&rsquo;s own product category; <strong>&ldquo;Your AI Pipeline May Have a Document Observability Problem&rdquo; is by Jeff Sidell, CTO of the OCR vendor 8am<\/strong>, and contains no numbers whatsoever. <strong>Dynatrace appears three times<\/strong> &mdash; the sponsored post, a platform-engineering blog and a Cursor plugin announcement &mdash; and the first two draw on the <em>same<\/em> Dynatrace-commissioned 2026 State of SRE and Platform Engineering survey, so they are one datapoint, not two. Honeycomb&rsquo;s <em>&ldquo;Wide Events vs. Three Pillars: AI Observability Costs&rdquo;<\/em> contains no cost figures at all despite its title, and its adaptive tail-sampling processor is <strong>&ldquo;working toward alpha&rdquo; upstream &mdash; not merged into OpenTelemetry and not GA<\/strong>; today it ships only in Honeycomb&rsquo;s own Collector distribution. <strong>Red Hat AI 3.5<\/strong> is a genuinely interesting multi-tenancy release with <strong>no performance numbers of any kind<\/strong>. And the two O&rsquo;Reilly nonhuman-identity essays are a two-part series by the same authors published two days apart, not two independent reads on the problem. Read accordingly &mdash; the arguments are often good, but an argument is not a measurement.<\/p>\n<div class=\"watchlist\">\n<h2>On our watch list<\/h2>\n<ul>\n<li><strong>Whether Honeycomb&rsquo;s adaptive tail-sampling processor actually lands upstream.<\/strong> It is described as &ldquo;working toward alpha&rdquo; in the OpenTelemetry Collector and currently ships only in Honeycomb&rsquo;s own distribution, derived from Refinery. Watch the contrib repository rather than the blog: an upstream merge would make dynamic, rate-targeting tail sampling a vendor-neutral capability, and until then it is a reason to pick a distribution.<\/li>\n<li><strong>Whether Red Hat ships storage offloading out of developer preview.<\/strong> Everything else in AI 3.5&rsquo;s multi-tenancy story &mdash; fair-share scheduling, admission control, tenant isolation, CPU offloading, per-user token metering &mdash; is generally available. Storage offloading for GPU memory is not, and it is the piece that decides whether you can oversubscribe accelerators without a latency cliff.<\/li>\n<li><strong>Whether OpenAI&rsquo;s March 2028 automated-researcher target survives its own supervision bottleneck.<\/strong> The same dataset that shows 3.1 agent-workdays per human workday also shows humans intervening on more than half of successful long-running tasks. Watch whether the next publication reports intervention rate falling, because throughput that scales while review capacity does not is the constraint that target has to clear.<\/li>\n<li><strong>Whether anyone publishes agent-observability cost data that is not vendor-supplied.<\/strong> Every cost figure attached to AI observability this week came from a company selling the remedy. A single engineering organisation publishing its own $\/GB, span volume and retention numbers for an agent workload would be worth more than the rest of this category combined.<\/li>\n<li><strong>Whether AWS&rsquo;s prefix-routing numbers reproduce outside AWS&rsquo;s benchmark harness.<\/strong> The long-context result is large and the short-context result is nearly nil, which is exactly the shape you would expect &mdash; and exactly why an independent run on a different model and traffic mix matters. Watch for anyone publishing their own before\/after KV cache hit rate after switching the routing strategy.<\/li>\n<li><strong>Whether per-tenant metric isolation becomes a platform default.<\/strong> Adobe&rsquo;s <code>MetricAccess<\/code> CRD is still <code>v1alpha1<\/code> despite production use, and the pattern &mdash; RBAC-scoped, label-enforced, self-service metric reads &mdash; is the thing most multi-tenant GPU platforms are missing. Watch whether it graduates, and whether Prometheus operators absorb the idea rather than leaving it to a proxy.<\/li>\n<li><strong>Whether the two nonhuman-identity ratios ever converge.<\/strong> Rubrik Zero Labs says 45:1, CyberArk says 82:1, and the gap is nearly 2x on the single number most agent-governance budgets are being sized against. Watch for a methodology disclosure from either side, because without one the ratio is a talking point rather than an input.<\/li>\n<li><strong>Whether Conductor&rsquo;s v4 architecture reaches anyone outside Netflix.<\/strong> The public OSS repository has been unmaintained since December 2023 and the 420-million-executions work lives in an internal fork. The design lessons are transferable; the code is not. Watch whether Netflix re-opens the fork or whether Temporal and the surviving forks absorb the pattern instead.<\/li>\n<li><strong>Whether value-level data assertions show up in mainstream pipeline tooling.<\/strong> Job-status monitoring caught none of the failures described this week. Watch for schema registries and output-range assertions becoming default scaffolding in Airflow and Spark templates rather than something each team rebuilds after its own 11-day incident.<\/li>\n<li><strong>Whether detection-as-code vendors publish latency and volume figures.<\/strong> Mars Security&rsquo;s intel-to-rule automation has no availability date, no pricing and no published latency; the only quantity disclosed is that generated rules are backtested against 30 days of customer data. Watch for a number attached to &ldquo;real time&rdquo; before treating it as a pipeline component.<\/li>\n<li><strong>Whether OpenTelemetry&rsquo;s generative-AI semantic conventions stabilise fast enough to matter.<\/strong> Session-scoped agent tracing is converging on client, server and MCP spans, but the conventions are still moving. Watch for the first agent framework that emits them by default rather than behind a flag &mdash; that is when cross-vendor agent traces stop being a porting exercise.<\/li>\n<\/ul><\/div>\n<p>            <!-- Topic map --><\/p>\n<div class=\"topic-map\">\n              <img decoding=\"async\" src=\"https:\/\/www.cybersecurityinstitute.com\/blog\/wp-content\/uploads\/2026\/09\/topic-map-aiops-2026-09-13.png\" alt=\"Topic map of this week's AI Ops themes: AI Ops at the centre linked to agent observability, telemetry pipeline cost, inference economics, nonhuman identity governance and platform engineering; an inference cluster tying KV cache and prefix routing to SageMaker Inference, AWS, vLLM and TensorRT-LLM; a multi-tenant GPU cluster linking Red Hat AI 3.5, EvalHub and MLflow, Adobe, the NVIDIA DCGM exporter, Prometheus, idle GPU detection and token metering or showback on Kubernetes; an agent observability cluster linking session-scoped tracing, agent cost controls, metric cardinality, Langfuse and Phoenix, StackGen, Dynatrace and the Cursor plugin; a telemetry cost cluster linking tail sampling, the OpenTelemetry Collector, Honeycomb, Bindplane, Bronto, Apica Ascent 3.0 and wide events versus three pillars; a reliability cluster linking Netflix Conductor, workflow orchestration scale, data-value validation, Spark, Airflow and Snowflake with InMarket; an identity cluster linking nonhuman identity governance, the OWASP Agentic Top 10, Model Context Protocol, CyberArk and Oasis, Mars Security and detection engineering automation; and a supervision cluster linking OpenAI, the agent supervision bottleneck, Meta, DeepSeek Elastic Compute and vendor-commissioned survey data\" loading=\"eager\"><\/p>\n<p class=\"caption\">This week&rsquo;s topic map &mdash; inference economics anchors the left, wiring KV-cache prefix routing, vLLM and SageMaker to AWS, and multi-tenant GPU control to Red Hat AI 3.5, Adobe&rsquo;s DCGM-and-Prometheus stack, idle-GPU detection and token showback. Agent observability sits upper-centre through session-scoped tracing, agent cost controls, metric cardinality, Langfuse and StackGen; telemetry pipeline cost runs to the right through tail sampling, the OpenTelemetry Collector, Honeycomb, Bindplane, Bronto and Apica. Nonhuman identity governance anchors the lower right with the OWASP Agentic Top 10, MCP and Mars Security; Netflix Conductor and data-value validation form the reliability cluster below; and a vendor-commissioned-survey node ties Dynatrace, Bronto, Bindplane and the identity research together, because that is where most of this week&rsquo;s percentages actually come from.<\/p>\n<p>              <!-- INTERACTIVE_MAP_LINK_START --><\/p>\n<p style=\"margin:10px 0 0;text-align:center;\"><a href=\"https:\/\/www.cybersecurityinstitute.com\/blog\/?p=5867\" target=\"_blank\" rel=\"noopener\" style=\"display:inline-block;padding:8px 18px;background-color:#0f172a;color:#ffffff !important;text-decoration:none;border-radius:6px;font-size:13px;font-weight:600;\">View interactive topic map &rarr;<\/a><\/p>\n<p><!-- INTERACTIVE_MAP_LINK_END -->\n            <\/div>\n<p>            <!-- Article index --><\/p>\n<h2>Article index<\/h2>\n<p style=\"font-size:13px;color:#6b7280;font-style:italic;margin:0 0 6px 0;\">21 articles, grouped by sub-theme. Fifteen are from this week&rsquo;s coverage window (September 7&ndash;13); six are longer-form foundational reads on the beat. Sponsorships and author affiliations are named in each group.<\/p>\n<h3>Inference and GPU infrastructure economics<\/h3>\n<div class=\"cluster-intro\">Where the accelerator money goes when several tenants share the same fleet &mdash; a routing strategy that turns shared prompt prefixes into cache hits, per-tenant GPU metrics that make idle capacity visible, a multi-tenancy release with no benchmark attached, and a hiring signal about agent sandbox capacity. AWS&rsquo;s figures are its own internal benchmarks; Red Hat&rsquo;s release carries no performance numbers at all.<\/div>\n<table class=\"index-table\">\n<tr>\n<th>Article<\/th>\n<th>Source<\/th>\n<th>Published<\/th>\n<\/tr>\n<tr>\n<td>1. <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/reduce-llm-latency-with-prefix-aware-routing-on-amazon-sagemaker-inference\/\">Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference<\/a><\/td>\n<td class=\"src\">AWS Machine Learning Blog<\/td>\n<td class=\"dt\">Sep 10, 2026<\/td>\n<\/tr>\n<tr>\n<td>2. <a href=\"https:\/\/www.cncf.io\/blog\/2026\/09\/09\/whose-gpus-are-these-anyway-secure-self-service-metrics-for-multi-tenant-kubernetes\/\">Whose GPUs are these, anyway? Secure, self-service metrics for multi-tenant Kubernetes<\/a><\/td>\n<td class=\"src\">CNCF<\/td>\n<td class=\"dt\">Sep 9, 2026<\/td>\n<\/tr>\n<tr>\n<td>3. <a href=\"https:\/\/thenewstack.io\/red-hat-ai-multitenancy\/\">Red Hat AI 3.5 tackles the GPU queue that can stall AI pilots<\/a><\/td>\n<td class=\"src\">The New Stack<\/td>\n<td class=\"dt\">Sep 10, 2026<\/td>\n<\/tr>\n<tr>\n<td>4. <a href=\"https:\/\/thenewstack.io\/deepseek-dsec-agent-hiring\/\">DeepSeek is hiring 150 engineers, and none of them will touch a model<\/a><\/td>\n<td class=\"src\">The New Stack<\/td>\n<td class=\"dt\">Sep 8, 2026<\/td>\n<\/tr>\n<\/table>\n<h3>Agent observability and trace debugging<\/h3>\n<div class=\"cluster-intro\">Why request-scoped APM does not explain an agent failure, and what a session-scoped trace carries instead. Note the affiliations: the InfoQ piece reports on a CNCF post by a StackGen engineer; <strong>&ldquo;It passed CI&rdquo; is a Dynatrace-sponsored post written by a Dynatrace employee<\/strong> and its trace timings are an illustrative example; the Cursor plugin is a Dynatrace product announcement; and <strong>the document-observability column is by the CTO of the vendor 8am and contains no numbers at all<\/strong>.<\/div>\n<table class=\"index-table\">\n<tr>\n<th>Article<\/th>\n<th>Source<\/th>\n<th>Published<\/th>\n<\/tr>\n<tr>\n<td>5. <a href=\"https:\/\/www.infoq.com\/news\/2026\/09\/observability-ai-agents\/\">Session Traces and Cost Controls Help Diagnose AI Agent Failures<\/a><\/td>\n<td class=\"src\">InfoQ<\/td>\n<td class=\"dt\">Sep 11, 2026<\/td>\n<\/tr>\n<tr>\n<td>6. <a href=\"https:\/\/thenewstack.io\/ai-agent-trace-debugging\/\">It passed CI. It passed your evals. The customer still got the wrong answer.<\/a> <em>(Dynatrace-sponsored)<\/em><\/td>\n<td class=\"src\">The New Stack<\/td>\n<td class=\"dt\">Sep 13, 2026<\/td>\n<\/tr>\n<tr>\n<td>7. <a href=\"https:\/\/www.dynatrace.com\/news\/blog\/investigate-production-issues-from-cursor-with-the-dynatrace-plugin\/\">Investigate production issues from Cursor with the Dynatrace plugin<\/a> <em>(vendor blog)<\/em><\/td>\n<td class=\"src\">Dynatrace<\/td>\n<td class=\"dt\">Sep 11, 2026<\/td>\n<\/tr>\n<tr>\n<td>8. <a href=\"https:\/\/www.apmdigest.com\/your-ai-pipeline-may-have-document-observability-problem\">Your AI Pipeline May Have a Document Observability Problem<\/a> <em>(column by 8am&rsquo;s CTO)<\/em><\/td>\n<td class=\"src\">APMdigest<\/td>\n<td class=\"dt\">Sep 9, 2026<\/td>\n<\/tr>\n<\/table>\n<h3>Telemetry pipelines, sampling and storage cost<\/h3>\n<div class=\"cluster-intro\">The group with the highest claim-to-evidence ratio in the issue. <strong>&ldquo;Observability has a data problem&rdquo; is sponsored by Bronto<\/strong> and its retention and search figures are Bronto product claims; <strong>&ldquo;The SIEM Isn&rsquo;t the Problem&rdquo; is a contributed column by Bindplane&rsquo;s CEO and co-founder<\/strong>; the Honeycomb sampling processor is <strong>not merged upstream and not GA<\/strong>; and the Honeycomb wide-events post has no cost figures despite its title. Apica&rsquo;s market numbers are Futurum forecasts, not product results.<\/div>\n<table class=\"index-table\">\n<tr>\n<th>Article<\/th>\n<th>Source<\/th>\n<th>Published<\/th>\n<\/tr>\n<tr>\n<td>9. <a href=\"https:\/\/devops.com\/apica-adds-ai-agents-and-mcp-server-to-platform-for-managing-telemetry-data\/\">Apica Adds AI Agents and MCP Server to Platform for Managing Telemetry Data<\/a><\/td>\n<td class=\"src\">DevOps.com<\/td>\n<td class=\"dt\">Sep 8, 2026<\/td>\n<\/tr>\n<tr>\n<td>10. <a href=\"https:\/\/www.apmdigest.com\/siem-isnt-problem-your-telemetry-architecture\">The SIEM Isn&rsquo;t the Problem. Your Telemetry Architecture Is<\/a> <em>(column by Bindplane&rsquo;s CEO)<\/em><\/td>\n<td class=\"src\">APMdigest<\/td>\n<td class=\"dt\">Sep 8, 2026<\/td>\n<\/tr>\n<tr>\n<td>11. <a href=\"https:\/\/www.honeycomb.io\/blog\/wide-events-vs-three-pillars-ai-observability-costs\">Wide Events vs. Three Pillars: AI Observability Costs<\/a> <em>(foundational, vendor blog)<\/em><\/td>\n<td class=\"src\">Honeycomb<\/td>\n<td class=\"dt\">Sep 8, 2026<\/td>\n<\/tr>\n<tr>\n<td>12. <a href=\"https:\/\/www.honeycomb.io\/blog\/bringing-most-advanced-sampling-opentelemetry-collector\">Bringing the Most Advanced Sampling to the OpenTelemetry Collector<\/a> <em>(foundational, vendor blog)<\/em><\/td>\n<td class=\"src\">Honeycomb<\/td>\n<td class=\"dt\">Sep 1, 2026<\/td>\n<\/tr>\n<tr>\n<td>13. <a href=\"https:\/\/thenewstack.io\/opentelemetry-observability-telemetry-storage\/\">Observability has a data problem. AI is about to make it worse.<\/a> <em>(foundational, Bronto-sponsored)<\/em><\/td>\n<td class=\"src\">The New Stack<\/td>\n<td class=\"dt\">Aug 26, 2026<\/td>\n<\/tr>\n<\/table>\n<h3>Reliability at scale: workflows and data correctness<\/h3>\n<div class=\"cluster-intro\">The two best-evidenced engineering stories of the week, and the only ones where the numbers came out of a production system rather than a survey: an orchestration engine rebuilt around what it loads into memory, and a pipeline that passed every check it had while producing wrong answers.<\/div>\n<table class=\"index-table\">\n<tr>\n<th>Article<\/th>\n<th>Source<\/th>\n<th>Published<\/th>\n<\/tr>\n<tr>\n<td>14. <a href=\"https:\/\/www.infoq.com\/news\/2026\/09\/netflix-conductor-4-workflow\/\">Netflix Reworks Conductor for 420 Million Monthly Workflow Executions and 10X Larger Workflows<\/a><\/td>\n<td class=\"src\">InfoQ<\/td>\n<td class=\"dt\">Sep 11, 2026<\/td>\n<\/tr>\n<tr>\n<td>15. <a href=\"https:\/\/venturebeat.com\/orchestration\/most-pipeline-monitoring-checks-if-the-job-ran-this-one-didnt-check-if-the-numbers-were-right\">Most pipeline monitoring checks if the job ran. This one didn&rsquo;t check if the numbers were right.<\/a><\/td>\n<td class=\"src\">VentureBeat<\/td>\n<td class=\"dt\">Sep 8, 2026<\/td>\n<\/tr>\n<\/table>\n<h3>Security ops and nonhuman identity governance<\/h3>\n<div class=\"cluster-intro\">Agent credentials as an operations problem rather than a security-team problem. The two O&rsquo;Reilly essays are <strong>a two-part series by the same two authors, published two days apart<\/strong> &mdash; read them as one argument, not two independent sources &mdash; and nearly every statistic in them is drawn from an identity-vendor survey. The Mars Security item is a vendor announcement carried under a staff byline, with no availability date, pricing or latency figure.<\/div>\n<table class=\"index-table\">\n<tr>\n<th>Article<\/th>\n<th>Source<\/th>\n<th>Published<\/th>\n<\/tr>\n<tr>\n<td>16. <a href=\"https:\/\/www.helpnetsecurity.com\/2026\/09\/08\/mars-security-real-time-intel-based-detection\/\">Mars Security brings threat intelligence to detection in real time<\/a><\/td>\n<td class=\"src\">Help Net Security<\/td>\n<td class=\"dt\">Sep 8, 2026<\/td>\n<\/tr>\n<tr>\n<td>17. <a href=\"https:\/\/www.oreilly.com\/radar\/the-identity-crisis-no-one-planned-for-governing-non-human-agents-at-enterprise-scale\/\">The Identity Crisis No One Planned For: Governing Nonhuman Agents at Enterprise Scale<\/a> <em>(foundational)<\/em><\/td>\n<td class=\"src\">O&rsquo;Reilly Radar<\/td>\n<td class=\"dt\">Aug 27, 2026<\/td>\n<\/tr>\n<tr>\n<td>18. <a href=\"https:\/\/www.oreilly.com\/radar\/shadow-agents-standing-privileges-and-the-governance-gap-between-deployment-and-discovery\/\">Shadow Agents, Standing Privileges, and the Governance Gap Between Deployment and Discovery<\/a> <em>(foundational, part one of the same series)<\/em><\/td>\n<td class=\"src\">O&rsquo;Reilly Radar<\/td>\n<td class=\"dt\">Aug 25, 2026<\/td>\n<\/tr>\n<\/table>\n<h3>Platform teams, agent org design and the supervision bottleneck<\/h3>\n<div class=\"cluster-intro\">What happens to an engineering organisation when agent throughput outruns human review. OpenAI&rsquo;s figures are its own published data; Meta&rsquo;s claims are self-reported and largely unquantified with no named engineer; and <strong>every percentage in the Dynatrace platform-engineering post comes from Dynatrace&rsquo;s own commissioned 2026 State of SRE and Platform Engineering report<\/strong> &mdash; the same survey behind the sponsored post in the agent-observability group above.<\/div>\n<table class=\"index-table\">\n<tr>\n<th>Article<\/th>\n<th>Source<\/th>\n<th>Published<\/th>\n<\/tr>\n<tr>\n<td>19. <a href=\"https:\/\/thenewstack.io\/openai-agent-research-bottleneck\/\">AI agents are creating more work, not less &mdash; and OpenAI&rsquo;s own numbers back it up<\/a><\/td>\n<td class=\"src\">The New Stack<\/td>\n<td class=\"dt\">Sep 7, 2026<\/td>\n<\/tr>\n<tr>\n<td>20. <a href=\"https:\/\/www.infoq.com\/news\/2026\/09\/meta-organizational-agents\/\">Meta&rsquo;s Recipe for Building Agents as &ldquo;Organizational Second Brains&rdquo;<\/a><\/td>\n<td class=\"src\">InfoQ<\/td>\n<td class=\"dt\">Sep 9, 2026<\/td>\n<\/tr>\n<tr>\n<td>21. <a href=\"https:\/\/www.dynatrace.com\/news\/blog\/how-to-tackle-platform-engineerings-biggest-challenges-in-2026\/\">How to tackle platform engineering&rsquo;s biggest challenges in 2026<\/a> <em>(foundational, vendor survey)<\/em><\/td>\n<td class=\"src\">Dynatrace<\/td>\n<td class=\"dt\">Sep 8, 2026<\/td>\n<\/tr>\n<\/table>\n<p>            <!-- Detailed write-ups --><\/p>\n<h2>Detailed write-ups<\/h2>\n<div class=\"article\">\n<h4>1. Prefix-aware routing on SageMaker: a large win on long shared prefixes, and almost none on short ones<\/h4>\n<p class=\"meta\">AWS Machine Learning Blog &middot; September 10, 2026<\/p>\n<p><strong>Amazon SageMaker Inference<\/strong> gained a third routing strategy, <code>PREFIX_AWARE<\/code>, alongside the existing <code>RANDOM<\/code> and <code>LEAST_OUTSTANDING_REQUESTS<\/code>, and it is <strong>generally available rather than preview<\/strong>. The idea is simple enough to adopt in an afternoon: instead of spreading requests evenly, the endpoint routes requests that share a prompt prefix to the instance that already holds that prefix in its KV cache. The post is written by <strong>Kareem Syed-Mohammed<\/strong>, a principal product manager at AWS, with five AWS colleagues &mdash; a vendor engineering blog, and every number in it is AWS&rsquo;s own internal benchmark run on AWS infrastructure with no third-party verification. Say that out loud before you put it in a capacity plan.<\/p>\n<p>With that caveat in place, the shape of the result is more useful than the headline. On a one-hour long-context benchmark using <strong>8,000-token shared prefixes<\/strong>, P50 time-to-first-token fell <strong>71&ndash;77%<\/strong>, P90 TTFT fell <strong>33&ndash;37%<\/strong>, KV cache hit rate rose from roughly <strong>25% to 82%<\/strong>, and throughput improved <strong>15&ndash;16%<\/strong>. On a thirty-minute short-context run the same configuration produced P50 TTFT down only <strong>13&ndash;16%<\/strong>, P90 down <strong>24&ndash;37%<\/strong>, a hit rate of roughly <strong>30% to 80%<\/strong> &mdash; and throughput up <strong>1.7&ndash;2.0%<\/strong>. That second set is the number to reason from. The benefit is concentrated almost entirely in workloads with long shared prefixes: large system prompts, RAG preambles, few-shot blocks, agent scaffolding reused across every call. If your prompts are short and mostly unique, prefix-aware routing is close to a no-op, and AWS&rsquo;s own data says so.<\/p>\n<p>The operational detail is where the post earns its place. Routing overhead is <strong>1.3&ndash;1.9&nbsp;ms per request<\/strong> against model TTFTs of <strong>63&ndash;280&nbsp;ms<\/strong> across the tests &mdash; small enough to ignore, but worth knowing before you argue about it. Two tunables decide behaviour: <code>PrefixLength<\/code>, settable from <strong>1,024 to 65,536 tokens<\/strong> (AWS&rsquo;s example uses 4,096), which controls how much of the prompt is considered for affinity; and <code>ConcurrencyThreshold<\/code>, from <strong>1 to 1,024<\/strong>, above which the router abandons affinity and falls back to load-based routing. That fallback is the safety valve that stops cache affinity turning into a hot-instance problem, and it is the setting most likely to need tuning against your real traffic shape. The benchmark configuration is published &mdash; <strong>seven <code>ml.p5.48xlarge<\/code> instances serving Llama 3.1 70B Instruct on vLLM with prefix caching enabled<\/strong> &mdash; and the strategy works with both vLLM and TensorRT-LLM containers, across <code>InvokeEndpoint<\/code>, <code>InvokeEndpointWithResponseStream<\/code> and the OpenAI-compatible Chat Completion API. The prerequisite nobody should skip: prefix caching has to be enabled in the serving engine for any of this to do anything. AWS&rsquo;s own summary line &mdash; &ldquo;In our benchmarks on Llama 3.1 70B, this reduced P50 TTFT by up to 77 percent&rdquo; &mdash; is accurate and vendor-self-reported, and the phrase &ldquo;up to&rdquo; is doing the work.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/reduce-llm-latency-with-prefix-aware-routing-on-amazon-sagemaker-inference\/\">AWS Machine Learning Blog (reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference) &mdash; https:\/\/aws.amazon.com\/blogs\/machine-learning\/reduce-llm-latency-with-prefix-aware-routing-on-amazon-sagemaker-inference\/<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>2. Netflix rebuilt Conductor around what it loads into memory &mdash; and the code is not coming back<\/h4>\n<p class=\"meta\">InfoQ &middot; September 11, 2026<\/p>\n<p>The most instructive architecture story of the week is a diagnosis, not a product. <strong>Netflix<\/strong>&rsquo;s Conductor now runs <strong>420 million workflow executions per month<\/strong> across <strong>200,000 workflow definitions<\/strong> used by <strong>150 applications<\/strong> spanning Content and Studio Engineering, Ads and Games &mdash; and the ceiling it broke through was a single design decision. <strong>Aravind Ramkumar<\/strong>, a Conductor maintainer at Netflix, states it plainly: &ldquo;The engine loaded the entire running workflow into memory for evaluation.&rdquo; Every evaluation of a running workflow paid for the whole workflow, so the cost of a decision grew with the size of the thing being decided about. Version 4 separates workflow metadata from task and user data and splits pending from terminal task state, which is why the maximum supported workflow size grew roughly <strong>10x, from about 2,500 tasks to 30,000<\/strong>.<\/p>\n<p>The reported gains are <strong>p99 latency down 40%<\/strong> and a lock-contention result that is the most transferable number here: failed lock-acquisition attempts fell from about <strong>2,700 per interval to essentially zero<\/strong> once workflow evaluation moved to asynchronous processing on exclusive queues. If you run any orchestrator where multiple workers contend for the right to advance the same workflow, that is the pattern worth copying &mdash; not a bigger lock, but a design where only one consumer is ever eligible. The datastore changes underneath are equally legible: execution data migrated from <strong>Dynomite to Apache Cassandra<\/strong>; large task inputs and outputs moved out of the database to <strong>Amazon S3<\/strong>; DynoQueues replaced by <strong>Timestone<\/strong>; <strong>Apache Kafka<\/strong> added to decouple indexing from execution so that an indexing backlog can no longer stall execution; and <strong>Elasticsearch plus Apache Iceberg<\/strong> for near-term indexing and long-term storage respectively. Reported by InfoQ&rsquo;s <strong>Leela Kumili<\/strong>; the figures originate with Netflix.<\/p>\n<p>One thing to be clear about before anyone plans a migration: <strong>none of this is open source<\/strong>. Netflix stopped maintaining the public Conductor repository in <strong>December 2023<\/strong> and moved to an internal fork, and these improvements live there. What you can take is the architecture &mdash; do not load the whole workflow to make one decision, move large payloads to object storage, decouple indexing from the execution path, and give each workflow a single eligible evaluator. What you cannot take is the implementation.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/www.infoq.com\/news\/2026\/09\/netflix-conductor-4-workflow\/\">InfoQ (Netflix reworks Conductor for 420 million monthly workflow executions) &mdash; https:\/\/www.infoq.com\/news\/2026\/09\/netflix-conductor-4-workflow\/<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>3. A GPU idle for eleven days, and the cardinality trick that made it visible<\/h4>\n<p class=\"meta\">CNCF Blog &middot; September 9, 2026<\/p>\n<p>The best practitioner post of the week comes from two <strong>Adobe<\/strong> engineers &mdash; <strong>Bingi Narasimha Karthik<\/strong>, a senior cloud engineer, and <strong>Ramkumar Nagaraj<\/strong>, a senior computer scientist &mdash; writing as CNCF Kubestronaut contributors, and it starts from an observation most platform teams will recognise: the platform scrapes GPU metrics centrally, tenants cannot query their own, and so nobody who could act on a wasted GPU can see it. Their example is <strong>one GPU idle at 0% utilisation for eleven straight days<\/strong>. Their framing line is the right one: &ldquo;You can&rsquo;t fix what you&rsquo;re not allowed to see.&rdquo;<\/p>\n<p>The engineering answer is a per-tenant read path built from parts you already run &mdash; <strong>Prometheus<\/strong> centrally and per tenant, the <strong>NVIDIA DCGM exporter<\/strong>, <code>kube-rbac-proxy<\/code>, <code>prom-label-proxy<\/code>, Nginx, and the Kubernetes API for service discovery and RBAC &mdash; governed by a custom resource, <code>MetricAccess<\/code> in API group <code>observability.ethos.io\/v1alpha1<\/code>. Note the <code>v1alpha1<\/code>: this is in production use at Adobe and still carries an alpha API version. The headline number is a cardinality result rather than a cost result, and it is the one to carry into your own cluster: with <code>metricIsolation: false<\/code> a tenant view produced <strong>more than 10,000 time series<\/strong>; with <code>metricIsolation: true<\/code> it produced <strong>about 300<\/strong>, a <strong>~97% reduction<\/strong>. That is what makes the rest affordable &mdash; GPU utilisation sampled <strong>every second<\/strong> and retained for months, which is what an idle-detection query actually needs, against a default collection interval of <strong>30 seconds<\/strong> that is configurable per <code>MetricAccess<\/code> object. The three detection rules are worth writing down verbatim: <strong>under 5% GPU utilisation sustained for an hour<\/strong> means idle or oversized allocation; <strong>over 70% utilisation with fewer than one request per five minutes<\/strong> means stuck or leaking; <strong>more than ten requests per five minutes with under 5% GPU utilisation<\/strong> means CPU-bound serving with the accelerator wasted. The proxy is open source under Apache 2.0 at <code>github.com\/adobe\/prometheus-multi-tenant-proxy<\/code> and was demoed at KubeCon + CloudNativeCon India 2025.<\/p>\n<p>It is worth reading this against the rest of the week&rsquo;s telemetry-cost coverage, because it is the only item in that cluster with a measured number attached. <strong>Honeycomb<\/strong>&rsquo;s <strong>Mike Goldsmith<\/strong> describes an adaptive tail-sampling processor for the OpenTelemetry Collector, and the status needs stating precisely: it is <strong>&ldquo;working toward alpha&rdquo; upstream &mdash; not merged into OpenTelemetry and not generally available<\/strong>. Today it ships only in Honeycomb&rsquo;s own Collector distribution, derived from Honeycomb&rsquo;s Refinery, and the figures in the post &mdash; a 2-second decision delay, a 30-second trace timeout, 15-second rate recalculation, a 10% sampling goal, 1,000 spans per second &mdash; are <strong>default configuration values and worked examples, not benchmark results<\/strong>. Honeycomb&rsquo;s companion post, <em>&ldquo;Wide Events vs. Three Pillars: AI Observability Costs&rdquo;<\/em> by <strong>Nick Travaglini<\/strong>, is mis-titled: despite &ldquo;Costs&rdquo; in the headline it contains <strong>no cost figures at all<\/strong> &mdash; no dollars per gigabyte, no event sizes, no cardinality counts, no percentage savings &mdash; and its e-commerce chatbot scenario is explicitly illustrative. It is a good conceptual argument for one wide event per unit of work over three disconnected signals; it is not a cost analysis. <strong>&ldquo;Observability has a data problem. AI is about to make it worse.&rdquo; is a Bronto-sponsored post<\/strong> on The New Stack, written by freelance writer <strong>Will Kelly<\/strong>; its figures &mdash; 100x more data retained, twelve months of full-fidelity retention, sub-second search across petabytes &mdash; are <strong>Bronto product claims<\/strong>, and its &ldquo;observability costs reach 20&ndash;30% of infrastructure spend&rdquo; is vendor framing; the piece also discloses that The New Stack&rsquo;s owner, Insight Partners, is an investor in ClickHouse, a named partner. <strong>&ldquo;The SIEM Isn&rsquo;t the Problem. Your Telemetry Architecture Is&rdquo; is a contributed column by Mike Kelly, CEO and co-founder of Bindplane<\/strong>, carried by APMdigest with no sponsorship disclosure; its &ldquo;40%+ reduction in total telemetry volume&rdquo; is a Bindplane claim for Bindplane&rsquo;s own category, and its AI-incident figures (1.7% of unplanned incidents attributed to AI providers in 2023 against 10.7% year-to-date in 2026; 86% of IT leaders confident in AI governance against 30% with written policies) are cited with <strong>no named source at all<\/strong>. And <strong>Apica<\/strong>&rsquo;s Ascent 3.0, launched <strong>8 September<\/strong> with the Venn AI agent and an MCP server, offers one product metric &mdash; a vendor-claimed &ldquo;10x faster&rdquo; flow-only mode, unbenchmarked. The large market numbers in that write-up (14% DevOps tools spending growth in 2027, 72% planning increased investment, $226B by 2030 against $111.1B in 2025) are <strong>Futurum Group forecasts, not Apica results<\/strong>; do not attach them to the product. Adobe&rsquo;s 97% is the only figure in this paragraph that anyone measured.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/www.cncf.io\/blog\/2026\/09\/09\/whose-gpus-are-these-anyway-secure-self-service-metrics-for-multi-tenant-kubernetes\/\">CNCF (whose GPUs are these, anyway?) &mdash; https:\/\/www.cncf.io\/blog\/2026\/09\/09\/whose-gpus-are-these-anyway-secure-self-service-metrics-for-multi-tenant-kubernetes\/<\/a> &middot; <a href=\"https:\/\/www.honeycomb.io\/blog\/bringing-most-advanced-sampling-opentelemetry-collector\">Honeycomb (bringing the most advanced sampling to the OpenTelemetry Collector) &mdash; https:\/\/www.honeycomb.io\/blog\/bringing-most-advanced-sampling-opentelemetry-collector<\/a> &middot; <a href=\"https:\/\/www.honeycomb.io\/blog\/wide-events-vs-three-pillars-ai-observability-costs\">Honeycomb (wide events vs. three pillars) &mdash; https:\/\/www.honeycomb.io\/blog\/wide-events-vs-three-pillars-ai-observability-costs<\/a> &middot; <a href=\"https:\/\/thenewstack.io\/opentelemetry-observability-telemetry-storage\/\">The New Stack (observability has a data problem) &mdash; https:\/\/thenewstack.io\/opentelemetry-observability-telemetry-storage\/<\/a> &middot; <a href=\"https:\/\/www.apmdigest.com\/siem-isnt-problem-your-telemetry-architecture\">APMdigest (the SIEM isn&rsquo;t the problem) &mdash; https:\/\/www.apmdigest.com\/siem-isnt-problem-your-telemetry-architecture<\/a> &middot; <a href=\"https:\/\/devops.com\/apica-adds-ai-agents-and-mcp-server-to-platform-for-managing-telemetry-data\/\">DevOps.com (Apica adds AI agents and MCP server) &mdash; https:\/\/devops.com\/apica-adds-ai-agents-and-mcp-server-to-platform-for-managing-telemetry-data\/<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>4. Red Hat AI 3.5 makes the GPU queue a first-class control &mdash; and publishes no numbers for it<\/h4>\n<p class=\"meta\">The New Stack &middot; September 10, 2026<\/p>\n<p>Announced on <strong>10 September<\/strong>, <strong>Red Hat AI 3.5<\/strong> is about who gets the accelerator rather than which model runs on it, and for platform teams that is the more pressing question. Reported by technology journalist <strong>Adrian Bridgwater<\/strong> for The New Stack with no sponsorship disclosure; the product claims are Red Hat&rsquo;s. <strong>Generally available<\/strong> in this release: fair-share GPU scheduling across tenants, priority-aware admission control and request routing, tenant isolation, CPU offloading for GPU memory management, observability dashboards, per-user token consumption tracking framed as showback, agent templates and starter kits, and <strong>EvalHub<\/strong> pre-deployment safety benchmarking. <strong>Developer preview only<\/strong>: storage offloading for GPU memory management. That split matters &mdash; storage offloading is the piece that would let you oversubscribe most aggressively, and it is the one you cannot yet run in production.<\/p>\n<p>The design argument is sound and worth stating in your own capacity terms: priority-aware serving lets low-priority batch work consume spare capacity instead of requiring separately provisioned GPUs, while admission control protects the real-time inference path from being crowded out by it. Per-user token metering turns &ldquo;the AI platform&rdquo; into a cost line each team can see. <strong>EvalHub<\/strong> is open source and on GitHub; <strong>MLflow<\/strong> provides visual agentic tracing; the whole thing sits on Red Hat&rsquo;s OpenShift AI and Kubernetes foundation. The piece also carries dissent, which is unusual enough to note: a Nutanix vice president argues that virtualization is the better isolation primitive for this problem. <strong>What the release does not contain is a single performance number<\/strong> &mdash; no utilisation gain, no latency figure, no throughput result, no cost comparison. Tushar Katarki, senior director of product for Red Hat AI, offers the framing instead: &ldquo;With Red Hat AI 3.5, we are delivering the operational guardrails, verifiable trust, and multi-tenant controls needed to run AI as a mission-critical service rather than an unpredictable experiment.&rdquo; Treat that as a statement of intent and plan a proof-of-concept with your own measurements, because there are none on offer.<\/p>\n<p>The same multi-tenancy pressure shows up from an unexpected direction: <strong>DeepSeek<\/strong> is hiring <strong>150 backend engineers, none of whom will work on models<\/strong>. The roles are for <strong>DeepSeek Elastic Compute (DSec)<\/strong>, sandbox infrastructure targeting <strong>hundreds of thousands of concurrent agent sandboxes on a single cluster<\/strong>, built on <strong>3FS, Firecracker, QEMU and Docker<\/strong>; harness team lead <strong>Cui Tianyi<\/strong> joined in March 2026, and DeepSeek said in June 2026 it planned at least to double headcount. The headline framing is rhetorical but the substance holds: a frontier lab is spending its hiring budget on isolation, scheduling and sandbox lifecycle rather than on modelling. That is the same problem Red Hat is selling a product for, and it is a reasonable signal about where agent-platform work actually sits.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/thenewstack.io\/red-hat-ai-multitenancy\/\">The New Stack (Red Hat AI 3.5 tackles the GPU queue) &mdash; https:\/\/thenewstack.io\/red-hat-ai-multitenancy\/<\/a> &middot; <a href=\"https:\/\/thenewstack.io\/deepseek-dsec-agent-hiring\/\">The New Stack (DeepSeek is hiring 150 engineers) &mdash; https:\/\/thenewstack.io\/deepseek-dsec-agent-hiring\/<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>5. 3.1 agent-workdays per human workday, and a review queue that did not grow<\/h4>\n<p class=\"meta\">The New Stack &middot; September 7, 2026<\/p>\n<p>The most quotable number of the week comes from OpenAI&rsquo;s own publication, <em>&ldquo;Research Acceleration: View Inside OpenAI&rdquo;<\/em> (August 2026), covering January to August 2026, and reported by <strong>Amanda Caswell<\/strong> for The New Stack. As of mid-August, OpenAI&rsquo;s research organisation logged <strong>3.1 agent-workdays per human workday<\/strong>. Inference cost per researcher ran to a <strong>median of $600 a day and a 90th percentile of $7,000<\/strong> &mdash; roughly a <strong>12x spread<\/strong> across people doing nominally the same job, which on its own is an argument for per-user metering before it is an argument about models. The finding with the most operational weight is this: humans had to intervene on <strong>more than 50% of <em>successful<\/em> long-running tasks<\/strong> in the four-to-eight-hour band. Not failed tasks &mdash; successful ones. The cost of agent work is not concentrated in failures you can alert on; it is spread across successes that needed a person anyway.<\/p>\n<p>One caveat on the headline. The article&rsquo;s framing &mdash; that agents are &ldquo;creating more work, not less&rdquo; &mdash; is <strong>the author&rsquo;s interpretation, not an OpenAI claim<\/strong>. What the data supports is narrower and more useful: agent throughput scaled, human review capacity did not, and supervision became the binding constraint. OpenAI&rsquo;s own line is the more careful one: &ldquo;More agent hours don&rsquo;t automatically mean more useful research.&rdquo; A second figure shows how workloads move rather than shrink under governance: after usage restrictions, GPU allocation to the <strong>Astra<\/strong> model fell <strong>59.2%<\/strong> while allocation to other models rose <strong>17.2%<\/strong>, absorbing roughly <strong>85% of the Astra drop<\/strong> rather than reducing total spend. If you throttle one model without a capacity policy behind it, you have moved the bill, not cut it. OpenAI&rsquo;s stated goal remains an automated AI researcher by <strong>March 2028<\/strong> &mdash; a target that has to clear the very bottleneck this dataset documents.<\/p>\n<p>Two organisational reads sit alongside it, and both need their provenance stated. <strong>Meta<\/strong>&rsquo;s recipe for agents as &ldquo;organizational second brains&rdquo;, reported by InfoQ&rsquo;s <strong>Sergio De Simone<\/strong>, describes a four-layer design that is genuinely worth borrowing: a knowledge system, a reasoning layer of composable recipes, an evaluation framework, and a self-improvement loop that requires no model retraining. The substance is in that structure, because the results are <strong>self-reported by Meta, largely unquantified, and attributed to no named engineer<\/strong> &mdash; assessments dropping &ldquo;from days to minutes&rdquo;, results &ldquo;useful almost all the time&rdquo;, &ldquo;zero regressions across improvement cycles&rdquo;. The only hard number in the piece is <strong>200+ structured files<\/strong> in the knowledge layer. Separately, <strong>Dynatrace<\/strong>&rsquo;s platform-engineering post by <strong>Klint Finley<\/strong>, a principal content editor at Dynatrace, reports that <strong>89%<\/strong> of organisations have an internal developer platform, <strong>74%<\/strong> provide dashboards, <strong>53%<\/strong> expected toil reduction against <strong>45%<\/strong> who observed it, <strong>53%<\/strong> expected MTTD\/MTTR gains against <strong>46%<\/strong> who observed them, and <strong>52%<\/strong> monitor AI tools. <strong>Every one of those percentages comes from Dynatrace&rsquo;s own commissioned State of SRE and Platform Engineering 2026 report<\/strong> &mdash; the same survey that supplies the statistics in the Dynatrace-sponsored New Stack post discussed below. They are one source, not two, and the expectation-versus-observation gap is the only part worth carrying: platform teams are consistently getting less toil relief than they budgeted for.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/thenewstack.io\/openai-agent-research-bottleneck\/\">The New Stack (AI agents are creating more work, not less) &mdash; https:\/\/thenewstack.io\/openai-agent-research-bottleneck\/<\/a> &middot; <a href=\"https:\/\/www.infoq.com\/news\/2026\/09\/meta-organizational-agents\/\">InfoQ (Meta&rsquo;s recipe for building agents as organizational second brains) &mdash; https:\/\/www.infoq.com\/news\/2026\/09\/meta-organizational-agents\/<\/a> &middot; <a href=\"https:\/\/www.dynatrace.com\/news\/blog\/how-to-tackle-platform-engineerings-biggest-challenges-in-2026\/\">Dynatrace (how to tackle platform engineering&rsquo;s biggest challenges in 2026) &mdash; https:\/\/www.dynatrace.com\/news\/blog\/how-to-tackle-platform-engineerings-biggest-challenges-in-2026\/<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>6. Eleven days green, forty per cent wrong: the monitoring gap under every RAG and feature pipeline<\/h4>\n<p class=\"meta\">VentureBeat &middot; September 8, 2026<\/p>\n<p>This is a contributed column &mdash; <strong>Siddharth Arun<\/strong> is a senior member of technical staff at <strong>Salesforce<\/strong>, and the incidents he describes happened at a prior employer, <strong>InMarket<\/strong> (formerly NinthDecimal), not at Salesforce &mdash; and it is the most directly actionable piece in the issue. The first incident: a pipeline <strong>ran green for eleven days<\/strong> while producing audience counts that were <strong>about 40% wrong<\/strong>. Every job-level check passed, because every job-level check was asking whether the job ran, and none was asking whether the output was right. The second, at the same organisation: <strong>three days of audience data vanished<\/strong> without an alert firing. The scale gives the failure teeth &mdash; <strong>over a petabyte<\/strong> of location and advertising event data, <strong>billions of events per day<\/strong>, on Spark ETL, Airflow DAGs, Snowflake and AWS S3 with MapR legacy and an Iceberg lakehouse pattern. His sentence is the one to circulate internally: &ldquo;The system ran successfully while producing incorrect results.&rdquo;<\/p>\n<p>The prescribed fix is not more job-status monitoring; it is <strong>value-level assertions and schema validation registries inside the pipeline<\/strong> &mdash; expected ranges, row-count deltas against history, distribution checks, null-rate bounds, referential expectations &mdash; evaluated on the data the job emits rather than on the job&rsquo;s exit code. For AI-operations teams, the relevance is direct: the same pattern of green-but-wrong sits underneath every retrieval corpus and every feature table feeding a model, and a silently degraded embedding index produces confidently wrong answers rather than an outage. One number in the column is not the author&rsquo;s: the &ldquo;68% of data teams report a mean time-to-detect of four or more hours for data incidents&rdquo; figure is sourced to a <strong>Monte Carlo 2023 vendor survey<\/strong>, and should be read as such.<\/p>\n<p>The same argument arrives in a weaker form from <strong>Jeff Sidell, CTO of 8am<\/strong>, an OCR and document-pipeline vendor, writing in APMdigest with no disclosure on the page. His case is that document ingestion &mdash; extraction fidelity, layout handling, OCR error rates &mdash; is an unmonitored stage feeding AI pipelines, and that the observability stack watches the model and ignores the documents going into it. That is a reasonable argument and a fair gap to name. It is worth stating plainly that <strong>the article contains no numbers at all<\/strong>: no error rates, no volumes, no accuracy deltas, no customer figures. Use it as a prompt to instrument your document-ingestion stage; do not use it as evidence for anything.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/venturebeat.com\/orchestration\/most-pipeline-monitoring-checks-if-the-job-ran-this-one-didnt-check-if-the-numbers-were-right\">VentureBeat (most pipeline monitoring checks if the job ran) &mdash; https:\/\/venturebeat.com\/orchestration\/most-pipeline-monitoring-checks-if-the-job-ran-this-one-didnt-check-if-the-numbers-were-right<\/a> &middot; <a href=\"https:\/\/www.apmdigest.com\/your-ai-pipeline-may-have-document-observability-problem\">APMdigest (your AI pipeline may have a document observability problem) &mdash; https:\/\/www.apmdigest.com\/your-ai-pipeline-may-have-document-observability-problem<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>7. Session-scoped traces: the span model that explains why an agent looped<\/h4>\n<p class=\"meta\">InfoQ &middot; The New Stack &middot; Dynatrace &middot; September 8&ndash;13, 2026<\/p>\n<p>InfoQ&rsquo;s <strong>Mark Silvester<\/strong> reports on a CNCF member post by <strong>Sabith K Soopy<\/strong>, a principal engineer at <strong>StackGen<\/strong>, and the core claim is one most teams discover the hard way: conventional APM tells you whether services responded, not why an agent looped, called an invalid endpoint, or silently skipped work. Agent failure is a <strong>session-scoped<\/strong> phenomenon, and a request-scoped trace cannot represent it. His summary: &ldquo;The hardest part isn&rsquo;t building them; it&rsquo;s understanding what they&rsquo;re doing when they go wrong.&rdquo; Note up front that <strong>the article contains no measured numbers<\/strong> &mdash; no latencies, no costs, no volumes. It is a design pattern, and a good one.<\/p>\n<p>The pattern: every LLM call, tool execution and sub-agent delegation becomes its own span carrying execution latency and token cost, with nested child spans preserving the delegation chain, assembled through <strong>Langfuse<\/strong> into a session trace. The export design deliberately trades completeness for availability &mdash; an asynchronous batch exporter queues spans in memory and flushes periodically, so a telemetry outage drops data rather than blocking the agent. Cost controls are enforced <strong>before execution rather than after<\/strong>: hard iteration caps, per-tool call limits, and pre-execution blocking of identical consecutive tool requests, with session cost compared against a rolling average to flag model-routing errors and tool-hallucination loops. The cardinality warning is stated explicitly and is the one most teams will violate first: <strong>dynamic session IDs must never be used as Prometheus metric labels<\/strong> &mdash; only bounded operational metrics get exported, and the session detail lives in traces. The post-incident record is an append-only searchable log of tool calls, governance decisions and memory operations with credentials and PII redacted. The surrounding tool landscape is named usefully: Langfuse for nested session traces, <strong>LangSmith<\/strong> for turning anomalous traces into test datasets, <strong>Arize Phoenix<\/strong> as an OpenTelemetry-native option with LLM-as-a-judge evaluation, and the OpenTelemetry generative-AI semantic conventions covering client, server and MCP spans.<\/p>\n<p>Two adjacent items make the same argument with declared commercial interests, and both need labelling. <strong>&ldquo;It passed CI. It passed your evals. The customer still got the wrong answer.&rdquo; is a Dynatrace-sponsored post on The New Stack, written by Sean O&rsquo;Dell, who works at Dynatrace.<\/strong> Its statistics &mdash; <strong>77%<\/strong> of platform-engineering teams embedding observability in some services and <strong>40%<\/strong> fully integrated, n=919 &mdash; come from <strong>Dynatrace&rsquo;s own commissioned 2026 State of SRE report<\/strong>, the same survey behind the Dynatrace platform-engineering blog in this issue; they corroborate each other only in the sense that a number corroborates itself. Its trace timings &mdash; a 12.4-second total run, an 8.1-second generation call, 2.6 seconds across three identical searches &mdash; are an <strong>illustrative worked example, not measured production data<\/strong>. Read for the argument that passing CI and passing evals says nothing about what a specific customer session did; ignore the numbers. Separately, <strong>Dynatrace<\/strong>&rsquo;s own blog, by <strong>Philipp Hamann<\/strong> and <strong>Christoph Enzinger<\/strong>, announces a <strong>Cursor plugin<\/strong>, generally available in the Cursor Marketplace, connecting the IDE to production telemetry through an MCP server bundling <strong>30 Dynatrace skills<\/strong>. That 30 is the only figure in the post &mdash; no time-savings claim, no MTTR figure, no version number. Worth noting that this is the <strong>third Dynatrace-origin item in a single issue<\/strong>; the pattern of pulling production context into the editor is real and several vendors are shipping it, but one vendor should not carry three slots of framing unchallenged.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/www.infoq.com\/news\/2026\/09\/observability-ai-agents\/\">InfoQ (session traces and cost controls help diagnose AI agent failures) &mdash; https:\/\/www.infoq.com\/news\/2026\/09\/observability-ai-agents\/<\/a> &middot; <a href=\"https:\/\/thenewstack.io\/ai-agent-trace-debugging\/\">The New Stack (it passed CI, it passed your evals) &mdash; https:\/\/thenewstack.io\/ai-agent-trace-debugging\/<\/a> &middot; <a href=\"https:\/\/www.dynatrace.com\/news\/blog\/investigate-production-issues-from-cursor-with-the-dynatrace-plugin\/\">Dynatrace (investigate production issues from Cursor with the Dynatrace plugin) &mdash; https:\/\/www.dynatrace.com\/news\/blog\/investigate-production-issues-from-cursor-with-the-dynatrace-plugin\/<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>8. Nonhuman identity: an ownership problem wearing a credentials problem&rsquo;s clothes<\/h4>\n<p class=\"meta\">O&rsquo;Reilly Radar &middot; Help Net Security &middot; August 25 &ndash; September 8, 2026<\/p>\n<p><strong>Tushar Badlani<\/strong> and <strong>Mohit Bansal<\/strong> wrote a <strong>two-part series<\/strong> on O&rsquo;Reilly Radar &mdash; <em>Shadow Agents<\/em> on 25 August and <em>The Identity Crisis No One Planned For<\/em> on 27 August &mdash; and it should be read as one argument by two authors, not as two independent confirmations. The analysis is independent and every statistic is attributed to a named third party, but most of those third parties are identity-security vendors, which is the standing caveat on this entire category. The fault line is visible in the first number: the nonhuman-to-human identity ratio is cited as <strong>45:1 by Rubrik Zero Labs<\/strong> and <strong>82:1 by CyberArk&rsquo;s 2025 Identity Security Landscape<\/strong> &mdash; two sources disagreeing by nearly <strong>2x<\/strong> on the single figure most agent-governance budgets are sized against.<\/p>\n<p>The readiness numbers are consistent even where the ratios are not. <strong>92%<\/strong> lack confidence that legacy IAM tooling can govern AI agents and <strong>78%<\/strong> have no AI identity policy (Oasis Security with the Cloud Security Alliance, January 2026, n=383). On credential hygiene, <strong>47%<\/strong> of nonhuman identities have gone a year or more without rotation, and <strong>62%<\/strong> of AWS identities inactive for 90 or more days still retain full access (both Entro). The ownership gap is the part that is genuinely an operations problem rather than a security one: <strong>8%<\/strong> of identities have lost their HR owner, <strong>51%<\/strong> of organizations have no clear owner for AI identities (One Identity\/GigaOm; WEF 2025), and <strong>66%<\/strong> of enterprises report an NHI-related compromise. The authors put the operational consequence better than a control framework would: &ldquo;An identity with no owner can&rsquo;t be reviewed on a schedule, rotated with confidence, or disabled without someone first proving a negative: that nothing still depends on it.&rdquo; The 2026 attack evidence is concrete &mdash; poisoned Model Context Protocol tool descriptions (Microsoft, June 2026), a fake AI-agent skill that reached about <strong>26,000 agents<\/strong> (The Hacker News, June 2026), and the WriteOut cross-tenant session-token exposure. Governance scaffolding exists in the <strong>OWASP Top 10 for Agentic Applications<\/strong>, released December 2025 with 100+ contributors, where categories <strong>ASI03<\/strong> and <strong>ASI04<\/strong> cover identity and privilege. The market sizing &mdash; $11.3B in 2025 to a projected $38.8B by 2036 at 12.2% CAGR (Meticulous Research) &mdash; is analyst forecasting and adds nothing operational.<\/p>\n<p>The first instalment carries figures worth keeping separately. <strong>1%<\/strong> have fully implemented just-in-time privileged access while <strong>91%<\/strong> have at least half of privileged access always-on (CyberArk, January 2026, n=500 US practitioners); <strong>45%<\/strong> of the workforce uses unapproved AI tools (Verizon 2026 DBIR); <strong>75%<\/strong> of CISOs found unsanctioned AI tools in production (Saviynt 2026); <strong>10 of 11<\/strong> surveyed open source coding agents were found vulnerable (Adversa AI GuardFall); and Sysdig Threat Research recorded <strong>31 seconds<\/strong> to lateral progression after an authentication failure and <strong>1,342<\/strong> Nacos configuration records encrypted in the JADEPUFFER operation. The direction both pieces argue for is intent-bound, short-lived authorization replacing long-lived tokens &mdash; while conceding that reliably discovering every shadow agent remains unsolved, which is the honest part.<\/p>\n<p>On the tooling side of the same beat, <strong>Mars Security<\/strong> announced a system that turns threat-intelligence advisories into validated, deployable detection rules, backtested against <strong>30 days<\/strong> of customer data before deployment, with integrations to <strong>CrowdStrike Falcon, Wiz, Splunk, Snowflake and Databricks<\/strong>. Read as detection-engineering automation rather than as a threat story, it belongs in the same conversation as every other intel-to-artefact pipeline: the value is in eliminating the manual translation step between an advisory and a rule that actually runs, and in the backtest that stops a new rule flooding the queue. Help Net Security carried it under an <strong>&ldquo;Industry News&rdquo; staff byline from the vendor announcement<\/strong>, and it comes with <strong>no availability date, no pricing, no latency figure and no detection-volume figure<\/strong> &mdash; the 30-day backtest window is the only quantity disclosed. &ldquo;Real time&rdquo; in the headline is currently a positioning claim.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/www.oreilly.com\/radar\/the-identity-crisis-no-one-planned-for-governing-non-human-agents-at-enterprise-scale\/\">O&rsquo;Reilly Radar (the identity crisis no one planned for) &mdash; https:\/\/www.oreilly.com\/radar\/the-identity-crisis-no-one-planned-for-governing-non-human-agents-at-enterprise-scale\/<\/a> &middot; <a href=\"https:\/\/www.oreilly.com\/radar\/shadow-agents-standing-privileges-and-the-governance-gap-between-deployment-and-discovery\/\">O&rsquo;Reilly Radar (shadow agents, standing privileges, and the governance gap) &mdash; https:\/\/www.oreilly.com\/radar\/shadow-agents-standing-privileges-and-the-governance-gap-between-deployment-and-discovery\/<\/a> &middot; <a href=\"https:\/\/www.helpnetsecurity.com\/2026\/09\/08\/mars-security-real-time-intel-based-detection\/\">Help Net Security (Mars Security brings threat intelligence to detection in real time) &mdash; https:\/\/www.helpnetsecurity.com\/2026\/09\/08\/mars-security-real-time-intel-based-detection\/<\/a><\/p>\n<\/p><\/div>\n<p>            <!-- Calls to action --><\/p>\n<div class=\"watchlist\">\n<h2>Calls to action<\/h2>\n<ul>\n<li><strong>Measure your shared-prefix length before enabling prefix-aware routing.<\/strong> AWS&rsquo;s own benchmark gives 71&ndash;77% P50 TTFT reduction on 8,000-token shared prefixes and only 1.7&ndash;2.0% throughput gain on short contexts. Pull the actual prefix distribution from your prompt logs first; if your shared preamble is small, set <code>PrefixLength<\/code> and <code>ConcurrencyThreshold<\/code> from that measurement or skip the change entirely.<\/li>\n<li><strong>Turn on prefix caching in the serving engine before you touch routing.<\/strong> The SageMaker result depends on vLLM prefix caching being enabled; routing affinity to an instance whose cache is off buys nothing. Confirm the engine flag, then measure KV cache hit rate as your primary indicator rather than latency.<\/li>\n<li><strong>Add value-level assertions to your top three pipelines this week.<\/strong> Eleven days green with 40% wrong counts, and three days of vanished data with no alert, both passed every job-status check. Assert expected ranges, row-count deltas against history and null-rate bounds on the output, and register schemas &mdash; especially on anything feeding a retrieval corpus or a feature table.<\/li>\n<li><strong>Give tenants read access to their own GPU metrics, with label enforcement.<\/strong> Adobe&rsquo;s pattern &mdash; <code>kube-rbac-proxy<\/code> plus <code>prom-label-proxy<\/code> in front of per-tenant Prometheus &mdash; cut a tenant view from 10,000+ series to about 300. Ship the three idle-detection rules with it: under 5% for an hour, over 70% with under one request per five minutes, and over ten requests per five minutes with under 5% utilisation.<\/li>\n<li><strong>Ban dynamic session IDs from your metric labels today.<\/strong> This is the cheapest fix in the issue and the one most agent deployments get wrong. Session detail belongs in traces; export only bounded operational metrics, and audit your existing agent instrumentation for unbounded label values before your Prometheus bill finds them for you.<\/li>\n<li><strong>Enforce agent cost limits before execution, not after.<\/strong> Hard iteration caps, per-tool call limits and pre-execution blocking of identical consecutive tool calls stop a hallucination loop at the point it starts. Compare session cost against a rolling average to catch model-routing errors; a post-hoc dashboard only tells you what you already spent.<\/li>\n<li><strong>Meter inference cost per user and publish the spread.<\/strong> OpenAI&rsquo;s own data shows a 12x gap between the median researcher at $600\/day and the 90th percentile at $7,000\/day. You cannot manage a spread you have not measured, and per-user showback &mdash; the capability Red Hat just made generally available &mdash; is the mechanism that makes it visible without rationing anyone.<\/li>\n<li><strong>Track human intervention rate on successful agent runs, not just failure rate.<\/strong> More than half of OpenAI&rsquo;s successful four-to-eight-hour tasks needed a person. If your only agent metric is task success, your supervision load is invisible and your capacity plan is wrong by whatever that intervention rate turns out to be.<\/li>\n<li><strong>Verify the upstream status of any sampling processor before you plan around it.<\/strong> Honeycomb&rsquo;s adaptive tail sampler is &ldquo;working toward alpha&rdquo; and ships only in Honeycomb&rsquo;s own Collector distribution. Check the OpenTelemetry contrib repository yourself for anything described as coming to the Collector, and treat published config defaults as defaults rather than as benchmark results.<\/li>\n<li><strong>Find your unowned nonhuman identities and give each one a named human.<\/strong> 8% have lost their HR owner and 51% of organizations have no clear owner for AI identities. Start with the AWS case &mdash; identities inactive 90+ days that still hold full access &mdash; and rotate or disable rather than waiting for a discovery tool that can prove nothing depends on them.<\/li>\n<li><strong>Ask every vendor for the survey methodology behind the percentage they quoted you.<\/strong> Three items in this issue rest on one Dynatrace-commissioned survey, two headline claims are product marketing inside sponsored posts, and the two identity ratios disagree by nearly 2x. Sample size, sponsor and question wording, or the number does not enter your business case.<\/li>\n<\/ul><\/div>\n<\/td>\n<\/tr>\n<p>        <!-- Footer --><\/p>\n<tr>\n<td class=\"footer\">\n<p class=\"brand\">AI Ops<\/p>\n<p>A weekly intelligence bulletin from Security Radar LLC.<br \/>\n            Curated by Paul Davis &middot; <a href=\"mailto:paul.davis@security-radar.com\">paul.davis@security-radar.com<\/a><\/p>\n<p>&copy; 2026 Security Radar LLC. All rights reserved.<\/p>\n<p>Article titles and summaries are excerpted for review and commentary; all linked articles remain the copyright of their respective publishers and authors.<\/p>\n<p>*|LIST:ADDRESS|*<\/p>\n<p><a href=\"*|ARCHIVE|*\">View this email in your browser<\/a> &middot; <a href=\"*|UNSUB|*\">Unsubscribe<\/a><\/p>\n<\/td>\n<\/tr>\n<\/table>\n<\/td>\n<\/tr>\n<\/table>\n","protected":false},"excerpt":{"rendered":"<p>September 13, 2026 &middot; Weekly Edition AI Ops A thin week for evidence and a heavy one for marketing. Five items in this issue carry numbers anyone measured: AWS cut P50 time-to-first-token 71&ndash;77% with prefix-aware routing on SageMaker (its own benchmark, on its own hardware); Netflix rebuilt Conductor to 420&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[48],"tags":[],"class_list":["post-5868","post","type-post","status-publish","format-standard","hentry","category-ai-ops"],"_links":{"self":[{"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5868","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5868"}],"version-history":[{"count":1,"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5868\/revisions"}],"predecessor-version":[{"id":5896,"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5868\/revisions\/5896"}],"wp:attachment":[{"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5868"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5868"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5868"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}