{"id":5825,"date":"2026-09-06T12:30:57","date_gmt":"2026-09-06T17:30:57","guid":{"rendered":"https:\/\/www.cybersecurityinstitute.com\/blog\/?p=5825"},"modified":"2026-09-06T12:30:57","modified_gmt":"2026-09-06T17:30:57","slug":"ai-ops-weekly-september-6-2026","status":"publish","type":"post","link":"https:\/\/www.cybersecurityinstitute.com\/blog\/?p=5825","title":{"rendered":"AI Ops Weekly &mdash; September 6, 2026"},"content":{"rendered":"<style>\n.single .entry-title,\n.single .entry-header .entry-title,\n.single .post-title,\n.single header.entry-header h1,\n.single h1.entry-title,\n.single .page-title,\n.post-template-default h1.entry-title,\n.post-template-default .entry-header,\narticle .entry-header,\narticle .entry-title { display: none !important; }\n.single .entry-header { margin: 0 !important; padding: 0 !important; }\n.single .entry-content { margin-top: 0 !important; padding-top: 0 !important; }\n<\/style>\n<table role=\"presentation\" class=\"wrapper\" cellpadding=\"0\" cellspacing=\"0\" border=\"0\" width=\"100%\">\n<tr>\n<td align=\"center\">\n<table role=\"presentation\" class=\"container\" cellpadding=\"0\" cellspacing=\"0\" border=\"0\" width=\"680\">\n<p>        <!-- Banner --><\/p>\n<tr>\n<td class=\"banner\" style=\"background-color:#0e7490;background:linear-gradient(135deg,#0e7490 0%,#0891b2 100%);padding:36px 32px;color:#ffffff;\">\n<p class=\"date\" style=\"color:#ffffff !important;\">September 6, 2026 &middot; Weekly Edition<\/p>\n<h1 style=\"color:#ffffff !important;\">AI Ops<\/h1>\n<p class=\"tagline\" style=\"color:#ffffff !important;\">A week of measurements rather than announcements. AWS took a GPU cold start apart phase by phase and got eight minutes down to under thirty seconds on a warm node; Google Cloud benchmarked Gemma 3 on TPU v6e and found a hard concurrency wall at 64 users on the larger model; NVIDIA published the memory arithmetic behind GPU sizing; Datadog showed its own books, saving more than $1 million a month by changing which model does what; Elastic explained why bills triple while token prices fall; and Broadcom moved the whole argument on-premises with VMware Private AI Cloud. Underneath: DoorDash running 130,000 engineering tasks a month through cloud agents, AgentCore memory lifecycle defaults, and an SRE profession that now spends most of its time watching models. Twenty-five stories.<\/p>\n<\/td>\n<\/tr>\n<p>        <!-- At a glance --><\/p>\n<tr>\n<td class=\"content\">\n<h2>This week at a glance<\/h2>\n<p>This was the week the AI-operations beat stopped arguing about architecture and started publishing numbers, and almost every one of them points at the same thing: <strong>capacity and context you pay for but never use<\/strong>. AWS took a GPU inference cold start apart and found not one bottleneck but <strong>six sequential phases<\/strong> totalling eight minutes &mdash; driver init, image pull, weight loading, compile &mdash; and got the warm-node path under <strong>30 seconds<\/strong>, with the dominant phase depending on model size (a 64&nbsp;GB model spends 53 seconds in <code>torch.compile<\/code>; a 203&nbsp;GB one spends 423 seconds just loading weights). Google Cloud benchmarked the same question from the serving side and found <strong>Gemma 3 27B hits a wall past 64 concurrent users<\/strong> on decode-heavy generation, plateauing at a 4.12x throughput multiplier where the 12B model reaches 8.19x &mdash; which makes &ldquo;use the bigger model&rdquo; an expensive default rather than a safe one. NVIDIA supplied the memory arithmetic underneath both (<strong>FP8 quantisation cuts Llama-3.1-8B weights 43.5%<\/strong>, from 16.06&nbsp;GB to 9.08&nbsp;GB, with no retraining) and, on Jetson, showed distilled 2026 models delivering reasoning at the edge with <strong>3.37x to 6.28x decode speedups<\/strong>. The money strand converged on the same diagnosis from the software side: Elastic&rsquo;s audit of 30 production engineering teams attributed <strong>62% of the inference bill to context and lack of guidance<\/strong> rather than model price, at a moment when token prices have fallen <strong>75%<\/strong> and agentic workflows consume 5&ndash;30x more tokens per task; Datadog published its own ledger, <strong>saving more than $1 million a month<\/strong> mostly by routing internal coding work from Opus to Sonnet ($687,000\/month for a measured 8% proficiency loss) and dialling effort levels down; and a systems guide to token optimisation showed prompt caching cutting input cost roughly <strong>70&ndash;76%<\/strong> on two worked reference systems. Meanwhile <strong>Broadcom<\/strong> made the structural bet that all of this runs on-premises, launching VMware Private AI Cloud and AI Factory on Cloud Foundation 9 against repatriation figures showing public-cloud share for these workloads down <strong>15% year over year to 41%<\/strong>; <strong>DoorDash<\/strong> showed what the agent-native platform actually looks like at scale with <strong>130,000 engineering tasks in a month<\/strong> on Firecracker microVMs behind an MCP gateway; AWS documented lifecycle policies that stop agent memory becoming a standing bill; and the observability side did the unglamorous groundwork &mdash; span anatomy, sampling strategy, and instrumentation quality you can actually score &mdash; while Dynatrace&rsquo;s State of SRE research reported that <strong>67% of SREs now monitor AI models<\/strong>. The through-line for a platform team is blunt: the wins this week came from measuring one layer more precisely, not from buying one layer more.<\/p>\n<div class=\"watchlist\">\n<h2>On our watch list<\/h2>\n<ul>\n<li><strong>Whether repatriation figures hold up outside vendor decks.<\/strong> Broadcom cites 56% running or planning production inference in private cloud and public-cloud share for these workloads down 15% year over year to 41%. Watch for independent measurement, and watch whether the driver stays cost or shifts decisively to data control.<\/li>\n<li><strong>Agent identity becoming an infrastructure product.<\/strong> VMware&rsquo;s AgentMinder bundles agent identity, runtime policy and observability into the virtualization layer. Watch whether the control plane for non-human principals settles with the infrastructure vendors, the identity vendors, or the agent frameworks &mdash; whoever wins it inherits the audit trail.<\/li>\n<li><strong>The gap between planning LLM observability and having it.<\/strong> Elastic&rsquo;s figures have 85% of enterprise IT organisations planning it and 8% finished. With cost varying up to 30x between identical runs of the same task, that gap is where the unexplained portion of every AI budget currently lives.<\/li>\n<li><strong>Whether published internal AI ledgers become normal.<\/strong> Datadog put real dollar figures and a measured proficiency trade on the record. Watch whether other engineering organisations follow, because a few more public numbers would do more for AI cost discipline than any amount of vendor benchmarking.<\/li>\n<li><strong>Autonomous remediation moving from claim to evidence.<\/strong> DataAgent launched the day its funding was announced, and its cost-reduction figure is its own. Watch for a customer-verified deployment fixing production faults unattended, and for what the rollback story looks like when the fix is wrong.<\/li>\n<li><strong>The CPU side of the latency budget.<\/strong> Akamai&rsquo;s figure that CPU-side processing can account for up to 90.6% of agentic latency is the most actionable claim on the list and the least instrumented. Watch whether orchestration frameworks start emitting the spans needed to confirm or refute it on real workloads.<\/li>\n<li><strong>Golden paths rewritten for machine consumers.<\/strong> Gartner naming &ldquo;agent experience&rdquo; as a persona and a reported ~40% cost reduction from field-selection tooling both point the same way. Watch whether internal developer platforms start publishing machine-readable contracts &mdash; schemas, tool descriptions, permitted action sets &mdash; alongside their human documentation.<\/li>\n<li><strong>Per-tenant persistence economics at agent scale.<\/strong> If agents provision backing services as readily as they provision compute, idle database capacity becomes a standing bill nobody approved. Watch whether sub-second provisioning and scale-to-zero become table stakes, or whether teams retreat to shared multi-tenant schemas and inherit the isolation problems that come with them.<\/li>\n<\/ul><\/div>\n<p>            <!-- Topic map --><\/p>\n<div class=\"topic-map\">\n              <img decoding=\"async\" src=\"https:\/\/www.cybersecurityinstitute.com\/blog\/wp-content\/uploads\/2026\/09\/topic-map-aiops-2026-09-06.png\" alt=\"Topic map of this week's AI Ops themes: AI Ops at the centre linked to AI cost control, observability practice, agent platforms and inference economics; an inference cluster tying GPU cold starts, GPU memory sizing, quantization, concurrency limits, vLLM, NVIDIA, NVIDIA Jetson, TPU v6e with Gemma 3 and Google Cloud; a token-economics cluster linking prompt caching, Datadog, Headroom and Elastic; an observability cluster linking OpenTelemetry, instrumentation quality, Grafana Labs, Dynatrace and SRE practice; an agent-platform cluster linking DoorDash Flux, Model Context Protocol, agent persistence, AgentCore memory, agent evaluation loops and retrieval engineering; plus an on-prem cluster around Broadcom, VMware Cloud Foundation 9 and AWS, and an operations cluster around Kubernetes, federated identity, DataAgent and agentic latency with Akamai\" loading=\"eager\"><\/p>\n<p class=\"caption\">This week&rsquo;s topic map &mdash; inference economics anchors the left-hand cluster, wiring GPU cold starts, memory sizing, quantisation and concurrency limits to vLLM, NVIDIA, Jetson and TPU v6e. AI cost control sits centre, joining token economics, prompt caching, Datadog and Elastic. Agent platforms run to the right through DoorDash Flux, the Model Context Protocol, agent persistence, AgentCore memory and evaluation loops; observability practice anchors the lower-left through OpenTelemetry, instrumentation quality, Grafana and Dynatrace; and Kubernetes ties federated identity, DataAgent and the on-prem VMware stack back into the spine.<\/p>\n<p>              <!-- INTERACTIVE_MAP_LINK_START --><\/p>\n<p style=\"margin:10px 0 0;text-align:center;\"><a href=\"https:\/\/www.cybersecurityinstitute.com\/blog\/?p=5824\" target=\"_blank\" rel=\"noopener\" style=\"display:inline-block;padding:8px 18px;background-color:#0f172a;color:#ffffff !important;text-decoration:none;border-radius:6px;font-size:13px;font-weight:600;\">View interactive topic map &rarr;<\/a><\/p>\n<p><!-- INTERACTIVE_MAP_LINK_END -->\n            <\/div>\n<p>            <!-- Article index --><\/p>\n<h2>Article index<\/h2>\n<p style=\"font-size:13px;color:#6b7280;font-style:italic;margin:0 0 6px 0;\">25 articles, grouped by sub-theme. Twenty are from this week&rsquo;s coverage window (August 31&ndash;September 6); five are longer-form foundational reads on the beat.<\/p>\n<h3>Inference economics: sizing GPUs, TPUs and the on-prem shift<\/h3>\n<div class=\"cluster-intro\">Where the accelerator money actually goes &mdash; a cold start taken apart phase by phase, the memory arithmetic behind GPU selection, reasoning models compressed onto edge hardware, a concurrency wall found by benchmark, and Broadcom&rsquo;s bet that the whole stack comes back on-premises.<\/div>\n<table class=\"index-table\">\n<tr>\n<th>Article<\/th>\n<th>Source<\/th>\n<th>Published<\/th>\n<\/tr>\n<tr>\n<td>1. <a href=\"https:\/\/thenewstack.io\/cut-gpu-cold-starts\/\">Cut GPU inference cold start from 8 minutes to less than a minute<\/a><\/td>\n<td class=\"src\">The New Stack<\/td>\n<td class=\"dt\">Sep 3, 2026<\/td>\n<\/tr>\n<tr>\n<td>2. <a href=\"https:\/\/developer.nvidia.com\/blog\/how-to-size-gpus-for-ai-inference-and-tco-without-overspending\/\">How to Size GPUs for AI Inference and TCO Without Overspending<\/a><\/td>\n<td class=\"src\">NVIDIA Developer Blog<\/td>\n<td class=\"dt\">Sep 1, 2026<\/td>\n<\/tr>\n<tr>\n<td>3. <a href=\"https:\/\/developer.nvidia.com\/blog\/frontier-reasoning-reaches-the-edge-how-to-deploy-and-optimize-models-on-nvidia-jetson\/\">Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson<\/a><\/td>\n<td class=\"src\">NVIDIA Developer Blog<\/td>\n<td class=\"dt\">Sep 4, 2026<\/td>\n<\/tr>\n<tr>\n<td>4. <a href=\"https:\/\/cloud.google.com\/blog\/topics\/developers-practitioners\/not-all-llm-workloads-are-equal-benchmarking-tpu-performance-on-classification-vs-generation\/\">Not All LLM Workloads Are Equal: Benchmarking TPU Performance on Classification vs. Generation<\/a><\/td>\n<td class=\"src\">Google Cloud Blog<\/td>\n<td class=\"dt\">Sep 4, 2026<\/td>\n<\/tr>\n<tr>\n<td>5. <a href=\"https:\/\/www.nextplatform.com\/cloud\/2026\/09\/01\/vmware-intros-private-ai-cloud-ai-factory-as-workloads-shift-to-on-prem\/5293559\">VMware Intros Private AI Cloud, AI Factory As Workloads Shift To On-Prem<\/a><\/td>\n<td class=\"src\">The Next Platform<\/td>\n<td class=\"dt\">Sep 1, 2026<\/td>\n<\/tr>\n<tr>\n<td>6. <a href=\"https:\/\/www.techtarget.com\/it-infrastructure\/news\/366649184\/VMware-wants-to-become-the-enterprise-home-for-AI-Agents\">VMware wants to become the enterprise home for AI agents<\/a><\/td>\n<td class=\"src\">TechTarget<\/td>\n<td class=\"dt\">Sep 2, 2026<\/td>\n<\/tr>\n<\/table>\n<h3>Token economics and LLM cost control<\/h3>\n<div class=\"cluster-intro\">The software half of the same bill: what a production system actually spends per session, how caching changes the arithmetic, and two organisations publishing real dollar figures instead of percentages.<\/div>\n<table class=\"index-table\">\n<tr>\n<th>Article<\/th>\n<th>Source<\/th>\n<th>Published<\/th>\n<\/tr>\n<tr>\n<td>7. <a href=\"https:\/\/thenewstack.io\/production-token-optimization-guide\/\">The systems guide to production token optimization<\/a><\/td>\n<td class=\"src\">The New Stack<\/td>\n<td class=\"dt\">Sep 3, 2026<\/td>\n<\/tr>\n<tr>\n<td>8. <a href=\"https:\/\/www.datadoghq.com\/blog\/monitor-prompt-caching-optimize-token-usage\/\">Monitor prompt caching to optimize your token usage<\/a><\/td>\n<td class=\"src\">Datadog<\/td>\n<td class=\"dt\">Sep 2, 2026<\/td>\n<\/tr>\n<tr>\n<td>9. <a href=\"https:\/\/www.elastic.co\/blog\/token-costs-ai-bills\">Why your AI bill tripled while token prices fell 75%<\/a><\/td>\n<td class=\"src\">Elastic<\/td>\n<td class=\"dt\">Sep 4, 2026<\/td>\n<\/tr>\n<tr>\n<td>10. <a href=\"https:\/\/www.datadoghq.com\/blog\/how-datadog-saves-money-by-optimizing-ai-usage\/\">How Datadog saves over $1 million each month by optimizing AI usage<\/a> <em>(foundational)<\/em><\/td>\n<td class=\"src\">Datadog<\/td>\n<td class=\"dt\">Aug 26, 2026<\/td>\n<\/tr>\n<\/table>\n<h3>Observability practice and telemetry quality<\/h3>\n<div class=\"cluster-intro\">The groundwork underneath every AI cost number in this issue: what a span is made of, how to decide what to keep, how to score the instrumentation you already have, and what SREs report they are now spending their time on.<\/div>\n<table class=\"index-table\">\n<tr>\n<th>Article<\/th>\n<th>Source<\/th>\n<th>Published<\/th>\n<\/tr>\n<tr>\n<td>11. <a href=\"https:\/\/thenewstack.io\/tracing-data-overload-sampling\/\">How to find failures without drowning in tracing data<\/a><\/td>\n<td class=\"src\">The New Stack<\/td>\n<td class=\"dt\">Sep 3, 2026<\/td>\n<\/tr>\n<tr>\n<td>12. <a href=\"https:\/\/devops.com\/observability-2-0-why-devops-teams-are-moving-from-monitoring-to-intelligent-system-understanding\/\">Observability 2.0: Why DevOps Teams Are Moving From Monitoring to Intelligent System Understanding<\/a><\/td>\n<td class=\"src\">DevOps.com<\/td>\n<td class=\"dt\">Sep 3, 2026<\/td>\n<\/tr>\n<tr>\n<td>13. <a href=\"https:\/\/www.dynatrace.com\/news\/blog\/ai-is-changing-the-reliability-game-for-sres\/\">AI is changing the reliability game for SREs<\/a><\/td>\n<td class=\"src\">Dynatrace<\/td>\n<td class=\"dt\">Sep 1, 2026<\/td>\n<\/tr>\n<tr>\n<td>14. <a href=\"https:\/\/www.dynatrace.com\/news\/blog\/opentelemetry-series-anatomy-of-an-otel-span\/\">OpenTelemetry series: Anatomy of an OTel span<\/a> <em>(foundational)<\/em><\/td>\n<td class=\"src\">Dynatrace<\/td>\n<td class=\"dt\">Aug 28, 2026<\/td>\n<\/tr>\n<tr>\n<td>15. <a href=\"https:\/\/grafana.com\/blog\/how-to-measure-and-improve-instrumentation-quality-for-better-full-stack-observability\/\">How to measure and improve instrumentation quality for better full-stack observability<\/a> <em>(foundational)<\/em><\/td>\n<td class=\"src\">Grafana Labs<\/td>\n<td class=\"dt\">Aug 27, 2026<\/td>\n<\/tr>\n<\/table>\n<h3>Agent platforms and agent-native platform engineering<\/h3>\n<div class=\"cluster-intro\">What it takes to run agents as first-class platform users: sandboxes and playbooks at DoorDash scale, workflows built from chat, the trace-to-experiment loop, memory that expires on purpose, persistence economics, retrieval, and golden paths for a user who is not human.<\/div>\n<table class=\"index-table\">\n<tr>\n<th>Article<\/th>\n<th>Source<\/th>\n<th>Published<\/th>\n<\/tr>\n<tr>\n<td>16. <a href=\"https:\/\/www.infoq.com\/news\/2026\/08\/doordash-flux-cloud-agent\/\">DoorDash&rsquo;s Flux Runs 130,000 Engineering Tasks through Cloud-Based Agents<\/a><\/td>\n<td class=\"src\">InfoQ<\/td>\n<td class=\"dt\">Aug 31, 2026<\/td>\n<\/tr>\n<tr>\n<td>17. <a href=\"https:\/\/www.datadoghq.com\/blog\/build-datadog-workflows-ai-agents\/\">Build and run Datadog workflows from Bits Chat or AI agents<\/a><\/td>\n<td class=\"src\">Datadog<\/td>\n<td class=\"dt\">Sep 3, 2026<\/td>\n<\/tr>\n<tr>\n<td>18. <a href=\"https:\/\/www.datadoghq.com\/blog\/from-traces-to-experiments-a-loop-for-improving-ai-agents\/\">From traces to experiments: A loop for improving AI agents<\/a><\/td>\n<td class=\"src\">Datadog<\/td>\n<td class=\"dt\">Sep 1, 2026<\/td>\n<\/tr>\n<tr>\n<td>19. <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/designing-lifecycle-policies-for-agentcore-memory\/\">Designing lifecycle policies for AgentCore memory<\/a><\/td>\n<td class=\"src\">AWS Machine Learning Blog<\/td>\n<td class=\"dt\">Sep 4, 2026<\/td>\n<\/tr>\n<tr>\n<td>20. <a href=\"https:\/\/thenewstack.io\/agent-scale-database-persistence\/\">When agents build, deploy, and maintain, persistence becomes the hard problem<\/a><\/td>\n<td class=\"src\">The New Stack<\/td>\n<td class=\"dt\">Sep 1, 2026<\/td>\n<\/tr>\n<tr>\n<td>21. <a href=\"https:\/\/thenewstack.io\/ai-agent-retrieval-infrastructure\/\">Want to scale AI agents without breaking anything? Retrieval engineering is the answer.<\/a><\/td>\n<td class=\"src\">The New Stack<\/td>\n<td class=\"dt\">Sep 3, 2026<\/td>\n<\/tr>\n<tr>\n<td>22. <a href=\"https:\/\/www.datadoghq.com\/blog\/golden-paths-for-ai-agents\/\">Golden Paths for AI agents: What changes when platform users aren&rsquo;t human?<\/a> <em>(foundational)<\/em><\/td>\n<td class=\"src\">Datadog<\/td>\n<td class=\"dt\">Aug 25, 2026<\/td>\n<\/tr>\n<\/table>\n<h3>Agents in production operations: identity, autonomous remediation, latency<\/h3>\n<div class=\"cluster-intro\">Agents inside the blast radius &mdash; carrying a user&rsquo;s identity across federated clusters, a funded attempt at autonomous fault repair in Kubernetes, and the argument that agentic latency is a wait-state problem more compute cannot fix.<\/div>\n<table class=\"index-table\">\n<tr>\n<th>Article<\/th>\n<th>Source<\/th>\n<th>Published<\/th>\n<\/tr>\n<tr>\n<td>23. <a href=\"https:\/\/developer.nvidia.com\/blog\/how-to-carry-user-identity-across-federated-kubernetes-and-ai-platforms\/\">How to Carry User Identity Across Federated Kubernetes and AI Platforms<\/a><\/td>\n<td class=\"src\">NVIDIA Developer Blog<\/td>\n<td class=\"dt\">Sep 3, 2026<\/td>\n<\/tr>\n<tr>\n<td>24. <a href=\"https:\/\/siliconangle.com\/2026\/09\/01\/dataagent-raises-10m-to-let-ai-fix-production-faults-inside-kubernetes-clusters\/\">DataAgent raises $10M to let AI fix production faults inside Kubernetes clusters<\/a><\/td>\n<td class=\"src\">SiliconANGLE<\/td>\n<td class=\"dt\">Sep 1, 2026<\/td>\n<\/tr>\n<tr>\n<td>25. <a href=\"https:\/\/thenewstack.io\/agentic-ai-latency-infrastructure\/\">Agentic AI has a latency problem that more compute won&rsquo;t solve<\/a> <em>(foundational)<\/em><\/td>\n<td class=\"src\">The New Stack<\/td>\n<td class=\"dt\">Aug 18, 2026<\/td>\n<\/tr>\n<\/table>\n<p>            <!-- Detailed write-ups --><\/p>\n<h2>Detailed write-ups<\/h2>\n<div class=\"article\">\n<h4>1. Eight minutes, six phases: where a GPU cold start actually goes<\/h4>\n<p class=\"meta\">The New Stack &middot; NVIDIA Developer Blog &middot; September 1&ndash;4, 2026<\/p>\n<p>The most useful engineering post of the week is also the most quotable. <strong>Sajjan Gundapuneedi<\/strong>, senior manager of software development at <strong>AWS<\/strong>, measured a GPU inference cold start end to end and reported: &ldquo;Eight minutes. Six sequential phases. We expected one bottleneck. We found six, and which one dominates depends on model size.&rdquo; The optimised path reaches <strong>under 30 seconds on a warm node<\/strong> &mdash; an 80&ndash;93% reduction &mdash; and about <strong>five minutes on a genuinely cold node<\/strong>, down from eight to fifteen. All of it was measured on <strong>p5.48xlarge<\/strong> instances on <strong>Amazon EKS Auto Mode<\/strong>, which matters, because two of the six phases are properties of that environment rather than of your code: node provisioning runs 60&ndash;90 seconds, and GPU driver initialisation takes two to three minutes on a standard setup versus seconds on EKS Auto Mode. Container image pull is the next two to four minutes on a standard path and 30&ndash;60 seconds with <strong>SOCI<\/strong> parallel pull.<\/p>\n<p>The model-size dependency is the part to carry into your own capacity plan. For a <strong>64&nbsp;GB<\/strong> model (Qwen3.6-35B-A3B) weight loading took <strong>29 seconds<\/strong> and <code>torch.compile<\/code> took <strong>53<\/strong>; for a <strong>203&nbsp;GB<\/strong> model (Llama-4-Scout at tensor parallel TP=4) weight loading took <strong>423 seconds<\/strong> and compile only <strong>34<\/strong>. Optimise the wrong phase and you will move a number that was never on your critical path. Loading weights from S3 with a parallel streamer took the 64&nbsp;GB case from 29 seconds to <strong>12<\/strong> and the 203&nbsp;GB case from 423 seconds to <strong>25<\/strong>; a warm <code>torch.compile<\/code> cache brings compilation down to <strong>16&ndash;32 seconds<\/strong> on subsequent pods. The economics behind the urgency are stated plainly &mdash; a p5.48xlarge is <strong>$55 an hour<\/strong> on demand and G-family instances run <strong>$10&ndash;20<\/strong> &mdash; so every minute a pod spends initialising is billed at full accelerator rate for zero served tokens.<\/p>\n<p><strong>NVIDIA<\/strong>&rsquo;s companion piece, by <strong>Prerana Gambhir<\/strong> and <strong>Manasa Manohara<\/strong>, supplies the memory arithmetic that decides which instance you were going to cold-start in the first place. Despite the title it is a sizing and model-optimisation guide rather than a costed TCO model, and it is stronger for it. The headline result: <strong>FP8 quantisation of Llama-3.1-8B cuts weight memory from 16.06&nbsp;GB to 9.08&nbsp;GB &mdash; 43.5% &mdash; with no retraining<\/strong>. Sizing then follows workload shape rather than model name, and the token-shape table is the part worth pinning up: chatbots and copilots run 2,000&ndash;8,000 input and 200&ndash;800 output tokens; agents carry <strong>more than 128,000 cached input tokens<\/strong> but emit only 500&ndash;1,000 input and 200&ndash;300 output; content generation inverts that with 1,000&ndash;4,000 output tokens. The heuristics that fall out are concrete: a <strong>24&nbsp;GB<\/strong> GPU for a 7&ndash;8B model and <strong>48&nbsp;GB<\/strong> for a 13B in a financial-services copilot at 10&ndash;50 concurrent sessions with a sub-one-second time-to-first-token target; <strong>more than 80&nbsp;GB<\/strong> per unit for a life-sciences agent handling 20,000 input tokens at 20&ndash;30 concurrent users under two seconds; 16&ndash;24&nbsp;GB per GPU for media generation at 50&ndash;100-plus users; 8&ndash;16&nbsp;GB entry-level parts for translation at hundreds of concurrent requests. On pruning, <strong>Qwen3-8B<\/strong> reduced to roughly 6B parameters, and width pruning beat depth pruning on validation loss &mdash; <strong>3.21 versus 3.60<\/strong>. The toolchain is pinned: NeMo container <code>nvcr.io\/nvidia\/nemo:25.11<\/code> with <code>nvidia-modelopt==0.37.0<\/code>, alongside Model Optimizer and TensorRT, against H100, A100 80GB and Blackwell parts.<\/p>\n<p>The third piece takes the same compression logic to the edge, and its title oversells slightly in a way worth naming: the reasoning arrives on <strong>Jetson<\/strong> because the models got smaller, not because frontier-scale inference now fits on a dev kit. <strong>Nemotron 3.5 Lightning<\/strong> &mdash; 30B total parameters with 3B activated &mdash; running with DSpark delivered a <strong>3.37x decode speedup at 123.01&ndash;138.02 tokens per second<\/strong>, and <strong>Qwen3.8-27B<\/strong> with DFlash2 delivered <strong>6.28x at 27.69&ndash;34.44 tokens per second<\/strong>; <strong>NVFP4<\/strong> quantisation on its own accounts for 2.2x&ndash;2.33x. The hardware spans Jetson AGX Thor, AGX Orin and Orin Nano, with vLLM v0.28.0 and llama.cpp as the runtimes, and NVIDIA&rsquo;s own caveat is the one to respect: the speedups varied by workload. Read the three together and the week&rsquo;s inference lesson is singular &mdash; the cheapest capacity is the capacity you stopped provisioning, and every result above came from measuring one layer more precisely rather than buying a larger one.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/thenewstack.io\/cut-gpu-cold-starts\/\">The New Stack (cut GPU inference cold start from 8 minutes to less than a minute)<\/a> &middot; <a href=\"https:\/\/developer.nvidia.com\/blog\/how-to-size-gpus-for-ai-inference-and-tco-without-overspending\/\">NVIDIA (how to size GPUs for AI inference)<\/a> &middot; <a href=\"https:\/\/developer.nvidia.com\/blog\/frontier-reasoning-reaches-the-edge-how-to-deploy-and-optimize-models-on-nvidia-jetson\/\">NVIDIA (reasoning models on Jetson)<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>2. The concurrency wall: a TPU benchmark that changes an autoscaling policy<\/h4>\n<p class=\"meta\">Google Cloud Blog &middot; The New Stack &middot; August 18 &ndash; September 4, 2026<\/p>\n<p><strong>Rupjit Chakraborty<\/strong> of Google Cloud ran the benchmark most teams assume someone else already ran, and the result contradicts a common default. On <strong>TPU v6e<\/strong> in a 2&#215;2 single-host chip topology on <strong>GKE Autopilot<\/strong>, serving <strong>Gemma 3 12B and 27B<\/strong> through <strong>vLLM<\/strong> via the <code>vllm-project\/tpu-inference<\/code> stack, he separated two workload shapes that platform teams routinely treat as one: classification, roughly 4,000 input tokens to about 10 output tokens, and generation, 500 input tokens to about 1,000 output. Normalised to 1.00x at 16 concurrent users, generation on the <strong>12B<\/strong> model scales 1.98x at 32 users, 2.96x at 64 and <strong>8.19x at 128<\/strong>. The <strong>27B<\/strong> model does not: &ldquo;For decode-heavy generation tasks, the Gemma 3 27B model hits a strict performance wall past 64 concurrent users, plateauing at a 4.12x normalized throughput multiplier at 128 users.&rdquo;<\/p>\n<p>On prefill-heavy classification the two sizes behave far more alike &mdash; <strong>6.37x<\/strong> for the 12B model at 128 users against <strong>6.04x<\/strong> for the 27B &mdash; which is the finding with operational teeth. The penalty for over-sizing your model is small when the workload is dominated by reading input and large when it is dominated by producing output, so a single model choice applied across a mixed workload is systematically wrong in one direction. The serving configuration is published (<code>max-model-len=128000<\/code>, <code>max-num-batched-tokens=8192<\/code>, <code>max-num-seqs=512<\/code>), and four recommendations fall out of it: downsize to the 12B model for high-concurrency generation; cap concurrent requests at <strong>64 per replica<\/strong> in pod autoscaling; autoscale on end-to-end latency rather than CPU or memory utilisation; and tune <code>VLLM_TPU_BUCKET_PADDING_GAP<\/code>. If your HPA is currently watching accelerator utilisation, that third point is a config change worth making this week &mdash; utilisation stays high while the thing your users feel has already collapsed.<\/p>\n<p>The foundational read alongside it explains why the ceiling arrives sooner than a throughput chart suggests. <strong>Akamai<\/strong>&rsquo;s survey of 200 AI practitioners found <strong>82%<\/strong> of organisations require end-to-end responses under <strong>500&nbsp;ms<\/strong> for critical use cases and <strong>64%<\/strong> need under <strong>250&nbsp;ms<\/strong> for their most important one &mdash; while half of enterprise AI deployments miss their latency targets at peak load. The number that reframes the problem is where the time goes: <strong>CPU-side processing accounts for up to 90.6% of total latency<\/strong> in agentic workloads. Jon Alexander, SVP of product for Akamai&rsquo;s cloud technology group, states the consequence directly &mdash; &ldquo;More GPU capacity does nothing for this. You can&rsquo;t brute-force your way out of a wait state.&rdquo; With LangChain&rsquo;s 2026 survey of more than 1,300 professionals reporting <strong>57.3%<\/strong> of organisations now running agents in production, up from 51% the prior year, the population exposed to that wait state is growing faster than the hardware answer to it. Put the two pieces together and the practical instruction is the same one: instrument the orchestration path, not just the accelerator, then set your concurrency ceiling from a latency measurement rather than a utilisation graph.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/cloud.google.com\/blog\/topics\/developers-practitioners\/not-all-llm-workloads-are-equal-benchmarking-tpu-performance-on-classification-vs-generation\/\">Google Cloud (benchmarking TPU performance on classification vs. generation)<\/a> &middot; <a href=\"https:\/\/thenewstack.io\/agentic-ai-latency-infrastructure\/\">The New Stack (agentic AI has a latency problem that more compute won&rsquo;t solve)<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>3. The token bill: caching, cascading, and two organisations showing their books<\/h4>\n<p class=\"meta\">The New Stack &middot; Datadog &middot; Elastic &middot; August 26 &ndash; September 4, 2026<\/p>\n<p>The systems guide to production token optimisation, by <strong>Boris Chabeda<\/strong>, is the reference read behind every cost story in this issue, and its framing is the correct one: &ldquo;Token optimization is not accounting; it&rsquo;s a distributed systems challenge.&rdquo; It works from a stated baseline &mdash; about <strong>$3 per million input tokens<\/strong> and <strong>$15 per million output<\/strong> for a mid-tier frontier model, so output runs four to five times the price of input &mdash; and two worked reference systems rather than adjectives. &ldquo;Concierge&rdquo;, a 10-turn customer-support chat, burns <strong>45,300 tokens per session<\/strong>; &ldquo;Pathfinder&rdquo;, a 15-step CI-debugging agent, burns <strong>150,000 tokens per session<\/strong> and <strong>570,000<\/strong> in a 30-step worst case. The interventions are measured against those. Rewriting Concierge&rsquo;s system prompt took it from <strong>3,100 to 380 tokens<\/strong>, a 60% overall reduction; compressing incoming tool observations cut Pathfinder&rsquo;s inbound volume <strong>3x<\/strong>; schema enforcement dropped malformed model outputs below <strong>0.5%<\/strong>. Prompt caching, priced at a <strong>90% discount on cache reads<\/strong>, cut input cost roughly <strong>70%<\/strong> over ten turns on Concierge and <strong>76%<\/strong> on Pathfinder. Semantic caching found <strong>34%<\/strong> of Concierge tickets were duplicates and served them in <strong>sub-50&nbsp;ms<\/strong>. Model cascading moved <strong>70%<\/strong> of Concierge chats to a cheaper model, and hybrid orchestration cut Pathfinder&rsquo;s token cost by more than half.<\/p>\n<p><strong>Datadog<\/strong>&rsquo;s <strong>Thomas Sobolik<\/strong> supplies the operational detail for the single biggest of those levers, and opens with the statistic that justifies it: as of <strong>March 2026, 69% of all input tokens<\/strong> observed in Datadog customer traces were system prompts. That is the most cacheable payload in the stack, and the pricing mechanics decide whether caching pays. Anthropic charges cache writes at <strong>1.25x<\/strong> the base rate at a five-minute TTL and <strong>2x<\/strong> at one hour, with reads at <strong>0.1x<\/strong>, and supports up to <strong>four explicit cache breakpoints<\/strong> ordered tools, then system, then messages &mdash; so a breakpoint placed after volatile content silently invalidates everything downstream of it. OpenAI caches automatically at prompts of <strong>1,024 tokens or more<\/strong>, with newer models accepting an explicit <code>prompt_cache_breakpoint<\/code> per block. The reason to monitor rather than assume is that cache hit rate is a function of prompt stability, and prompt stability is a thing your own engineers change weekly without telling anyone.<\/p>\n<p><strong>Elastic<\/strong>&rsquo;s <strong>Sunile Manjee<\/strong> explains the paradox in the headline &mdash; token prices fell <strong>75%<\/strong> over the past year and bills went up anyway &mdash; and his diagnosis is structural: &ldquo;Agentic AI costs are fundamentally a control problem rather than a token pricing problem.&rdquo; Agentic workflows consume <strong>5&ndash;30x more tokens<\/strong> than a comparable chatbot task, and a 2026 audit of <strong>30 production engineering teams<\/strong> attributed <strong>62% of the total inference bill to context and lack of guidance<\/strong> rather than to model price. Two supporting numbers make the variance vivid: token cost varies by up to <strong>30x between identical runs of the same task<\/strong>, and the Stanford Digital Economy Lab figure he cites has agentic coding costing up to <strong>1,000x<\/strong> more tokens than a simple code-chat exchange. The organisational context is just as unhelpful &mdash; <strong>81%<\/strong> of enterprises run three or more models and <strong>88%<\/strong> use two or more providers, only <strong>23%<\/strong> of AI projects launched in the past year were fully deployed and meeting their original ROI objectives, and while <strong>85%<\/strong> of enterprise IT organisations plan to implement LLM observability, just <strong>8%<\/strong> have completed it. A 30x spread between identical runs is not a pricing problem you can negotiate; it is an instrumentation gap.<\/p>\n<p>Which makes <strong>Datadog<\/strong>&rsquo;s own ledger the most valuable foundational read on the list, because it shows what closing that gap returns. Datadog cut its monthly AI spend by <strong>more than $1 million<\/strong>, and the largest single line is a routing decision: moving internal coding work from <strong>Claude Opus to Claude Sonnet<\/strong> saved <strong>$687,000 a month<\/strong> at a measured <strong>8% loss in proficiency<\/strong> for a <strong>36.7% cost reduction<\/strong>. Dropping the Claude Code CLI effort level from high to medium saved another <strong>$288,000 a month<\/strong>. A cost-alert rollout returned <strong>more than $150,000 in its first seven days<\/strong>, with <strong>768 distinct users<\/strong> triggering alerts in that week &mdash; evidence that most overspend is unwitting rather than deliberate. And <strong>Headroom<\/strong>, Datadog&rsquo;s open-source context-optimisation tool, piloted with more than <strong>1,000 engineers<\/strong> over a one-week snapshot, cut input tokens <strong>39.3%<\/strong>, output tokens <strong>35.7%<\/strong>, tool-result size <strong>15.4%<\/strong> and cost per user <strong>27.0%<\/strong>, with an internal evaluation showing a <strong>47%<\/strong> cost reduction; the team ran more than <strong>140 different evaluations<\/strong> on its agent evaluation platform to keep those trades honest. The team&rsquo;s own summary is the transferable part: &ldquo;Similar to rightsizing cloud infrastructure, tuning the configurations of AI tools &mdash; such as model type and effort level &mdash; can yield cost savings without hurting performance objectives.&rdquo; The 8% proficiency figure is what makes it a decision rather than a slogan; without a measured quality delta, a model downgrade is a guess.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/thenewstack.io\/production-token-optimization-guide\/\">The New Stack (the systems guide to production token optimization)<\/a> &middot; <a href=\"https:\/\/www.datadoghq.com\/blog\/monitor-prompt-caching-optimize-token-usage\/\">Datadog (monitor prompt caching to optimize token usage)<\/a> &middot; <a href=\"https:\/\/www.elastic.co\/blog\/token-costs-ai-bills\">Elastic (why your AI bill tripled while token prices fell 75%)<\/a> &middot; <a href=\"https:\/\/www.datadoghq.com\/blog\/how-datadog-saves-money-by-optimizing-ai-usage\/\">Datadog (how Datadog saves over $1 million each month)<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>4. Broadcom builds the on-prem AI estate, and gives agents an identity layer<\/h4>\n<p class=\"meta\">The Next Platform &middot; TechTarget &middot; September 1&ndash;2, 2026<\/p>\n<p>At VMware Explore 2026 on <strong>2 September<\/strong>, <strong>Broadcom<\/strong> made the largest structural bet of the week: that production inference is moving back inside the data centre and that VMware should be where it lands. <strong>VMware Private AI Cloud<\/strong> and <strong>VMware AI Factory<\/strong> arrive on <strong>VMware Cloud Foundation 9<\/strong>, alongside VMware AI Assistant, Tanzu, vDefend, the Avi Load Balancer and TrueSource. AI Factory is the interesting half operationally &mdash; validated metal-to-model stacks built with <strong>Dell Technologies, Cisco Systems, Lenovo, Supermicro and AMD<\/strong>, using <strong>MetalSoft<\/strong> for bare-metal orchestration, with <strong>AMD Instinct MI350 Series<\/strong> GPUs and the ROCm software platform supported. Models named as served on the platform include <strong>NVIDIA Nemotron 3<\/strong>, <strong>Google DeepMind Gemma 4<\/strong> and <strong>Alibaba Qwen 3.7-Max<\/strong>. Prashanth Shenoy, VP of marketing for Broadcom&rsquo;s VMware Cloud Foundation Division, gives the thesis in a sentence: &ldquo;They want to bring data to the model and not model to the data.&rdquo;<\/p>\n<p>The repatriation figures Broadcom cites are the argument underneath it: <strong>56%<\/strong> of IT decision-makers are running or planning production inference in private cloud, <strong>62%<\/strong> are concerned about costs, <strong>51%<\/strong> are repatriating AI workloads over security concerns, and public-cloud share for these workloads has fallen <strong>15% year over year to 41%<\/strong>. One hardware detail is worth pulling out for anyone sizing a private estate: <strong>NVMe memory tiering is said to reduce per-host costs by 42%<\/strong> &mdash; the same lever that makes an on-prem inference host cheaper than its GPU line item suggests. The market framing is the usual vertigo &mdash; global datacenter investment projected at <strong>$1.6 trillion by 2030<\/strong>, with tech enterprises spending <strong>$600 billion<\/strong> on AI infrastructure this year.<\/p>\n<p>The complementary report puts a version history against the announcements, which is what a platform team actually needs. Seven items shipped: generally available now are <strong>Private AI Cloud on VCF<\/strong>, <strong>TrueSource<\/strong>, and the new <strong>AgentMinder<\/strong> &mdash; agent identity, runtime policy and observability in one component, which is the piece to read twice, because it is an infrastructure vendor claiming the control plane for non-human principals. <strong>Model-as-a-Service on VCF<\/strong> serves <strong>more than 150 open models<\/strong> on a <strong>vLLM<\/strong> runtime, including Nemotron 3, Gemma 4, Qwen3.8-27B, GLM-5.2 and NEC cotomi. <strong>AI-Ready Data Foundations<\/strong> in Tanzu Platform is slated for the fall 2026 release rather than shipping now. The dated lineage matters when you are judging maturity: NVMe tiering landed in VCF 9.0 in June 2025, GPU metrics in VCF 9.1 in May 2026, and agent buildpacks moved from an April 2026 tech preview to fully supported. Omdia principal analyst <strong>Torsten Volk<\/strong>&rsquo;s read is that &ldquo;these seven announcements answer the Omdia survey almost line by line&rdquo; &mdash; the survey in question finding <strong>69%<\/strong> of organisations currently seeking virtualization platform alternatives, which is the commercial pressure this release is answering as much as it is answering AI demand. If you are one of that 69%, the useful question is not whether the AI stack is credible but whether adopting it deepens exactly the dependency you were trying to reduce.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/www.nextplatform.com\/cloud\/2026\/09\/01\/vmware-intros-private-ai-cloud-ai-factory-as-workloads-shift-to-on-prem\/5293559\">The Next Platform (VMware intros Private AI Cloud, AI Factory)<\/a> &middot; <a href=\"https:\/\/www.techtarget.com\/it-infrastructure\/news\/366649184\/VMware-wants-to-become-the-enterprise-home-for-AI-Agents\">TechTarget (VMware wants to become the enterprise home for AI agents)<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>5. Agent-native platform engineering: 130,000 tasks a month, and a user who is not human<\/h4>\n<p class=\"meta\">InfoQ &middot; Datadog &middot; August 25 &ndash; September 3, 2026<\/p>\n<p><strong>DoorDash<\/strong>&rsquo;s <strong>Flux<\/strong> platform is the most concrete evidence yet of what agent-native platform engineering looks like once it is past pilot scale, and the numbers, reported by <strong>Leela Kumili<\/strong>, are operational rather than aspirational: <strong>130,000 engineering tasks<\/strong> run through cloud-based agents in a single month in 2026, <strong>more than 25,000 automated code reviews weekly<\/strong>, and <strong>more than 300 YAML-based playbooks<\/strong> drawing <strong>more than 10,000 weekly invocations<\/strong>. The engineering choices behind those figures are the transferable part. Agent sandboxes run on <strong>Firecracker<\/strong> micro virtual machines rather than shared containers, tool access is mediated through an <strong>MCP gateway<\/strong> (Agent Gateway) rather than granting agents direct credentials, and the platform holds a <strong>95th-percentile SLO of under five seconds<\/strong> for end-to-end sandbox setup &mdash; an SLO on the platform&rsquo;s responsiveness to agents, which is a telling thing for a platform team to have written down. The comment quoted in the piece from Duy Nguyen of TOP GROUP Vietnam names the shift precisely: &ldquo;Once agents handle background work, code review, internal tools, and concurrent execution, the hard problem shifts from model choice to control.&rdquo; Three hundred playbooks is a governance surface, and 10,000 weekly invocations is a change-management problem wearing an automation costume.<\/p>\n<p><strong>Datadog<\/strong> shipped the small end of the same idea: workflows that can now be built and run from conversation rather than from the workflow editor. The paths are specific &mdash; <strong>Bits Chat<\/strong>; external coding agents such as Claude Code, Cursor and Codex working through the <strong>Datadog MCP Server<\/strong>; custom agents built in <strong>Bits Agent Builder<\/strong>; and the Datadog Slack app. That is narrower than &ldquo;from AI agents&rdquo; sounds, and the narrowness is the point: each path is a named, brokered integration rather than an open endpoint, which is the same architectural instinct as DoorDash&rsquo;s gateway. The operational question to answer before enabling it is not whether an agent can author a workflow, but who reviews a workflow an agent authored, and against which runbook.<\/p>\n<p>The foundational essay that frames all of it comes from Datadog&rsquo;s platform-engineering side, by <strong>Candace Shamieh<\/strong>, <strong>Shlomo Benyaminov<\/strong> and <strong>James Eastham<\/strong>, and asks the right question: what does a golden path mean when the platform&rsquo;s user is not human? Their anchor is Gartner&rsquo;s 2026 Hype Cycle for Agentic AI introducing <strong>&ldquo;agent experience (AX)&rdquo;<\/strong> as a formal persona &mdash; which sounds like analyst vocabulary until you notice that every affordance a golden path offers a human (discoverable defaults, guardrails, a paved road that is easier than the alternative) has to be re-expressed as something a machine consumes: a schema, a tool description, a permitted action set. Their one hard number is a reminder that this is an efficiency problem as much as a governance one &mdash; a roughly <strong>40% cost reduction<\/strong> in specific evaluation scenarios using field-selection tools, which is to say that telling an agent precisely which fields it needs is worth about as much as any model-level optimisation in this issue. If you run an internal developer platform, the cheapest useful exercise this quarter is to take your three most-used golden paths and ask what an agent would have to guess in order to use them.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/www.infoq.com\/news\/2026\/08\/doordash-flux-cloud-agent\/\">InfoQ (DoorDash&rsquo;s Flux runs 130,000 engineering tasks through cloud-based agents)<\/a> &middot; <a href=\"https:\/\/www.datadoghq.com\/blog\/build-datadog-workflows-ai-agents\/\">Datadog (build and run Datadog workflows from Bits Chat or AI agents)<\/a> &middot; <a href=\"https:\/\/www.datadoghq.com\/blog\/golden-paths-for-ai-agents\/\">Datadog (Golden Paths for AI agents)<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>6. What agents remember, what it costs to keep it, and how you improve them<\/h4>\n<p class=\"meta\">AWS Machine Learning Blog &middot; The New Stack &middot; Datadog &middot; September 1&ndash;4, 2026<\/p>\n<p>The best-specified piece in this cluster is <strong>AWS<\/strong>&rsquo;s guide to lifecycle policies for <strong>AgentCore memory<\/strong>, by <strong>Akarsha Sehwag<\/strong>, <strong>Himanshu Sah<\/strong> and <strong>Nicolo Cosimo Albanese<\/strong>, because it converts &ldquo;agents should have memory&rdquo; into defaults you can argue with. Episodic memory gets a <strong>90-day TTL<\/strong>; semantic memory <strong>six to twelve months<\/strong>; the default prune window is <strong>45 days<\/strong>; consolidation runs in batches of <strong>10 memories<\/strong>; the relevance threshold sits at <strong>0.3<\/strong> on a 0.0&ndash;1.0 scale; and the scoring weights are <strong>recency 0.4, access 0.35, frequency 0.25<\/strong>. The whole thing runs as a nightly <strong>AWS Step Functions<\/strong> workflow, and the cost illustration is refreshingly small: roughly <strong>$0.01&ndash;$0.02 per nightly run<\/strong> for an agent with 1,000 memories where 20% score below threshold, rising to roughly <strong>$50&ndash;$100 per month at 100,000 memories<\/strong>. Those are not frightening numbers, which is exactly why they need a policy &mdash; unmanaged memory does not fail loudly, it accretes, and the retrieval quality degrades before the invoice does. Two of those defaults deserve local scrutiny: a 0.3 relevance threshold is a product decision about how much an agent is allowed to forget, and a 90-day episodic TTL is a compliance decision in any regulated context.<\/p>\n<p>The persistence argument runs alongside it from a vendor with a horse in the race. <strong>Max Liu<\/strong>, co-founder and CEO of TiDB (PingCAP), makes the case that per-tenant databases are the hidden cost of agent-built software: when agents build, deploy and maintain applications, the number of database instances grows with tenants rather than with load, and the <strong>idle-cost trap<\/strong> &mdash; paying for provisioned capacity that mostly sits doing nothing &mdash; becomes the dominant line item. His production example is <strong>Kimi by Moonshot AI<\/strong>, and his architectural target is roughly <strong>one-second database provisioning<\/strong>, on the reasoning that if creating a database is instant and idle capacity costs nothing, per-tenant isolation stops being an economic problem. It is a vendor&rsquo;s case for its own architecture, and it is also a real constraint that anyone letting agents spin up backing services will meet.<\/p>\n<p><strong>Datadog<\/strong>&rsquo;s trace-to-experiment piece supplies the improvement loop, and it is a methodology essay rather than a launch &mdash; read it that way and it is genuinely useful. The argument is that production traces are the highest-quality source of evaluation cases you will ever have, and that most teams waste them: an agent thread that went wrong is a test case with ground truth attached, but only if you capture it, generalise it into a dataset, run a changed prompt or model against it, and compare. That is a loop, not a dashboard, and the discipline it demands is version control over prompts and datasets rather than any particular tool. The final piece in this cluster is forward-looking rather than reported: <strong>Vespa.ai<\/strong> is running a webinar on <strong>24 September<\/strong> with GigaOm field CTO <strong>Whit Walters<\/strong> and Vespa&rsquo;s <strong>Bonnie Chase<\/strong>, framed around Walters&rsquo; report &ldquo;Defeating the Integration Tax&rdquo; and the case that retrieval infrastructure is what determines whether agent deployments scale. Worth a diary entry if retrieval is on your roadmap this half.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/aws.amazon.com\/blogs\/machine-learning\/designing-lifecycle-policies-for-agentcore-memory\/\">AWS (designing lifecycle policies for AgentCore memory)<\/a> &middot; <a href=\"https:\/\/thenewstack.io\/agent-scale-database-persistence\/\">The New Stack (when agents build, deploy, and maintain, persistence becomes the hard problem)<\/a> &middot; <a href=\"https:\/\/www.datadoghq.com\/blog\/from-traces-to-experiments-a-loop-for-improving-ai-agents\/\">Datadog (from traces to experiments)<\/a> &middot; <a href=\"https:\/\/thenewstack.io\/ai-agent-retrieval-infrastructure\/\">The New Stack (retrieval engineering for scaling AI agents)<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>7. Observability practice: span anatomy, sampling strategy, and instrumentation you can score<\/h4>\n<p class=\"meta\">Dynatrace &middot; Grafana Labs &middot; The New Stack &middot; DevOps.com &middot; August 27 &ndash; September 3, 2026<\/p>\n<p>Every cost number in this issue depends on telemetry that is actually complete, and this week produced the groundwork for that. <strong>Amrith Raj<\/strong>, principal AI telemetry specialist at Dynatrace, wrote the clearest available explanation of what a span is made of &mdash; ten components, the span kinds, and the semantic conventions that make a span comparable across services, including the HTTP and <strong>GenAI<\/strong> operation conventions that matter most for anyone tracing model calls. It is grounded in the real CNCF project, referencing <strong>OpenTelemetry schema 1.26.0<\/strong> and <strong>Java SDK 1.38.0<\/strong>, and the explanation is vendor-neutral even though the screenshots are Dynatrace Distributed Tracing and Playground. The reason to send it to your team is the semantic-conventions half: an attribute named by convention is queryable across every service that emits it, and an attribute named by a developer&rsquo;s preference is a per-service dialect that quietly makes cross-service analysis impossible.<\/p>\n<p><strong>Grafana Labs<\/strong> turned the same concern into something measurable. <strong>Arpit Kumar<\/strong> describes the <strong>instrumentation quality report<\/strong> in <strong>Grafana Cloud Knowledge Graph<\/strong>, which grades services across <strong>five quality tiers<\/strong>, from Incomplete\/Poor at 0&ndash;10% up to Perfect at 100%, and surfaces the result under Entity catalog &gt; Instrumentation quality. The checks are the useful specification even if you never use the product: service graph metrics present, Kubernetes labels applied, service names validated, span metrics emitted. Those four are precisely the things that break silently and are discovered during an incident, and putting a score on them turns instrumentation from a task nobody owns into a number a team can be asked about in a review.<\/p>\n<p>On the question of what to keep, <strong>The New Stack<\/strong> ran a short conversation with <strong>Sarah Hudspeth<\/strong> of Chronosphere laying out the three strategies teams actually choose between &mdash; head sampling, tail sampling and dynamic sampling &mdash; which is the right frame for the decision even at podcast length. The trade is structural rather than numeric: head sampling is cheap and decides before it knows whether the trace was interesting; tail sampling knows the outcome but must buffer to find out; dynamic sampling adjusts rates to what is happening. Set against Grafana&rsquo;s scoring work, the sequencing advice writes itself &mdash; measure instrumentation completeness before you tune sampling rates, because sampling a badly instrumented service just makes the gap harder to see. <strong>DevOps.com<\/strong>&rsquo;s contribution, by <strong>Sannan Ali<\/strong>, argues the cultural version of the same point: that teams are moving from monitoring toward understanding, and that the shift comes from linking logs, metrics and traces together and reading them alongside deployment context rather than from adding another dashboard. It is a mindset piece rather than an architecture one, and its value is as a way to explain to non-specialists why the correlated view is worth paying for.<\/p>\n<p>What SREs report they are actually doing closes the loop. <strong>Dynatrace<\/strong>&rsquo;s State of SRE 2026 research, written up by <strong>Klint Finley<\/strong>, finds <strong>67%<\/strong> of SRE respondents now monitor AI models and <strong>58%<\/strong> say monitoring AI performance is among their most-used capabilities &mdash; a profession redefined around a workload most job descriptions still do not mention. On service levels the picture is split: <strong>55%<\/strong> report wide SLO use across the organisation while <strong>nearly 90%<\/strong> use SLOs in at least some systems, which is the classic shape of a practice that has been adopted where it was easy and stalled where it was contested. Treat these as Dynatrace&rsquo;s research findings rather than settled industry fact, and use them the way survey data is genuinely useful &mdash; as a prompt to ask which of your own AI services currently has an SLO that someone would be paged against.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/www.dynatrace.com\/news\/blog\/opentelemetry-series-anatomy-of-an-otel-span\/\">Dynatrace (anatomy of an OTel span)<\/a> &middot; <a href=\"https:\/\/grafana.com\/blog\/how-to-measure-and-improve-instrumentation-quality-for-better-full-stack-observability\/\">Grafana Labs (measure and improve instrumentation quality)<\/a> &middot; <a href=\"https:\/\/thenewstack.io\/tracing-data-overload-sampling\/\">The New Stack (finding failures without drowning in tracing data)<\/a> &middot; <a href=\"https:\/\/devops.com\/observability-2-0-why-devops-teams-are-moving-from-monitoring-to-intelligent-system-understanding\/\">DevOps.com (from monitoring to intelligent system understanding)<\/a> &middot; <a href=\"https:\/\/www.dynatrace.com\/news\/blog\/ai-is-changing-the-reliability-game-for-sres\/\">Dynatrace (AI is changing the reliability game for SREs)<\/a><\/p>\n<\/p><\/div>\n<div class=\"article\">\n<h4>8. Agents inside the blast radius: identity across clusters, remediation within them<\/h4>\n<p class=\"meta\">NVIDIA Developer Blog &middot; SiliconANGLE &middot; September 1&ndash;3, 2026<\/p>\n<p><strong>NVIDIA<\/strong>&rsquo;s <strong>Bhagat Khemchandani<\/strong> and <strong>Rohan Somvanshi<\/strong> published the least glamorous and most immediately reusable pattern of the week: carrying a user&rsquo;s identity across federated Kubernetes clusters and AI platforms without making them log in again at every hop. The construction is deliberately boring &mdash; <strong>OpenID Connect<\/strong> for authentication, a <strong>shared Redis session store<\/strong>, stateless data-plane gateways so any gateway can serve any request, and a small identity-validation API sitting between them &mdash; and the single reported result is a <strong>55% reduction in repeated login events<\/strong> across NVIDIA internal developer platforms spanning Kubernetes clusters in <strong>AWS and OCI<\/strong>. The reason it belongs in an AI-ops bulletin rather than an identity one is what sits on the other end of those hops: when an agent or a notebook calls a model endpoint in another cluster, the question of <em>whose<\/em> identity is being carried decides whether your audit trail records a person or a service account. Stateless gateways plus a shared session store is the shape that makes the answer &ldquo;the person&rdquo;.<\/p>\n<p>At the other end of the autonomy spectrum, <strong>DataAgent Ltd.<\/strong> launched on <strong>1 September<\/strong> with <strong>$10 million<\/strong> in pre-seed funding from <strong>MizMaa Ventures<\/strong> and <strong>Alicorn Venture Partners<\/strong>, aiming an agent at fixing production faults inside Kubernetes clusters rather than merely reporting them. The founders carry relevant history: CEO <strong>Ishay Yaari<\/strong> and CTO <strong>Nati Shalom<\/strong> both came from Cloudify Platform, acquired by Dell Technologies in 2023 for about $100 million. The company was founded in <strong>January 2026<\/strong> and is based in Israel, so this is a launch and a thesis rather than a track record. The thesis is the interesting part, and Yaari states it sharply: &ldquo;When a vendor&rsquo;s revenue is your data ingest, it cannot cut your bill without cutting its own.&rdquo; The commercial model follows from it &mdash; the agent itself is open source and free, with a paid SaaS tier for fleet management and orchestration &mdash; and the company says its approach can deliver up to a <strong>90% reduction in observability costs<\/strong>, a figure it has published on its own behalf and which no independent evaluation yet supports. The structural argument about misaligned incentives in ingest-priced observability is worth taking seriously regardless of whether this particular company is the answer; the autonomous-remediation claim is the one to pilot in a non-production cluster and measure yourself.<\/p>\n<p style=\"font-size:13px;color:#6b7280;margin:0;\">Sources: <a href=\"https:\/\/developer.nvidia.com\/blog\/how-to-carry-user-identity-across-federated-kubernetes-and-ai-platforms\/\">NVIDIA (carrying user identity across federated Kubernetes and AI platforms)<\/a> &middot; <a href=\"https:\/\/siliconangle.com\/2026\/09\/01\/dataagent-raises-10m-to-let-ai-fix-production-faults-inside-kubernetes-clusters\/\">SiliconANGLE (DataAgent raises $10M)<\/a><\/p>\n<\/p><\/div>\n<p>            <!-- Calls to action --><\/p>\n<div class=\"watchlist\">\n<h2>Calls to action<\/h2>\n<ul>\n<li><strong>Measure your own cold start, phase by phase.<\/strong> AWS found six sequential phases in eight minutes, and which one dominates depends on model size &mdash; 53 seconds of compile on a 64&nbsp;GB model, 423 seconds of weight loading on a 203&nbsp;GB one. Time node provisioning, driver init, image pull, weight load and compile separately before you optimise any of them, because at $55 an hour for a p5.48xlarge you are paying full accelerator rate for every one of those minutes.<\/li>\n<li><strong>Set your concurrency ceiling from a latency measurement, not a utilisation graph.<\/strong> Google Cloud&rsquo;s TPU benchmark caps concurrent requests at 64 per replica and autoscales on end-to-end latency rather than CPU or memory. If your HPA currently watches accelerator utilisation, it will keep reporting healthy while the experience your users have has already collapsed.<\/li>\n<li><strong>Split your workloads into prefill-heavy and decode-heavy before choosing a model size.<\/strong> On classification, Gemma 3 12B and 27B scale almost alike (6.37x vs 6.04x at 128 users). On generation, the 27B model plateaus at 4.12x while the 12B reaches 8.19x. One model applied across a mixed workload is systematically wrong in one direction.<\/li>\n<li><strong>Audit what fraction of your input tokens is system prompt.<\/strong> Datadog observed 69% across customer traces as of March 2026. That is the most cacheable payload you have &mdash; and with cache reads at a tenth of base rate, the difference between a stable prompt prefix and one your engineers edit weekly is most of your input bill.<\/li>\n<li><strong>Put a measured quality delta next to every model downgrade.<\/strong> Datadog&rsquo;s Opus-to-Sonnet move saved $687,000 a month at a measured 8% proficiency loss for a 36.7% cost reduction. Without that second number a downgrade is a guess, and the first team to notice will be the one whose output got worse.<\/li>\n<li><strong>Give agent memory a written lifecycle policy.<\/strong> Start from AWS&rsquo;s defaults &mdash; 90-day episodic TTL, six to twelve months semantic, a 45-day prune window, relevance threshold 0.3 &mdash; then decide locally, because the threshold is a product decision about what an agent may forget and the TTL is a compliance decision in any regulated context.<\/li>\n<li><strong>Score your instrumentation before you tune sampling.<\/strong> Grafana&rsquo;s five-tier report checks service graph metrics, Kubernetes labels, service name validation and span metrics &mdash; the four things that break silently and get discovered mid-incident. Sampling a badly instrumented service only makes the gap harder to see.<\/li>\n<li><strong>Broker agent access rather than granting it.<\/strong> DoorDash routes tool access through an MCP gateway and runs sandboxes on Firecracker microVMs; Datadog exposes workflow authoring through named, brokered paths. The pattern is the same: a mediated integration you can revoke and audit, not credentials handed to an agent.<\/li>\n<\/ul><\/div>\n<\/td>\n<\/tr>\n<p>        <!-- Footer --><\/p>\n<tr>\n<td class=\"footer\">\n<p class=\"brand\">AI Ops<\/p>\n<p>A weekly intelligence bulletin from Security Radar LLC.<br \/>\n            Curated by Paul Davis &middot; <a href=\"mailto:paul.davis@security-radar.com\">paul.davis@security-radar.com<\/a><\/p>\n<p>&copy; 2026 Security Radar LLC. All rights reserved.<\/p>\n<p>Article titles and summaries are excerpted for review and commentary; all linked articles remain the copyright of their respective publishers and authors.<\/p>\n<p>*|LIST:ADDRESS|*<\/p>\n<p><a href=\"*|ARCHIVE|*\">View this email in your browser<\/a> &middot; <a href=\"*|UNSUB|*\">Unsubscribe<\/a><\/p>\n<\/td>\n<\/tr>\n<\/table>\n<\/td>\n<\/tr>\n<\/table>\n","protected":false},"excerpt":{"rendered":"<p>September 6, 2026 &middot; Weekly Edition AI Ops A week of measurements rather than announcements. AWS took a GPU cold start apart phase by phase and got eight minutes down to under thirty seconds on a warm node; Google Cloud benchmarked Gemma 3 on TPU v6e and found a hard&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[48],"tags":[],"class_list":["post-5825","post","type-post","status-publish","format-standard","hentry","category-ai-ops"],"_links":{"self":[{"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5825","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5825"}],"version-history":[{"count":1,"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5825\/revisions"}],"predecessor-version":[{"id":5852,"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5825\/revisions\/5852"}],"wp:attachment":[{"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5825"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5825"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cybersecurityinstitute.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5825"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}