Table of contents
In this article
In this article
The Ales Analytics blog
Practical reading on building AI systems that last: what makes them work, how to get your data ready, and how to plan what they cost to run. Every figure links to the study it came from.
Latest posts
- 01What makes enterprise AI workThe systems that last are the ones built for the data and the workflow they serve.What separates AI systems that reach production from those that stall, and why fitting the system to the workflow makes the difference.Free to read. Every figure links to the study it came from.
- 02Getting your data ready for AISearching documents is one tool among several; a well-mapped data layer lets the system pick the right one.How to build a data layer your systems can rely on: where every fact lives, how it stays current, and who can see what.The routing decision tree and the connector list, published in full.
- 03Planning AI costs from pilot to productionPlan production costs from day one, and choose where each workload runs from measured load.How AI running costs change as usage grows, and how to plan for them so production costs hold no surprises.The arithmetic is published in full, so you can run it on your own numbers.
Citation library
Every source the posts lean on, linked back to whoever published it. Where a study is behind a subscription, the link goes to public reporting of it, and the note says so.
Links open on the publisher's own site
- The Root Causes of Failure for Artificial Intelligence Projects (RR-A2680-1) (opens in a new tab)
RAND Corporation, Ryseff, De Bruhl & Newberry · Independent research · 2024
65 practitioner interviews. More than 80% of AI projects fail, twice the rate of IT projects without AI. Four of the five root causes are organisational, not technical.
- The GenAI Divide: State of AI in Business 2025 (opens in a new tab)
MIT NANDA · Independent research · 2025
300 public deployments reviewed, 52 structured interviews, 153 survey responses. 95% of enterprise GenAI pilots produced no measurable P&L impact against $30–40bn of investment. MIT does not keep a stable public copy, so the link goes to Fortune's report on the study.
- Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 (opens in a new tab)
Gartner · Industry analysis · 2025
A forecast, not a measurement, we label it as such wherever it appears. Cites escalating cost, unclear value and inadequate risk controls. Also estimates only ~130 of thousands of self-described agentic vendors are real.
- Cost of a Data Breach Report 2025 (opens in a new tab)
IBM Security / Ponemon Institute · Industry analysis · 2025
600 breached organisations across 17 industries. 97% of organisations with an AI-related breach lacked proper AI access controls; shadow AI added roughly $670,000 to average breach cost.
- Prompt injection is not SQL injection (it may be worse) (opens in a new tab)
UK National Cyber Security Centre · Regulator or statute · 2025
The national technical authority's position: because a language model cannot separate instruction from data, prompt injection may never be fully mitigated. Design for blast radius, not prevention.
- Top 10 for Large Language Model Applications (opens in a new tab)
OWASP · Industry analysis · 2025
Prompt injection is LLM01, the first-ranked risk for language-model applications.
- Seven Failure Points When Engineering a Retrieval Augmented System (opens in a new tab)
Barnett, Kurniawan, Thudumu, Brannelly & Abdelrazek · Peer-reviewed · 2024
CAIN '24. The canonical taxonomy of where retrieval-augmented systems break. Establishes that the failures are architectural rather than tunable.
- From Local to Global: A Graph RAG Approach to Query-Focused Summarization (opens in a new tab)
Edge et al. Microsoft Research · Peer-reviewed · 2024
Graph-based retrieval beat vector retrieval on comprehensiveness win-rates of 72–83%, using over 97% fewer tokens for root-level community summaries. Win-rates are LLM-judged, not ground truth.
- FRAMES: Fact, Retrieval, and Reasoning Measurement Set (opens in a new tab)
Krishna et al. Google & Harvard · Peer-reviewed · 2024
Single-step naive retrieval scores ~0.40 accuracy against 0.66 for multi-step and 0.73 with oracle documents. Multi-hop questions are where naive retrieval collapses.
- Don't Do RAG: Cache-Augmented Generation (opens in a new tab)
Chan et al. National Chengchi University · Peer-reviewed · 2024
For bounded, stable corpora, preloading the whole corpus into cached context removes the retrieval hop entirely, large latency reductions at comparable or better accuracy.
- DistilBERT, a distilled version of BERT (opens in a new tab)
Sanh, Debut, Chaumond & Wolf, Hugging Face · Peer-reviewed · 2019
40% smaller, 60% faster, ~97% of language understanding retained. Still the clearest published statement of what distillation actually costs you.
- A Benchmark to Understand the Role of Knowledge Graphs on LLM's Accuracy for Q&A on Enterprise SQL Databases (opens in a new tab)
Sequeda, Allemang & Jacob, data.world · Peer-reviewed · 2023
Accuracy on enterprise question answering rose from 16.7% on a raw schema to 54.2% when the same questions were grounded in a knowledge graph.
- Deep Entity Matching with Pre-Trained Language Models (Ditto) (opens in a new tab)
Li, Li, Suhara, Doan & Tan · Peer-reviewed · 2021
VLDB 2021. The entity-resolution method behind a workable authority index: F1 of 96.5% matching two company datasets of 789K and 412K records.
- Regulation (EU) 2024/1689, the AI Act (opens in a new tab)
European Union · Regulator or statute · 2024
The consolidated text as amended in July 2026. The Commission's own explainer page is the easier starting point.
- The State of AI (opens in a new tab)
McKinsey & Company (QuantumBlack) · Industry analysis · 2025
88% of organisations use AI somewhere; only ~39% report enterprise-level EBIT impact. Workflow redesign is the single attribute most correlated with that impact.
- Voice of the Enterprise: AI & Machine Learning (opens in a new tab)
S&P Global Market Intelligence · Industry analysis · 2025
The share of companies abandoning most AI initiatives before production rose from 17% to 42% year over year. Cost, data privacy and security lead the cited obstacles. The survey itself is subscription research, so the link goes to CIO Dive's report on it.
- OpenLineage, an open standard for lineage metadata collection (opens in a new tab)
OpenLineage (LF AI & Data) · Industry analysis · 2026
The spec we emit lineage against, so your lineage is portable rather than locked to our tooling.
- Apache Iceberg (opens in a new tab)
Apache Software Foundation · Industry analysis · 2026
Our default open table format: specification-first and engine-independent, which is what makes a vendor-neutral lakehouse possible.
- Model Context Protocol specification (opens in a new tab)
Model Context Protocol · Industry analysis · 2026
How we expose governed queries as tools an agent can call, with the server acting as the policy gatekeeper rather than the model.
- Presidio, data protection and PII anonymisation (opens in a new tab)
Microsoft · Industry analysis · 2026
The pattern our classification stage follows: regex and checksum recognisers for structural identifiers, NER for contextual ones, confidence scored per column.
- Regulatory framework for artificial intelligence (opens in a new tab)
European Commission · Regulator or statute · 2026
The Commission's plain summary of the AI Act.
- Regulation (EU) 2026/1744, the Digital Omnibus on AI (opens in a new tab)
European Union · Regulator or statute · 2026
The 2026 amendment that moved the high-risk deadlines.
- ISO/IEC 42001:2023, Artificial intelligence management system (opens in a new tab)
ISO/IEC · Regulator or statute · 2023
The certifiable standard for how an organisation governs AI.
- ISO/IEC 23894:2023, Guidance on AI risk management (opens in a new tab)
ISO/IEC · Regulator or statute · 2023
Guidance, not certifiable; feeds into 42001.
- ISO/IEC 27001:2022, Information security management systems (opens in a new tab)
ISO/IEC · Regulator or statute · 2022
Often paired with 42001 where AI handles sensitive data.
- Artificial Intelligence Risk Management Framework (AI RMF 1.0) (opens in a new tab)
NIST · Regulator or statute · 2023
Voluntary US framework: govern, map, measure, manage.
- AI RMF Generative AI Profile (NIST AI 600-1) (opens in a new tab)
NIST · Regulator or statute · 2024
The generative-AI companion to the framework.
- Regulation (EU) 2016/679, the General Data Protection Regulation (opens in a new tab)
European Union · Regulator or statute · 2016
Applies fully to AI systems that process personal data.
- Regulation (EU) 2022/2554, Digital Operational Resilience Act (opens in a new tab)
European Union · Regulator or statute · 2022
ICT resilience rules for EU financial entities, applicable since January 2025.
- Regulation (EU) 2024/2847, Cyber Resilience Act (opens in a new tab)
European Union · Regulator or statute · 2024
Security requirements for products with digital elements.
- Directive (EU) 2024/2853, Product Liability Directive (opens in a new tab)
European Union · Regulator or statute · 2024
Extends product liability to software and AI.
- SR 11-7, Supervisory Guidance on Model Risk Management (opens in a new tab)
Federal Reserve and OCC · Regulator or statute · 2011
The reference text for model inventories and independent validation.
- SB 26-189, Automated Decision-Making Technology (opens in a new tab)
Colorado General Assembly · Regulator or statute · 2026
Replaced the 2024 Colorado AI Act; effective 1 January 2027.
- HB 149, Texas Responsible Artificial Intelligence Governance Act (opens in a new tab)
Texas Legislature · Regulator or statute · 2025
Intent-based rules, in force since 1 January 2026.
- Guidance on AI and data protection (opens in a new tab)
UK Information Commissioner's Office · Regulator or statute · 2025
How UK data protection law applies to AI.
- Model AI Governance Framework (opens in a new tab)
IMDA and PDPC, Singapore · Regulator or statute · 2024
Voluntary framework, with a 2024 generative-AI edition.
- AI Verify testing framework (opens in a new tab)
AI Verify Foundation · Regulator or statute · 2026
A voluntary testing toolkit that complements the Singapore framework.
- Act on Promotion of Research, Development and Utilisation of AI-Related Technologies (opens in a new tab)
Government of Japan · Regulator or statute · 2025
A framework law in force since September 2025, with no penalty regime.
- Digital Personal Data Protection Act 2023 and Rules 2025 (opens in a new tab)
Ministry of Electronics and Information Technology, India · Regulator or statute · 2025
Consent-based data protection law, operational since the 2025 Rules.
- OECD AI Principles (opens in a new tab)
OECD · Regulator or statute · 2024
The first intergovernmental AI standard, updated May 2024.
- State of FinOps 2026 (opens in a new tab)
FinOps Foundation · Industry analysis · 2026
Practitioner survey on managing AI spend.
- The Widening AI Value Gap: Build for the Future 2025 (opens in a new tab)
Boston Consulting Group · Industry analysis · 2025
Survey of more than 1,250 firms. Only 5% achieve AI value at scale; 70% of the available value sits in core functions.
- 2025 CEO Study: CEOs double down on AI while navigating enterprise hurdles (opens in a new tab)
IBM Institute for Business Value · Industry analysis · 2025
2,000 chief executives in 33 countries. 25% of AI initiatives delivered expected return; 16% have scaled enterprise-wide.
- One year of agentic AI: Six lessons from the people doing the work (opens in a new tab)
McKinsey & Company (QuantumBlack) · Industry analysis · 2025
Draws on more than fifty agentic builds. Treat an agent like a new hire: a job description, onboarding and continual feedback.
- Agentic AI and Human-in-the-Loop Interventions: Field Experimental Evidence from Alibaba's Customer Service Operations (opens in a new tab)
Field experiment on Alibaba's Taobao customer service · Peer-reviewed · 2026
A randomised field experiment. Agentic AI shortened chats but lowered ratings on the chats it handled; human hand-over protected quality on technical problems far more than on emotional ones. A preprint.
- Klarna's return to human customer service (opens in a new tab)
Fortune · Industry analysis · 2025
Reporting on Klarna's chief executive saying cost had weighed too heavily in its AI customer service and quality had suffered.
- 80 percent of your data will be unstructured in five years (opens in a new tab)
IDC, via Solutions Review · Industry analysis · 2019
IDC's forecast that around 80% of data would be unstructured by 2025.
- Zero-click AI vulnerability exposes Microsoft 365 Copilot data (EchoLeak, CVE-2025-32711) (opens in a new tab)
The Hacker News · Industry analysis · 2025
A crafted email could make the assistant leak data with no user action. Fixed by Microsoft in 2025.
- Enterprise AI and SaaS Data Security Report 2025 (opens in a new tab)
TechRepublic, on LayerX Security · Vendor research · 2025
Browser telemetry showing most workplace generative AI use runs through personal, unmanaged accounts.
- Microsoft tells French lawmakers it can't protect user data from US demands (opens in a new tab)
SDxCentral · Industry analysis · 2025
Microsoft France's testimony to the French Senate, June 2025.
- Data quality: best practices for accurate insights (opens in a new tab)
Gartner · Industry analysis · 2024
The estimate that poor data quality costs organisations an average of $12.9 million a year.
- Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (opens in a new tab)
Lei et al. · Peer-reviewed · 2024
On real enterprise schemas the best agent tested solved 21.3% of tasks, against 91.2% on Spider 1.0.
- Small Language Models are the Future of Agentic AI (opens in a new tab)
Belcak et al., NVIDIA Research · Peer-reviewed · 2025
A position paper arguing small models are the right default for most calls inside agentic systems.
- LLM inference price trends (opens in a new tab)
Epoch AI · Independent research · 2025
The price of a given level of capability fell between 9 and 900 times a year, depending on the task.
- Published API pricing pages (opens in a new tab)
Anthropic, OpenAI and DeepSeek · Vendor research · 2026
List prices, batch discounts and prompt-caching rates. Also: openai.com/api/pricing and api-docs.deepseek.com.
- Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation (opens in a new tab)
Patil · Peer-reviewed · 2026
On identical H100 hardware the effective cost of a million output tokens spans $0.21 to $15.25 depending on utilisation, a 2.5 to 24 times penalty across ordinary enterprise load. Any estimate that assumes a saturated GPU understates self-hosted cost by 1/U. A preprint.
- Efficient Memory Management for Large Language Model Serving with PagedAttention (opens in a new tab)
Kwon et al. · Peer-reviewed · 2023
Pages the KV cache as an operating system pages memory. Near-zero waste and 2 to 4 times the throughput at equal latency against FasterTransformer and Orca. The design behind vLLM.
- Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region (opens in a new tab)
Amazon Web Services · Vendor research · 2025
AWS's post-event summary for 19 and 20 October 2025: a race in DynamoDB's DNS automation left an empty record, with impact in three phases ending at 2:20 PM PDT on 20 October.
- Google Cloud Service Health incident report, 12 June 2025 (opens in a new tab)
Google Cloud · Vendor research · 2025
A policy with blank fields, replicated globally, crashed an unflagged code path in Service Control in every region. Remediation: modularise, and fail open.
- Azure status history: Azure Front Door, tracking ID YKYN-BWZ (opens in a new tab)
Microsoft · Vendor research · 2025
Microsoft's post-incident review for 29 October 2025, 15:45 to 00:05 UTC: an invalid configuration state caused AFD nodes to fail to load, and changes were blocked while a last-known-good configuration rolled out.
- Cloudflare outage on November 18, 2025 (opens in a new tab)
Cloudflare · Vendor research · 2025
Cloudflare's postmortem. A database permission change doubled the Bot Management feature file past a limit of 200 features against normal use of about 60.
- Channel File 291 Incident: Root Cause Analysis (opens in a new tab)
CrowdStrike · Vendor research · 2024
The external technical root cause analysis of 6 August 2024: a template type defined 21 input fields, the integration code supplied 20, and reading the 21st was an out-of-bounds memory read.
- Key changes in the 2026-07-28 specification (opens in a new tab)
Model Context Protocol · Industry analysis · 2026
Removes protocol sessions and the initialize handshake, requires server/discover, makes broken streams re-issued as new requests, adds ttlMs and cacheScope, and deprecates Roots, Sampling, Logging and Dynamic Client Registration.
- OWASP Top 10 for Agentic Applications for 2026 (opens in a new tab)
OWASP GenAI Security Project · Industry analysis · 2025
Released 9 December 2025. ASI01 Agent Goal Hijack to ASI10 Rogue Agents.
- FOCUS: the FinOps Open Cost and Usage Specification (opens in a new tab)
FinOps Foundation · Industry analysis · 2026
Version 1.4, ratified 4 June 2026, adds two datasets, 47 columns and six attributes and no dedicated token columns. Token properties are proposed for 1.5.
- Dynamic Resource Allocation (opens in a new tab)
Kubernetes · Industry analysis · 2026
How Kubernetes allocates devices such as GPUs by attribute, and the feature state of each part.
- Kubernetes v1.36 release announcement (opens in a new tab)
Kubernetes · Industry analysis · 2026
Released 22 April 2026. DRA's Prioritized List graduates to stable; partitionable devices and related features remain beta.
- Version Skew Policy (opens in a new tab)
Kubernetes · Industry analysis · 2026
The supported version differences between the API server, kubelet, controller manager, scheduler and kubectl, and why minor versions can't be skipped.
- Policy evaluation logic, AWS IAM User Guide (opens in a new tab)
Amazon Web Services · Vendor research · 2026
The order in which AWS evaluates explicit deny, RCPs, SCPs, identity and resource policies, permission boundaries and session policies.
- Resource control policies (RCPs), AWS Organizations User Guide (opens in a new tab)
Amazon Web Services · Vendor research · 2026
RCPs set the maximum permissions on resources in member accounts and never grant permissions.
- Principal access boundary policies (opens in a new tab)
Google Cloud · Vendor research · 2026
Policies that define which resources a principal is eligible to access, evaluated before deny and allow.
- Deny policies (opens in a new tab)
Google Cloud · Vendor research · 2026
Deny policies are checked before allow policies, and a deny condition that cannot be evaluated applies.
- What is Azure role-based access control (Azure RBAC)? (opens in a new tab)
Microsoft · Vendor research · 2026
Azure RBAC is additive; role assignments are inherited from management group to resource.
- Understand Azure deny assignments (opens in a new tab)
Microsoft · Vendor research · 2026
Deny assignments block access even where a role grants it. All deny assignments are currently system-protected.
- What is Azure Policy? (opens in a new tab)
Microsoft · Vendor research · 2026
A separate engine that evaluates resource configuration, with effects such as deny, audit, modify and deployIfNotExists.
- Continuous access evaluation in Microsoft Entra (opens in a new tab)
Microsoft · Vendor research · 2026
Tokens up to 28 hours in CAE sessions, revoked on critical events at participating resources.
- Agent identity blueprints in Microsoft Entra Agent ID (opens in a new tab)
Microsoft · Vendor research · 2026
Credentials are configured on the blueprint; policy applied to it reaches every agent identity; inheritable permissions flow to agent identities after consent.
- OAuth 2.0 on-behalf-of flow, Microsoft identity platform (opens in a new tab)
Microsoft · Vendor research · 2026
How a middle tier exchanges the token it received for one issued to a downstream API.
- Microsoft Purview DLP for Microsoft 365 Copilot (opens in a new tab)
Microsoft · Vendor research · 2026
Custom template only; selecting the location disables other locations in the policy; updates take up to four hours to reach Copilot.
- Restricted Content Discovery for SharePoint sites (opens in a new tab)
Microsoft · Vendor research · 2026
A site setting that keeps content out of tenant-wide search and Copilot without changing permissions.
- Restricted SharePoint Search (opens in a new tab)
Microsoft · Vendor research · 2026
Retiring: new enablement blocked from 31 July 2026. Described by Microsoft as temporary, and not a security boundary.
- Overview of Selected permissions in OneDrive and SharePoint (opens in a new tab)
Microsoft · Vendor research · 2026
Sites.Selected and related scopes grant nothing until an explicit per-resource grant is made.
- Amazon S3 strong consistency (opens in a new tab)
Amazon Web Services · Vendor research · 2020
Strong read-after-write consistency for all GET, PUT and LIST operations, in all Regions, since December 2020.
- Amazon EC2 Capacity Blocks for ML pricing (opens in a new tab)
Amazon Web Services · Vendor research · 2026
Prices per accelerator-hour, described as dynamic. Used here for the rates effective 7 October 2026.
- AWS raises GPU prices 15% on a Saturday (opens in a new tab)
The Register · Independent research · 2026
Reports the January 2026 increase: p5e.48xlarge from $34.61 to $39.80 an hour, p5en.48xlarge from $36.18 to $41.61. No AWS announcement was found.
- SP 800-207, Zero Trust Architecture (opens in a new tab)
NIST · Regulator or statute · 2020
The reference definition of zero trust: per-request access decisions by a policy decision point and enforcement point.
- Static stability using Availability Zones (opens in a new tab)
Amazon Web Services · Vendor research · 2020
The Amazon Builders' Library account of designing data planes that keep working without control-plane action.
- Gemini Enterprise Agent Platform lets you build, govern and optimize your agents (opens in a new tab)
Google · Vendor research · 2026
Announced at Google Cloud Next on 22 April 2026 as the evolution of Vertex AI, through which Vertex AI services now ship.
- ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems (opens in a new tab)
MITRE · Independent research · 2026
A knowledge base of adversary tactics and techniques against AI systems, used to build red-team suites.
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (opens in a new tab)
Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le & Zhou · Independent research · 2022
Showing a model a few worked examples with their intermediate steps improves multi-step arithmetic, commonsense and symbolic tasks, and the effect appears only in sufficiently large models.
- Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (opens in a new tab)
Snell, Lee, Xu & Kumar · Independent research · 2024
Allocating thinking time per question is more than 4 times as efficient as a best-of-N baseline. With matched compute, a smaller model given more thinking time beat a model 14 times its size, but only on questions it could already sometimes answer.
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (opens in a new tab)
DeepSeek-AI, Guo, Yang et al. · Peer-reviewed · 2025
Nature 645, 633–638 (2025). Reasoning trained by reinforcement learning on verifiable answers, without human-written reasoning traces; self-reflection and verification emerged during training, and the patterns could be passed to smaller models.
- Reasoning Models Don't Always Say What They Think (opens in a new tab)
Chen et al. (Anthropic) · Independent research · 2025
When reasoning models used a hint planted in the question, their written reasoning often admitted it in fewer than 20% of the cases where it was used. Reading the reasoning is useful for spotting problems but cannot rule them out.
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (opens in a new tab)
Shojaee, Mirzadeh, Alizadeh, Horton, Bengio & Farajtabar · Independent research · 2025
On puzzles of controlled difficulty, standard models did better on easy problems, reasoning models on medium ones, and both collapsed on hard ones, with reasoning effort falling as problems got harder.
- Gemini: A Family of Highly Capable Multimodal Models (opens in a new tab)
Gemini Team, Google · Vendor research · 2023
A model family trained across image, audio, video and text from the start, reporting cross-modal reasoning and improved results on all 20 multimodal benchmarks examined.
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (opens in a new tab)
Yue, Ni, Zhang et al. · Peer-reviewed · 2024
CVPR 2024. About 11,500 college-level questions across 30 subjects, with 30 kinds of image including charts, diagrams, tables, maps and chemical structures. The strongest models at publication scored 56% and 59%.
- Evaluating Object Hallucination in Large Vision-Language Models (opens in a new tab)
Li, Du, Zhou, Wang, Zhao & Wen · Peer-reviewed · 2023
EMNLP 2023. Most vision-language models tested described objects that were not in the image, especially objects that commonly appear alongside what is there. Introduces POPE, a more stable way to measure it.
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (opens in a new tab)
Yao, Shinn, Razavi & Narasimhan · Independent research · 2024
Agents using tools under written business policies succeeded on fewer than half of the tasks, and solved the same retail task in all of eight attempts less than 25% of the time.
- Measuring AI Ability to Complete Long Software Tasks (opens in a new tab)
Kwa, West, Becker et al. (METR) · Peer-reviewed · 2025
NeurIPS 2025. The length of task, measured in human working time, that frontier agents complete half the time has doubled roughly every seven months since 2019; around 50 minutes at the time of the study.
- Dual-Use Foundation Models with Widely Available Model Weights (opens in a new tab)
National Telecommunications and Information Administration, US Department of Commerce · Regulator or statute · 2024
30 July 2024. Found the evidence insufficient to justify restricting open weights, recommended monitoring instead, and listed the benefits, including running a model without sharing data with a third party.
- Open-weight models lag state-of-the-art by around 3 months on average (opens in a new tab)
Epoch AI · Independent research · 2025
30 October 2025. On a composite of many benchmarks, the best open-weight models trailed the best closed models by about 3.5 months on average (90% interval 1.1 to 5.3 months), and at times the gap closed entirely.