# Blog - Building AI systems that last

> Practical, fully sourced reading on what makes AI systems work, getting your data ready, and planning what they cost to run in production.

- Web page: https://www.alesanalytics.com/research
- Markdown: https://www.alesanalytics.com/research.md
- Publisher: Ales Analytics

## Latest posts

| Report | Title | Published | Finding | Markdown |
| --- | --- | --- | --- | --- |
| 01 | What makes enterprise AI work | 2026-10-01 | The systems that last are the ones built for the data and the workflow they serve. | https://www.alesanalytics.com/research/why-enterprise-ai-fails.md |
| 02 | Getting your data ready for AI | 2026-10-04 | Searching documents is one tool among several; a well-mapped data layer lets the system pick the right one. | https://www.alesanalytics.com/research/data-foundation-layer.md |
| 03 | Planning AI costs from pilot to production | 2026-10-07 | Plan production costs from day one, and choose where each workload runs from measured load. | https://www.alesanalytics.com/research/cost-cliff.md |

## Citation library

1. [The Root Causes of Failure for Artificial Intelligence Projects (RR-A2680-1)](https://www.rand.org/pubs/research_reports/RRA2680-1.html). RAND Corporation, Ryseff, De Bruhl & Newberry, 2024. 65 practitioner interviews. More than 80% of AI projects fail, twice the rate of IT projects without AI. Four of the five root causes are organisational, not technical.
2. [The GenAI Divide: State of AI in Business 2025](https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/). MIT NANDA, 2025. 300 public deployments reviewed, 52 structured interviews, 153 survey responses. 95% of enterprise GenAI pilots produced no measurable P&L impact against $30–40bn of investment. MIT does not keep a stable public copy, so the link goes to Fortune's report on the study.
3. [Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027](https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027). Gartner, 2025. A forecast, not a measurement, we label it as such wherever it appears. Cites escalating cost, unclear value and inadequate risk controls. Also estimates only ~130 of thousands of self-described agentic vendors are real.
4. [Cost of a Data Breach Report 2025](https://www.ibm.com/reports/data-breach). IBM Security / Ponemon Institute, 2025. 600 breached organisations across 17 industries. 97% of organisations with an AI-related breach lacked proper AI access controls; shadow AI added roughly $670,000 to average breach cost.
5. [Prompt injection is not SQL injection (it may be worse)](https://www.ncsc.gov.uk/blog-post/prompt-injection-is-not-sql-injection). UK National Cyber Security Centre, 2025. The national technical authority's position: because a language model cannot separate instruction from data, prompt injection may never be fully mitigated. Design for blast radius, not prevention.
6. [Top 10 for Large Language Model Applications](https://genai.owasp.org/llm-top-10/). OWASP, 2025. Prompt injection is LLM01, the first-ranked risk for language-model applications.
7. [Seven Failure Points When Engineering a Retrieval Augmented System](https://arxiv.org/abs/2401.05856). Barnett, Kurniawan, Thudumu, Brannelly & Abdelrazek, 2024. CAIN '24. The canonical taxonomy of where retrieval-augmented systems break. Establishes that the failures are architectural rather than tunable.
8. [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://arxiv.org/abs/2404.16130). Edge et al. Microsoft Research, 2024. Graph-based retrieval beat vector retrieval on comprehensiveness win-rates of 72–83%, using over 97% fewer tokens for root-level community summaries. Win-rates are LLM-judged, not ground truth.
9. [FRAMES: Fact, Retrieval, and Reasoning Measurement Set](https://arxiv.org/abs/2409.12941). Krishna et al. Google & Harvard, 2024. Single-step naive retrieval scores ~0.40 accuracy against 0.66 for multi-step and 0.73 with oracle documents. Multi-hop questions are where naive retrieval collapses.
10. [Don't Do RAG: Cache-Augmented Generation](https://arxiv.org/abs/2412.15605). Chan et al. National Chengchi University, 2024. For bounded, stable corpora, preloading the whole corpus into cached context removes the retrieval hop entirely, large latency reductions at comparable or better accuracy.
11. [DistilBERT, a distilled version of BERT](https://arxiv.org/abs/1910.01108). Sanh, Debut, Chaumond & Wolf, Hugging Face, 2019. 40% smaller, 60% faster, ~97% of language understanding retained. Still the clearest published statement of what distillation actually costs you.
12. [A Benchmark to Understand the Role of Knowledge Graphs on LLM's Accuracy for Q&A on Enterprise SQL Databases](https://arxiv.org/abs/2311.07509). Sequeda, Allemang & Jacob, data.world, 2023. Accuracy on enterprise question answering rose from 16.7% on a raw schema to 54.2% when the same questions were grounded in a knowledge graph.
13. [Deep Entity Matching with Pre-Trained Language Models (Ditto)](https://arxiv.org/abs/2004.00584). Li, Li, Suhara, Doan & Tan, 2021. VLDB 2021. The entity-resolution method behind a workable authority index: F1 of 96.5% matching two company datasets of 789K and 412K records.
14. [Regulation (EU) 2024/1689, the AI Act](https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng). European Union, 2024. The consolidated text as amended in July 2026. The Commission's own explainer page is the easier starting point.
15. [The State of AI](https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai). McKinsey & Company (QuantumBlack), 2025. 88% of organisations use AI somewhere; only ~39% report enterprise-level EBIT impact. Workflow redesign is the single attribute most correlated with that impact.
16. [Voice of the Enterprise: AI & Machine Learning](https://www.ciodive.com/news/AI-project-fail-data-SPGlobal/742590/). S&P Global Market Intelligence, 2025. The share of companies abandoning most AI initiatives before production rose from 17% to 42% year over year. Cost, data privacy and security lead the cited obstacles. The survey itself is subscription research, so the link goes to CIO Dive's report on it.
17. [OpenLineage, an open standard for lineage metadata collection](https://openlineage.io/). OpenLineage (LF AI & Data), 2026. The spec we emit lineage against, so your lineage is portable rather than locked to our tooling.
18. [Apache Iceberg](https://iceberg.apache.org/). Apache Software Foundation, 2026. Our default open table format: specification-first and engine-independent, which is what makes a vendor-neutral lakehouse possible.
19. [Model Context Protocol specification](https://modelcontextprotocol.io/). Model Context Protocol, 2026. How we expose governed queries as tools an agent can call, with the server acting as the policy gatekeeper rather than the model.
20. [Presidio, data protection and PII anonymisation](https://microsoft.github.io/presidio/). Microsoft, 2026. The pattern our classification stage follows: regex and checksum recognisers for structural identifiers, NER for contextual ones, confidence scored per column.
21. [Regulatory framework for artificial intelligence](https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai). European Commission, 2026. The Commission's plain summary of the AI Act.
22. [Regulation (EU) 2026/1744, the Digital Omnibus on AI](https://eur-lex.europa.eu/eli/reg/2026/1744/oj/eng). European Union, 2026. The 2026 amendment that moved the high-risk deadlines.
23. [ISO/IEC 42001:2023, Artificial intelligence management system](https://www.iso.org/standard/42001). ISO/IEC, 2023. The certifiable standard for how an organisation governs AI.
24. [ISO/IEC 23894:2023, Guidance on AI risk management](https://www.iso.org/standard/77304.html). ISO/IEC, 2023. Guidance, not certifiable; feeds into 42001.
25. [ISO/IEC 27001:2022, Information security management systems](https://www.iso.org/standard/27001). ISO/IEC, 2022. Often paired with 42001 where AI handles sensitive data.
26. [Artificial Intelligence Risk Management Framework (AI RMF 1.0)](https://www.nist.gov/itl/ai-risk-management-framework). NIST, 2023. Voluntary US framework: govern, map, measure, manage.
27. [AI RMF Generative AI Profile (NIST AI 600-1)](https://doi.org/10.6028/NIST.AI.600-1). NIST, 2024. The generative-AI companion to the framework.
28. [Regulation (EU) 2016/679, the General Data Protection Regulation](https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng). European Union, 2016. Applies fully to AI systems that process personal data.
29. [Regulation (EU) 2022/2554, Digital Operational Resilience Act](https://eur-lex.europa.eu/eli/reg/2022/2554/oj/eng). European Union, 2022. ICT resilience rules for EU financial entities, applicable since January 2025.
30. [Regulation (EU) 2024/2847, Cyber Resilience Act](https://eur-lex.europa.eu/eli/reg/2024/2847/oj/eng). European Union, 2024. Security requirements for products with digital elements.
31. [Directive (EU) 2024/2853, Product Liability Directive](https://eur-lex.europa.eu/eli/dir/2024/2853/oj/eng). European Union, 2024. Extends product liability to software and AI.
32. [SR 11-7, Supervisory Guidance on Model Risk Management](https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm). Federal Reserve and OCC, 2011. The reference text for model inventories and independent validation.
33. [SB 26-189, Automated Decision-Making Technology](https://leg.colorado.gov/bills/sb26-189). Colorado General Assembly, 2026. Replaced the 2024 Colorado AI Act; effective 1 January 2027.
34. [HB 149, Texas Responsible Artificial Intelligence Governance Act](https://capitol.texas.gov/tlodocs/89R/analysis/html/HB00149S.htm). Texas Legislature, 2025. Intent-based rules, in force since 1 January 2026.
35. [Guidance on AI and data protection](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/). UK Information Commissioner's Office, 2025. How UK data protection law applies to AI.
36. [Model AI Governance Framework](https://www.pdpc.gov.sg/help-and-resources/2020/01/model-ai-governance-framework). IMDA and PDPC, Singapore, 2024. Voluntary framework, with a 2024 generative-AI edition.
37. [AI Verify testing framework](https://aiverifyfoundation.sg/). AI Verify Foundation, 2026. A voluntary testing toolkit that complements the Singapore framework.
38. [Act on Promotion of Research, Development and Utilisation of AI-Related Technologies](https://www.japaneselawtranslation.go.jp/en/laws/view/5066/en). Government of Japan, 2025. A framework law in force since September 2025, with no penalty regime.
39. [Digital Personal Data Protection Act 2023 and Rules 2025](https://www.meity.gov.in/data-protection-framework). Ministry of Electronics and Information Technology, India, 2025. Consent-based data protection law, operational since the 2025 Rules.
40. [OECD AI Principles](https://oecd.ai/en/ai-principles). OECD, 2024. The first intergovernmental AI standard, updated May 2024.
41. [State of FinOps 2026](https://www.linuxfoundation.org/press/state-of-finops-survey-ai-value-and-skills-top-priorities-as-finops-matures-across-technology-value-98-manage-ai-90-saas-64-licensing-48-data-center-1). FinOps Foundation, 2026. Practitioner survey on managing AI spend.
42. [The Widening AI Value Gap: Build for the Future 2025](https://media-publications.bcg.com/The-Widening-AI-Value-Gap-October-2025.pdf). Boston Consulting Group, 2025. Survey of more than 1,250 firms. Only 5% achieve AI value at scale; 70% of the available value sits in core functions.
43. [2025 CEO Study: CEOs double down on AI while navigating enterprise hurdles](https://newsroom.ibm.com/2025-05-06-ibm-study-ceos-double-down-on-ai-while-navigating-enterprise-hurdles). IBM Institute for Business Value, 2025. 2,000 chief executives in 33 countries. 25% of AI initiatives delivered expected return; 16% have scaled enterprise-wide.
44. [One year of agentic AI: Six lessons from the people doing the work](https://www.mckinsey.com/capabilities/quantumblack/our-insights/one-year-of-agentic-ai-six-lessons-from-the-people-doing-the-work). McKinsey & Company (QuantumBlack), 2025. Draws on more than fifty agentic builds. Treat an agent like a new hire: a job description, onboarding and continual feedback.
45. [Agentic AI and Human-in-the-Loop Interventions: Field Experimental Evidence from Alibaba's Customer Service Operations](https://arxiv.org/abs/2605.14830). Field experiment on Alibaba's Taobao customer service, 2026. A randomised field experiment. Agentic AI shortened chats but lowered ratings on the chats it handled; human hand-over protected quality on technical problems far more than on emotional ones. A preprint.
46. [Klarna's return to human customer service](https://fortune.com/2025/05/09/klarna-ai-humans-return-on-investment/). Fortune, 2025. Reporting on Klarna's chief executive saying cost had weighed too heavily in its AI customer service and quality had suffered.
47. [80 percent of your data will be unstructured in five years](https://solutionsreview.com/data-management/80-percent-of-your-data-will-be-unstructured-in-five-years/). IDC, via Solutions Review, 2019. IDC's forecast that around 80% of data would be unstructured by 2025.
48. [Zero-click AI vulnerability exposes Microsoft 365 Copilot data (EchoLeak, CVE-2025-32711)](https://thehackernews.com/2025/06/zero-click-ai-vulnerability-exposes.html). The Hacker News, 2025. A crafted email could make the assistant leak data with no user action. Fixed by Microsoft in 2025.
49. [Enterprise AI and SaaS Data Security Report 2025](https://www.techrepublic.com/article/news-employees-share-company-secrets-on-chatgpt/). TechRepublic, on LayerX Security, 2025. Browser telemetry showing most workplace generative AI use runs through personal, unmanaged accounts.
50. [Microsoft tells French lawmakers it can't protect user data from US demands](https://www.sdxcentral.com/news/microsoft-tells-french-lawmakers-it-cant-protect-user-data-from-us-demands/). SDxCentral, 2025. Microsoft France's testimony to the French Senate, June 2025.
51. [Data quality: best practices for accurate insights](https://www.gartner.com/en/data-analytics/topics/data-quality). Gartner, 2024. The estimate that poor data quality costs organisations an average of $12.9 million a year.
52. [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://arxiv.org/abs/2411.07763). Lei et al., 2024. On real enterprise schemas the best agent tested solved 21.3% of tasks, against 91.2% on Spider 1.0.
53. [Small Language Models are the Future of Agentic AI](https://arxiv.org/abs/2506.02153). Belcak et al., NVIDIA Research, 2025. A position paper arguing small models are the right default for most calls inside agentic systems.
54. [LLM inference price trends](https://epoch.ai/data-insights/llm-inference-price-trends). Epoch AI, 2025. The price of a given level of capability fell between 9 and 900 times a year, depending on the task.
55. [Published API pricing pages](https://www.anthropic.com/pricing). Anthropic, OpenAI and DeepSeek, 2026. List prices, batch discounts and prompt-caching rates. Also: openai.com/api/pricing and api-docs.deepseek.com.
56. [Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation](https://arxiv.org/abs/2606.11690). Patil, 2026. On identical H100 hardware the effective cost of a million output tokens spans $0.21 to $15.25 depending on utilisation, a 2.5 to 24 times penalty across ordinary enterprise load. Any estimate that assumes a saturated GPU understates self-hosted cost by 1/U. A preprint.
57. [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180). Kwon et al., 2023. Pages the KV cache as an operating system pages memory. Near-zero waste and 2 to 4 times the throughput at equal latency against FasterTransformer and Orca. The design behind vLLM.
58. [Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/101925/). Amazon Web Services, 2025. AWS's post-event summary for 19 and 20 October 2025: a race in DynamoDB's DNS automation left an empty record, with impact in three phases ending at 2:20 PM PDT on 20 October.
59. [Google Cloud Service Health incident report, 12 June 2025](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW). Google Cloud, 2025. A policy with blank fields, replicated globally, crashed an unflagged code path in Service Control in every region. Remediation: modularise, and fail open.
60. [Azure status history: Azure Front Door, tracking ID YKYN-BWZ](https://azure.status.microsoft/en-us/status/history/). Microsoft, 2025. Microsoft's post-incident review for 29 October 2025, 15:45 to 00:05 UTC: an invalid configuration state caused AFD nodes to fail to load, and changes were blocked while a last-known-good configuration rolled out.
61. [Cloudflare outage on November 18, 2025](https://blog.cloudflare.com/18-november-2025-outage/). Cloudflare, 2025. Cloudflare's postmortem. A database permission change doubled the Bot Management feature file past a limit of 200 features against normal use of about 60.
62. [Channel File 291 Incident: Root Cause Analysis](https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf). CrowdStrike, 2024. The external technical root cause analysis of 6 August 2024: a template type defined 21 input fields, the integration code supplied 20, and reading the 21st was an out-of-bounds memory read.
63. [Key changes in the 2026-07-28 specification](https://modelcontextprotocol.io/specification/2026-07-28/changelog). Model Context Protocol, 2026. Removes protocol sessions and the initialize handshake, requires server/discover, makes broken streams re-issued as new requests, adds ttlMs and cacheScope, and deprecates Roots, Sampling, Logging and Dynamic Client Registration.
64. [OWASP Top 10 for Agentic Applications for 2026](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/). OWASP GenAI Security Project, 2025. Released 9 December 2025. ASI01 Agent Goal Hijack to ASI10 Rogue Agents.
65. [FOCUS: the FinOps Open Cost and Usage Specification](https://focus.finops.org/). FinOps Foundation, 2026. Version 1.4, ratified 4 June 2026, adds two datasets, 47 columns and six attributes and no dedicated token columns. Token properties are proposed for 1.5.
66. [Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/). Kubernetes, 2026. How Kubernetes allocates devices such as GPUs by attribute, and the feature state of each part.
67. [Kubernetes v1.36 release announcement](https://kubernetes.io/blog/2026/04/22/kubernetes-v1-36-release/). Kubernetes, 2026. Released 22 April 2026. DRA's Prioritized List graduates to stable; partitionable devices and related features remain beta.
68. [Version Skew Policy](https://kubernetes.io/releases/version-skew-policy/). Kubernetes, 2026. The supported version differences between the API server, kubelet, controller manager, scheduler and kubectl, and why minor versions can't be skipped.
69. [Policy evaluation logic, AWS IAM User Guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html). Amazon Web Services, 2026. The order in which AWS evaluates explicit deny, RCPs, SCPs, identity and resource policies, permission boundaries and session policies.
70. [Resource control policies (RCPs), AWS Organizations User Guide](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_rcps.html). Amazon Web Services, 2026. RCPs set the maximum permissions on resources in member accounts and never grant permissions.
71. [Principal access boundary policies](https://cloud.google.com/iam/docs/principal-access-boundary-policies). Google Cloud, 2026. Policies that define which resources a principal is eligible to access, evaluated before deny and allow.
72. [Deny policies](https://cloud.google.com/iam/docs/deny-overview). Google Cloud, 2026. Deny policies are checked before allow policies, and a deny condition that cannot be evaluated applies.
73. [What is Azure role-based access control (Azure RBAC)?](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview). Microsoft, 2026. Azure RBAC is additive; role assignments are inherited from management group to resource.
74. [Understand Azure deny assignments](https://learn.microsoft.com/en-us/azure/role-based-access-control/deny-assignments). Microsoft, 2026. Deny assignments block access even where a role grants it. All deny assignments are currently system-protected.
75. [What is Azure Policy?](https://learn.microsoft.com/en-us/azure/governance/policy/overview). Microsoft, 2026. A separate engine that evaluates resource configuration, with effects such as deny, audit, modify and deployIfNotExists.
76. [Continuous access evaluation in Microsoft Entra](https://learn.microsoft.com/en-us/entra/identity/conditional-access/concept-continuous-access-evaluation). Microsoft, 2026. Tokens up to 28 hours in CAE sessions, revoked on critical events at participating resources.
77. [Agent identity blueprints in Microsoft Entra Agent ID](https://learn.microsoft.com/en-us/entra/agent-id/agent-blueprint). Microsoft, 2026. Credentials are configured on the blueprint; policy applied to it reaches every agent identity; inheritable permissions flow to agent identities after consent.
78. [OAuth 2.0 on-behalf-of flow, Microsoft identity platform](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-on-behalf-of-flow). Microsoft, 2026. How a middle tier exchanges the token it received for one issued to a downstream API.
79. [Microsoft Purview DLP for Microsoft 365 Copilot](https://learn.microsoft.com/en-us/purview/dlp-microsoft365-copilot-location-learn-about). Microsoft, 2026. Custom template only; selecting the location disables other locations in the policy; updates take up to four hours to reach Copilot.
80. [Restricted Content Discovery for SharePoint sites](https://learn.microsoft.com/en-us/sharepoint/restricted-content-discovery). Microsoft, 2026. A site setting that keeps content out of tenant-wide search and Copilot without changing permissions.
81. [Restricted SharePoint Search](https://learn.microsoft.com/en-us/sharepoint/restricted-sharepoint-search). Microsoft, 2026. Retiring: new enablement blocked from 31 July 2026. Described by Microsoft as temporary, and not a security boundary.
82. [Overview of Selected permissions in OneDrive and SharePoint](https://learn.microsoft.com/en-us/graph/permissions-selected-overview). Microsoft, 2026. Sites.Selected and related scopes grant nothing until an explicit per-resource grant is made.
83. [Amazon S3 strong consistency](https://aws.amazon.com/s3/consistency/). Amazon Web Services, 2020. Strong read-after-write consistency for all GET, PUT and LIST operations, in all Regions, since December 2020.
84. [Amazon EC2 Capacity Blocks for ML pricing](https://aws.amazon.com/ec2/capacityblocks/pricing/). Amazon Web Services, 2026. Prices per accelerator-hour, described as dynamic. Used here for the rates effective 7 October 2026.
85. [AWS raises GPU prices 15% on a Saturday](https://www.theregister.com/2026/01/05/aws_price_increase/). The Register, 2026. Reports the January 2026 increase: p5e.48xlarge from $34.61 to $39.80 an hour, p5en.48xlarge from $36.18 to $41.61. No AWS announcement was found.
86. [SP 800-207, Zero Trust Architecture](https://csrc.nist.gov/pubs/sp/800/207/final). NIST, 2020. The reference definition of zero trust: per-request access decisions by a policy decision point and enforcement point.
87. [Static stability using Availability Zones](https://aws.amazon.com/builders-library/static-stability-using-availability-zones/). Amazon Web Services, 2020. The Amazon Builders' Library account of designing data planes that keep working without control-plane action.
88. [Gemini Enterprise Agent Platform lets you build, govern and optimize your agents](https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/gemini-enterprise-agent-platform/). Google, 2026. Announced at Google Cloud Next on 22 April 2026 as the evolution of Vertex AI, through which Vertex AI services now ship.
89. [ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems](https://atlas.mitre.org/). MITRE, 2026. A knowledge base of adversary tactics and techniques against AI systems, used to build red-team suites.
90. [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903). Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le & Zhou, 2022. Showing a model a few worked examples with their intermediate steps improves multi-step arithmetic, commonsense and symbolic tasks, and the effect appears only in sufficiently large models.
91. [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://arxiv.org/abs/2408.03314). Snell, Lee, Xu & Kumar, 2024. Allocating thinking time per question is more than 4 times as efficient as a best-of-N baseline. With matched compute, a smaller model given more thinking time beat a model 14 times its size, but only on questions it could already sometimes answer.
92. [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/abs/2501.12948). DeepSeek-AI, Guo, Yang et al., 2025. Nature 645, 633–638 (2025). Reasoning trained by reinforcement learning on verifiable answers, without human-written reasoning traces; self-reflection and verification emerged during training, and the patterns could be passed to smaller models.
93. [Reasoning Models Don't Always Say What They Think](https://arxiv.org/abs/2505.05410). Chen et al. (Anthropic), 2025. When reasoning models used a hint planted in the question, their written reasoning often admitted it in fewer than 20% of the cases where it was used. Reading the reasoning is useful for spotting problems but cannot rule them out.
94. [The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity](https://arxiv.org/abs/2506.06941). Shojaee, Mirzadeh, Alizadeh, Horton, Bengio & Farajtabar, 2025. On puzzles of controlled difficulty, standard models did better on easy problems, reasoning models on medium ones, and both collapsed on hard ones, with reasoning effort falling as problems got harder.
95. [Gemini: A Family of Highly Capable Multimodal Models](https://arxiv.org/abs/2312.11805). Gemini Team, Google, 2023. A model family trained across image, audio, video and text from the start, reporting cross-modal reasoning and improved results on all 20 multimodal benchmarks examined.
96. [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://arxiv.org/abs/2311.16502). Yue, Ni, Zhang et al., 2024. CVPR 2024. About 11,500 college-level questions across 30 subjects, with 30 kinds of image including charts, diagrams, tables, maps and chemical structures. The strongest models at publication scored 56% and 59%.
97. [Evaluating Object Hallucination in Large Vision-Language Models](https://arxiv.org/abs/2305.10355). Li, Du, Zhou, Wang, Zhao & Wen, 2023. EMNLP 2023. Most vision-language models tested described objects that were not in the image, especially objects that commonly appear alongside what is there. Introduces POPE, a more stable way to measure it.
98. [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://arxiv.org/abs/2406.12045). Yao, Shinn, Razavi & Narasimhan, 2024. Agents using tools under written business policies succeeded on fewer than half of the tasks, and solved the same retail task in all of eight attempts less than 25% of the time.
99. [Measuring AI Ability to Complete Long Software Tasks](https://arxiv.org/abs/2503.14499). Kwa, West, Becker et al. (METR), 2025. NeurIPS 2025. The length of task, measured in human working time, that frontier agents complete half the time has doubled roughly every seven months since 2019; around 50 minutes at the time of the study.
100. [Dual-Use Foundation Models with Widely Available Model Weights](https://www.ntia.gov/issues/artificial-intelligence/open-model-weights-report). National Telecommunications and Information Administration, US Department of Commerce, 2024. 30 July 2024. Found the evidence insufficient to justify restricting open weights, recommended monitoring instead, and listed the benefits, including running a model without sharing data with a third party.
101. [Open-weight models lag state-of-the-art by around 3 months on average](https://epoch.ai/data-insights/open-weights-vs-closed-weights-models). Epoch AI, 2025. 30 October 2025. On a composite of many benchmarks, the best open-weight models trailed the best closed models by about 3.5 months on average (90% interval 1.1 to 5.3 months), and at times the gap closed entirely.

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt