# Multimodal AI

> How a model reads scanned documents, charts and recordings directly, what it gains over converting them to text first, and where it still sees things that are not there.

- Web page: https://www.alesanalytics.com/explainers/multimodal-ai
- Markdown: https://www.alesanalytics.com/explainers/multimodal-ai.md
- Last updated: 2026-10-31
- Publisher: Ales Analytics

Models that read pages, images and audio, not only text.

## In one paragraph

A multimodal model reads more than text. It takes in images, scanned pages, charts, diagrams, tables, audio and in some cases video, and answers questions about them in ordinary language. Early systems attached a separate image reader to a language model. Newer ones are trained on several kinds of input together from the start, so they can reason across them, for example by reading a chart and the paragraph that discusses it as a single piece of evidence.

## Why it matters

Most of what an organisation knows is not tidy text. It sits in scanned contracts, signed forms, engineering drawings, slide decks, photographs of equipment and recorded calls. A model that reads only text can use that material after something else has converted it, and every conversion loses detail: a table flattened into a line of numbers, a handwritten correction or a stamp lost altogether. A model that reads the page as it is can use the layout, the figures and the handwriting directly.

## How it works

- Each kind of input is turned into the same internal form the model uses for text, so an image or a stretch of audio becomes a sequence the model attends to alongside the words of the question.
- Models trained across image, audio, video and text from the start report reasoning that crosses between them, such as answering a question about a chart from the chart itself rather than from a description of it.
- The images in serious documents are varied. One standard benchmark of college-level questions draws on thirty kinds of image, including charts, diagrams, tables, maps, sheet music and chemical structures.
- An image or a page takes up far more of the model's input than a sentence of text, so reading a long scanned document is slower and more expensive than reading the same document as text.

## What the evidence says about the limits

- Reading a picture is not the same as understanding it. On that benchmark of college-level questions, the strongest models at the time of publication answered fewer than six in ten correctly, which is why results on your own documents are measured rather than assumed.
- Vision-language models describe things that are not there. A peer-reviewed evaluation found that most of the models tested named objects absent from the image, particularly objects that usually appear alongside what is actually shown.
- An answer drawn from an image therefore needs the same discipline as one drawn from text: a reference back to the page it came from, and a check on anything that will be acted upon.

## What it looks like in practice

Consider a supplier's scanned delivery note with a handwritten correction and a stamp. A text pipeline converts it into words and loses the correction. A multimodal model reads the page as it is, picks up the corrected quantity and answers a question about it, with a reference to the page so a person can see the original. Where the figure feeds a payment, it is checked against the system of record rather than taken from the model's reading alone.

## How this connects to our work

Multimodal models are used where your information lives in scans, drawings and forms rather than clean text, which is the work of [document intelligence](https://www.alesanalytics.com/capabilities/document-intelligence). Every value links back to the page it came from, and figures that matter are checked against your [system of record](https://www.alesanalytics.com/explainers/system-of-record).

## Read the original

1. [Gemini: A Family of Highly Capable Multimodal Models](https://arxiv.org/abs/2312.11805). Gemini Team, Google, 2023. A model family trained across image, audio, video and text from the start, reporting cross-modal reasoning and improved results on all 20 multimodal benchmarks examined.
2. [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://arxiv.org/abs/2311.16502). Yue, Ni, Zhang et al., 2024. CVPR 2024. About 11,500 college-level questions across 30 subjects, with 30 kinds of image including charts, diagrams, tables, maps and chemical structures. The strongest models at publication scored 56% and 59%.
3. [Evaluating Object Hallucination in Large Vision-Language Models](https://arxiv.org/abs/2305.10355). Li, Du, Zhou, Wang, Zhao & Wen, 2023. EMNLP 2023. Most vision-language models tested described objects that were not in the image, especially objects that commonly appear alongside what is there. Introduces POPE, a more stable way to measure it.

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt