Multimodal Models (Vision, Audio, Video)
Multimodal models work with more than text. They can process images, screenshots, audio, video frames, files, or combinations of these.
The basic pattern
text
image/audio/video/file -> encoder -> shared representation -> language model -> answer/action
The model converts non-text inputs into representations the language model can reason over.
Common use cases
| Modality | Use cases |
|---|---|
| image | chart reading, UI screenshots, medical images, document understanding |
| audio | voice agents, meeting summaries, call-center analysis |
| video | scene understanding, training review, surveillance summaries |
| documents | PDFs, slides, forms, invoices |
| screen | computer-use agents and UI automation |
Design questions
Before building, ask:
- Does the task require visual evidence or just OCR text?
- Are citations or bounding boxes needed?
- Is latency important?
- Does the model need to call tools?
- Are images sensitive?
- How will errors be reviewed?
Failure modes
Multimodal systems can:
- misread small text
- hallucinate objects
- miss details outside focus
- overtrust low-quality images
- confuse chart axes
- leak sensitive visual data
- fail on long videos without sampling strategy
Practical pattern
Use a staged pipeline:
text
extract key frames or pages
-> ask targeted visual questions
-> combine evidence
-> answer with citations or screenshots
Do not send huge videos blindly. Select frames and evaluate.
Knowledge check
Q1: Why is OCR sometimes enough?
If the task only needs text from an image, OCR plus text reasoning can be cheaper and easier.
Q2: Why do multimodal systems need special evals?
They can fail visually even when the final text sounds confident.