# Multimodal Model

A multimodal model processes or generates more than one kind of data, such as text, images, audio, or video, within a connected model experience.

- Type: term
- Domain: Engineering & architecture
- Level: Intermediate
- Last reviewed: 2026-09-26
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/multimodal-model/>)

## What it means

A multimodal model accepts, relates, or produces multiple modalities such as language, images, audio, video, or sensor data. Some models translate between modalities; others combine them in a shared representation. Multimodal capability does not mean every modality is equally accurate, accessible, safe, or available in every product configuration.

## Why it matters

Multimodal products can support natural workflows such as asking about a screenshot, describing a scene, transcribing a call, or creating visual content. Each added modality expands the input surface, evaluation burden, accessibility needs, privacy exposure, and potential for cross-modal attacks.

## When to use it

Use a multimodal model when combining modalities materially improves a validated task. Prefer specialized components when one modality dominates, requirements are strict, or a smaller pipeline is easier to test. Provide equivalent routes when a modality would otherwise exclude users.

## How it works

1. Define the cross-modal task, user benefit, supported formats, and modality-specific failure consequences.
2. Capture or upload inputs with consent, minimization, quality checks, and clear retention rules.
3. Encode each modality into representations the model can combine or transform.
4. Generate or classify the result, then validate it with modality-appropriate checks and provenance.
5. Evaluate combinations, accessibility, privacy, latency, cost, and adversarial content before and after release.

## A practical example

A maintenance app lets a technician photograph a machine label and ask a spoken question. The model proposes a relevant manual section, while optical text, audio transcript, and retrieved source remain visible for correction; safety-critical repair steps still require verified documentation.

## Where it breaks

Performance can differ sharply by modality, language, environment, and demographic group. Images and audio may contain bystanders, location data, or hidden instructions. Large inputs increase latency and cost, and generated descriptions can omit details that matter to disabled users.

## Common mistakes

- Assuming strong text performance transfers unchanged to vision or audio.
- Adding camera or microphone input without consent, minimization, and deletion controls.
- Testing each modality alone but not conflicting or malicious combinations.
- Using generated descriptions as a replacement for authored accessibility content without review.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/multimodal-model/#visualization-multimodal-model>)

## Sources

1. [Gemini: A Family of Highly Capable Multimodal Models](<https://arxiv.org/abs/2312.11805>) — Google DeepMind · Research report
2. [Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile](<https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf>) — National Institute of Standards and Technology · Government framework

## Related topics

- [Generative AI](<https://wiki.soroush.uk/topic/generative-ai/>)
- [Foundation Model](<https://wiki.soroush.uk/topic/foundation-model/>)
- [Human–AI Interaction](<https://wiki.soroush.uk/topic/human-ai-interaction/>)
