termEngineering & architecture

Multimodal Model

A multimodal model processes or generates more than one kind of data, such as text, images, audio, or video, within a connected model experience.

Download Markdown (.md)

01What it means

A multimodal model accepts, relates, or produces multiple modalities such as language, images, audio, video, or sensor data. Some models translate between modalities; others combine them in a shared representation. Multimodal capability does not mean every modality is equally accurate, accessible, safe, or available in every product configuration.[1][2]

Several modalities meet in a shared model experience Encode text, image, audio, or video, combine representations, then produce modality-specific output
  1. Modality inputsText, image, audio, video, or sensor data
  2. Modality encodersTransform each format into model representations
  3. Fusion or translationRelate representations across modalities
  4. Shared model reasoningInfer, classify, retrieve, or generate
  5. Output decoderText, visual, audio, action, or structured result
  6. Modality-specific validationQuality, safety, provenance, and accessibility

User correction and evaluation reveal cross-modal failures

Capability and quality can differ sharply by modality and combination; consent, accessibility, hidden content, provenance, latency, and cost require separate checks.

02Why it matters

Multimodal products can support natural workflows such as asking about a screenshot, describing a scene, transcribing a call, or creating visual content. Each added modality expands the input surface, evaluation burden, accessibility needs, privacy exposure, and potential for cross-modal attacks.

03When to use it

Use a multimodal model when combining modalities materially improves a validated task. Prefer specialized components when one modality dominates, requirements are strict, or a smaller pipeline is easier to test. Provide equivalent routes when a modality would otherwise exclude users.

04How it works

  1. Define the cross-modal task, user benefit, supported formats, and modality-specific failure consequences.
  2. Capture or upload inputs with consent, minimization, quality checks, and clear retention rules.
  3. Encode each modality into representations the model can combine or transform.
  4. Generate or classify the result, then validate it with modality-appropriate checks and provenance.
  5. Evaluate combinations, accessibility, privacy, latency, cost, and adversarial content before and after release.

05A practical example

Example

A maintenance app lets a technician photograph a machine label and ask a spoken question. The model proposes a relevant manual section, while optical text, audio transcript, and retrieved source remain visible for correction; safety-critical repair steps still require verified documentation.

06Where it breaks

Performance can differ sharply by modality, language, environment, and demographic group. Images and audio may contain bystanders, location data, or hidden instructions. Large inputs increase latency and cost, and generated descriptions can omit details that matter to disabled users.

Common mistakes

  • Assuming strong text performance transfers unchanged to vision or audio.
  • Adding camera or microphone input without consent, minimization, and deletion controls.
  • Testing each modality alone but not conflicting or malicious combinations.
  • Using generated descriptions as a replacement for authored accessibility content without review.

07Sources

  1. Gemini: A Family of Highly Capable Multimodal ModelsGoogle DeepMind · Research reportGemini: A Family of Highly Capable Multimodal Models
  2. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence ProfileNational Institute of Standards and Technology · Government frameworkArtificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
View saved topics

Start typing to search titles, definitions, categories, and frameworks.