RedLectern topic
Models combining language · images · audio · video · cross-modal representations.
For: Everyone · builders.
After this page you can: Distinguish supported inputs and outputs from demonstrated abilities.
Page reviewed
Part of: Artificial Intelligence
Multimodal AI concerns systems that work with more than one kind of information, such as language, images, speech, or video. Some systems connect representations across modalities; others combine inputs or generate one kind of content from another. A shared representation can support tasks such as matching text to images or retrieving images with text queries. These approaches connect different data, but the term “multimodal” by itself does not say which inputs a system accepts or which outputs it can produce.
A useful way to assess a system is to ask two separate questions: what can be supplied or generated, and what has been demonstrated on the task you care about? A listed input format is not evidence that the system can reliably interpret every detail in it. For example, video analysis may be incomplete or inaccurate, and image interpretation can fail on ambiguous details. Look for task-specific evidence, including the input conditions, the output being assessed, and examples of failure. This guide introduces the core vocabulary, cross-modal representations, and practical checks for making that distinction.
Use Carnegie Mellon’s tutorial to get oriented around representation, alignment, reasoning, generation, transfer, and evaluation.
Start with: Tutorial on Multimodal Machine Learning
A modality is a form of information, such as text, speech, or visual content. In multimodal work, these forms may be represented separately and then connected, or combined in a shared system. “Multimodal” is a broad label: it does not specify whether a system accepts images, audio, or video; whether it can generate any of them; or whether it handles combinations of inputs.
Keep the input and output questions separate. A system may take one kind of content and return another, or accept several kinds of input while producing only text. Even when an interface accepts a file, that fact alone does not establish how well a system performs a particular task on it. Treat the supported format as a capability boundary to check, then look for demonstrations or evaluations of the actual task.
A concise definition of multimodality and examples spanning text, images, audio, and video.
Multimodal model
A model that processes or generates more than one kind of data.
Modality
A form of information, such as text, an image, or audio.
Vision-language model
A model that connects visual information with language, for example by aligning image and text representations.
A system accepts an image and a text prompt, then returns a written answer. Name the input modalities and output modality. What additional evidence would you want before saying it can read small text in images?
The inputs are image and text; the output is text. To assess small-text reading, look for evaluations or examples that specifically test small text under stated image conditions. Accepting an image does not establish that ability.
Separate interface support from task evidence. A system may accept a media file, but that only establishes something about the input path; it does not show that the system fully or accurately interprets the file. For example, a service’s help documentation notes that video analysis can be incomplete or inaccurate, and that image interpretation may be less accurate when an image is unclear.
To judge a claim, identify the exact task and the conditions under which it was tested. Ask what information the system received, what output was judged, and whether the test checks integration across modalities or only an isolated subtask. A recent review of multimodal evaluation describes a gap: many benchmarks focus on isolated tasks and reveal little about whether a model integrates information across modalities. Treat a demo as an example, and seek task-specific evidence before generalizing.
It gives concrete examples of supported image and video inputs as well as limitations in their interpretation.
A product page says a system accepts video and can answer questions about clips. Write two follow-up questions that would help distinguish supported input from demonstrated ability.
Ask what kinds of questions and video conditions were evaluated, and whether the evaluation checked details across the whole clip, including audio and timing. Also ask what errors or limits were observed.
Generative methods; creation and editing workflows; control; evaluation; limitations.
0 subtopicsComputer visionImage and video understanding; recognition; detection; segmentation; document extraction.
0 subtopicsSpeech and audio AITranscription; speech generation; speaker separation; audio understanding; voice imitation.
0 subtopicsNo recent updates are available.
Francesco Camastra and Alessandro Vinciarelli · 2015
This second edition covers representing audio, image, and video data, then introduces machine-learning methods and recognition applications. Its examples offer a foundation for thinking about what media a system handles.