Multimodal AI: aligned inputs, provenance, and modality-specific failure

Multimodal systems combine representations from text, images, audio, video, or other signals. Joint processing can enable new tasks but does not guarantee grounding: preprocessing, alignment, missing context, hidden instructions, accessibility, and modality-specific bias remain.

Mental model

• CLIP learns image-text representations through contrastive natural-language supervision and reports transfer on selected datasets. Describe learned alignment without claiming universal visual understanding.
• Flamingo studies interleaved visual and textual inputs and few-shot adaptation across evaluated tasks. Treat multimodal in-context behavior as empirical and model-specific.
• The GPT-4V system card documents capabilities, limitations, and safety evaluations for one deployed vision model. Keep perceptual uncertainty and high-stakes limitations visible.
• NIST's Generative AI Profile covers risks across synthetic content, information integrity, privacy, and lifecycle testing. Apply provenance, evaluation, and incident controls across modalities.

Guarantees and boundaries

• Each decoded feature is linked to its source file and region or time span.
• Missing modalities and alignment failures are explicit.
• Extracted text, metadata, and model output remain untrusted.

Diagram scenarios

Aligned multimodal request. Each modality is validated and linked to source timing or region before fusion.

Missing or conflicting modalities. One source can be absent, corrupted, or disagree with another; fusion must not silently invent alignment.

Hidden multimodal prompt injection. Text embedded in images, audio, documents, or metadata can attempt to override policy.

Evaluate each modality and interaction. Aggregate accuracy can hide failures by modality, quality, language, accessibility need, or demographic slice.

Architecture decision

Preserve file provenance and alignment, scan and bound each modality before decoding, evaluate modality and subgroup slices, treat extracted text and metadata as untrusted, and use explicit abstention or human review where ambiguity is material. Do not infer sensitive attributes or high-impact decisions without governed evidence.

Failure modes

• OCR, ASR, frame sampling, and alignment errors compound.
• Hidden instructions can cross modality boundaries.
• Aggregate metrics can conceal accessibility and subgroup failures.

Operational checklist

• Pin every preprocessor, encoder, alignment rule, and model version.
• Test corrupt, missing, low-quality, multilingual, adversarial, and conflicting inputs.
• Preserve region and timestamp citations for review.
• Avoid unsupported sensitive-attribute inference and require human review for material ambiguity.

Related concept

Primary sources

1. https://arxiv.org/abs/2103.00020
2. https://arxiv.org/abs/2204.14198
3. https://cdn.openai.com/papers/GPTV_System_Card.pdf
4. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf