What Multimodal Generative AI Refers To: Architectures

Multimodal generative AI refers to machine learning models capable of processing and integrating information across multiple modalities such as text, images, and audio to produce outputs. In these systems, heterogeneous inputs are processed and combined before a language model predicts subsequent tokens in a sequence.12

Rather than relying on separate pipeline tools for distinct media types, multimodal generative models integrate these inputs into shared contextual spaces. This architectural integration alters both computational workflows and production deployment economics, as media streams are mapped directly into token units subject to distinct platform limits.123

The CMU taxonomy: heterogeneity, fusion, and the six challenges

A 2022 research paper from Carnegie Mellon University outlines the technical foundations of multimodal machine learning. The authors identify three core characteristics defining these systems: heterogeneity, connections, and interactions across differing input streams.1

The authors also identify six core engineering challenges encountered when building multimodal models: representation, alignment, reasoning, generation, transference, and quantification. Managing how information passes across modal boundaries is central to addressing these challenges.1

Model architectures handle this integration through data fusion across three distinct stages depending on where modalities are combined.1

Early fusion encodes modalities into the model to construct a common representation space. Mid fusion combines modalities at intermediate preprocessing stages, while late fusion aggregates modal outputs at the final decision boundary.1

Alongside structural fusion, modern generative systems perform multimodal transduction. In generative modeling, transduction refers to changing meaning across modes, such as converting text descriptions into synthesized images, audio waveforms, or video frames.4

Architectural implementations: LLaVA versus early-fusion Chameleon

Current frontier models employ divergent strategies to integrate visual data with natural language backbones. Two prominent structural approaches are decoupled encoder adapters and native token-level early fusion.56

LLaVA, which stands for Large Language and Vision Assistant, connects an external vision encoder directly to an LLM to build an end-to-end trained large multimodal model. In benchmark evaluations, LLaVA achieved an 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset.5

When fine-tuned on Science QA, the combined synergy of LLaVA and GPT-4 achieved an accuracy of 92.53%. This demonstrates how cross-modal projection layers allow text models to parse visual context effectively without retraining the core transformer weights from scratch.5

Chameleon takes the alternate path as a family of early-fusion token-based mixed-modal models designed to understand and generate images and text in arbitrary sequences. Rather than translating representations through external vision connectors, Chameleon unifies text and visual tokens into a single sequence space from the initial input layer.6

According to published evaluations, Chameleon outperforms Llama-2 on text-only benchmarks while remaining competitive with larger models such as Mixtral 8x7B and Gemini-Pro. Furthermore, Chameleon matches or exceeds the performance of Gemini Pro and GPT-4V in human evaluations for long-form mixed-modal generation tasks.6

Modality token pricing across frontier models

Running multimodal models in production requires budgeting across disparate per-token rates. OpenAI converts images into tokens and charges for them on a per-token basis rather than flat image processing fees, and similar per-token pricing governs audio interactions.3

Published rates reflect standard processing for context lengths under 272K on OpenAI endpoints. Notably, realtime audio tokens carry the highest unit costs, where processing voice on GPT-Realtime-2.1 costs $32.00 per 1M input tokens and $64.00 per 1M output tokens compared to $4.00 input and $24.00 output for text on the same engine.3

From our data: OpenAI gpt-6-astra costs $10 input and $50 output per million tokens, whereas gpt-6-luna runs at $0.10 input and $0.50 output per million tokens (captured 2026-10-13). For 1 million input plus 1 million output tokens, gpt-6-luna totals $0.60 while gpt-6-astra costs $60, exactly 100 times as much.7

Engineering teams orchestrating multi-step pipelines can reference Claude token pricing or evaluate AI agent orchestration topologies to manage compounding inference expenditures across mixed-modal states.7

Production payload and bandwidth constraints

Deploying multimodal applications requires managing network bandwidth and payload restrictions alongside token consumption. On the Claude API, developers can supply images through three supported source types: a base64-encoded string embedded in the request body, a URL reference to an image hosted online, or a file_id returned by the Files API.8

Cloud environments impose tighter architectural constraints. On Amazon Bedrock and Google Cloud, only base64-encoded image sources are supported for Claude endpoints, removing URL referencing and file_id passing options.8

This base64 constraint introduces significant network friction. Resending full base64 image data in multi-turn conversations can significantly increase request size and latency because the complete image bytes are repeated on every turn as context accumulates.8

API gateways also enforce strict volume and dimension boundaries across inference requests.8

Claude limits images to 20 per message on claude.ai, 100 per request on the API for models with a 200k-token context window, and 600 per request on the API for other models. The maximum allowed dimensions per image are 8000x8000 px.8

When sending large visual batches, Claude enforces a stricter rule: if an API request contains more than 20 images, each image must be resized so that neither dimension exceeds 2000 px.8

Per-image payload weight limits diverge between hosting environments. The maximum image size is 10 MB on claude.ai and the direct Claude API, but drops to 5 MB on Amazon Bedrock and Google Cloud. Standard direct Claude API endpoints cap overall request size at 32 MB, a threshold that developers hit quickly when passing uncompressed high-resolution images.8

System design decisions for multimodal deployment

Selecting an architecture depends on your throughput profile and operational envelope. When building document extraction or visual question-answering systems where text remains the primary output, decoupled architectures like LLaVA provide high accuracy while leveraging standard language backbones.5

When interactive conversational voice or cross-modal synthesis is mandatory, early-fusion frameworks like Chameleon or native audio models eliminate pipeline handoffs. For teams deploying on hosted infrastructure like Google Cloud or Bedrock, pre-resizing images below 2000 px and using cached inputs prevents hitting 5 MB payload caps and minimizes recurring latency overhead.368