Multimodal Model · 2026
SAM-MM
Structured Action Model — Multimodal
A multimodal model that perceives vision, audio, and physical dynamics, then reasons and acts. Runs entirely offline.
What SAM-MM does
SAM-MM is a multimodal model that reasons jointly over text, image frames, audio, and predicted physical dynamics. It extends SAM-G's dual-mode design to perception: given a scene, a sound, or a physical sequence, it produces either a step-by-step reasoning trace or a structured control action.
The model includes a latent physics world-model that predicts how a scene will evolve. This is not a decorative component: the physics reasoning benchmarks confirm it encodes real dynamics. From an eight-frame sequence, SAM-MM correctly identifies the governing physical law and predicts outcomes.
SAM-MM-Audio-Reasoning extends the base model with audio understanding, enabling the model to identify sound sources, match sounds to observed motion, and trigger control actions from audio events. It incorporates real environmental audio (ESC-50) for transfer to real-world sound conditions.
Input modalities
Text
Natural language instructions and questions
Vision
Image frames at 96×96 resolution
Audio
Log-mel spectrograms for sound reasoning
Physics
Latent world-model predicting scene evolution
Benchmark results
Perception and physics reasoning are saturated at 100%. The physics results are the central evidence that the world-model encodes dynamics rather than texture: from an eight-frame sequence the model names the governing dynamic and predicts outcomes correctly. Cross-modal control families are weaker, with one known dataset-design gap noted below.
| Capability | Modality | Score |
|---|---|---|
Physics: governing dynamic | Vision + Physics | 100% |
Physics: physical outcome prediction | Vision + Physics | 100% |
Vision: direction of motion | Vision | 100% |
Vision: read a number (OCR) | Vision | 100% |
Control: sensor reading to action | Vision | 42% |
Control: frame to robot action Dataset design gap — targets not derivable from input frame | Vision | 0% |
| Aggregate | 73.6% |
Audio Extension
SAM-MM-Audio-Reasoning
The audio-complete variant adds four new reasoning families: sound identification, audio-to-motion matching, sound-triggered control, and real environmental audio classification. Results for the audio families are reported on the HuggingFace model card as evaluation completes.
| Capability | Modality |
|---|---|
| Audio: identify sound source | Audio |
| Audio: match sound to motion | Vision + Audio + Physics |
| Audio: sound to control action | Audio |
| Audio: real environmental sounds (ESC-50) | Real audio |
Positioning
Large vision-language models outperform SAM-MM on raw optical character reading, as expected from scale. The relevant comparison is capability per parameter and offline deployability rather than absolute accuracy.
Vision-language-action models for robotics (RT-2, OpenVLA, Octo, and similar) are not directly comparable: they emit continuous motor commands evaluated by task-success rollouts in simulation or on hardware. SAM-MM shares the on-device, offline, low-latency niche while targeting structured reasoning and control-action emission rather than motor control.
SAM-MM operates at two to three orders of magnitude fewer parameters than the models it is positioned alongside, while delivering saturated performance on perception and physics reasoning tasks.
Model lineage
Foundation model. Dual-mode structured action generation across ten domains.
Multimodal extension. Adds vision, audio, and a latent physics world-model.
Audio-complete variant. Adds sound identification, audio-to-motion matching, and sound-triggered control.
Citation
@misc{samm2026,
title = {SAM-MM: A Multimodal Model with a Latent
Physics World-Model for Offline Edge Reasoning},
author = {Amega Mike, AMEFORGE Lab},
year = {2026}
}