AmeforgeAmeforge

Multimodal Model · 2026

SAM-MM

Structured Action Model — Multimodal

A multimodal model that perceives vision, audio, and physical dynamics, then reasons and acts. Runs entirely offline.

4
Input modalities
100%
Physics reasoning (saturated)
73.6%
Aggregate exact match
Offline
No cloud dependency

What SAM-MM does

SAM-MM is a multimodal model that reasons jointly over text, image frames, audio, and predicted physical dynamics. It extends SAM-G's dual-mode design to perception: given a scene, a sound, or a physical sequence, it produces either a step-by-step reasoning trace or a structured control action.

The model includes a latent physics world-model that predicts how a scene will evolve. This is not a decorative component: the physics reasoning benchmarks confirm it encodes real dynamics. From an eight-frame sequence, SAM-MM correctly identifies the governing physical law and predicts outcomes.

SAM-MM-Audio-Reasoning extends the base model with audio understanding, enabling the model to identify sound sources, match sounds to observed motion, and trigger control actions from audio events. It incorporates real environmental audio (ESC-50) for transfer to real-world sound conditions.

Input modalities

01

Text

Natural language instructions and questions

02

Vision

Image frames at 96×96 resolution

03

Audio

Log-mel spectrograms for sound reasoning

04

Physics

Latent world-model predicting scene evolution

Benchmark results

Perception and physics reasoning are saturated at 100%. The physics results are the central evidence that the world-model encodes dynamics rather than texture: from an eight-frame sequence the model names the governing dynamic and predicts outcomes correctly. Cross-modal control families are weaker, with one known dataset-design gap noted below.

CapabilityModalityScore
Physics: governing dynamic
Vision + Physics100%
Physics: physical outcome prediction
Vision + Physics100%
Vision: direction of motion
Vision100%
Vision: read a number (OCR)
Vision100%
Control: sensor reading to action
Vision42%
Control: frame to robot action
Dataset design gap — targets not derivable from input frame
Vision0%
Aggregate73.6%

Audio Extension

SAM-MM-Audio-Reasoning

The audio-complete variant adds four new reasoning families: sound identification, audio-to-motion matching, sound-triggered control, and real environmental audio classification. Results for the audio families are reported on the HuggingFace model card as evaluation completes.

CapabilityModality
Audio: identify sound sourceAudio
Audio: match sound to motionVision + Audio + Physics
Audio: sound to control actionAudio
Audio: real environmental sounds (ESC-50)Real audio

Positioning

Large vision-language models outperform SAM-MM on raw optical character reading, as expected from scale. The relevant comparison is capability per parameter and offline deployability rather than absolute accuracy.

Vision-language-action models for robotics (RT-2, OpenVLA, Octo, and similar) are not directly comparable: they emit continuous motor commands evaluated by task-success rollouts in simulation or on hardware. SAM-MM shares the on-device, offline, low-latency niche while targeting structured reasoning and control-action emission rather than motor control.

SAM-MM operates at two to three orders of magnitude fewer parameters than the models it is positioned alongside, while delivering saturated performance on perception and physics reasoning tasks.

Model lineage

SAM-G

Foundation model. Dual-mode structured action generation across ten domains.

SAM-MM CURRENT

Multimodal extension. Adds vision, audio, and a latent physics world-model.

SAM-MM-Audio-Reasoning

Audio-complete variant. Adds sound identification, audio-to-motion matching, and sound-triggered control.

Citation

@misc{samm2026,
  title  = {SAM-MM: A Multimodal Model with a Latent
             Physics World-Model for Offline Edge Reasoning},
  author = {Amega Mike, AMEFORGE Lab},
  year   = {2026}
}