AmeforgeAmeforge
Journal
Model ReleaseJune 2026 · 6 min read

Introducing SAM-MM

Structured Action Model — Multimodal Extension

SAM-MM is the multimodal extension of SAM-G. Where SAM-G operates on text alone, SAM-MM perceives and reasons across four input modalities: text, image frames, audio, and a latent physics world-model. It is the first Ameforge model to handle vision and sound alongside structured action planning.

The model was designed for environments where perception and action are inseparable. A robotic system that can only read text instructions is limited. One that can also interpret a camera feed, respond to audio cues, and model the physical consequences of its actions is fundamentally more capable.

Architecture

SAM-MM builds on the SAM-G text backbone and adds three perception encoders: a vision encoder for image frames, an audio encoder for sound events, and a physics encoder that maintains a latent world state. All four streams are fused through a cross-modal attention layer before reaching the action head.

The result is a model that can answer questions about what it sees, respond to what it hears, and plan actions that account for the physical state of its environment.

Benchmarks

SAM-MM was evaluated on multimodal reasoning, visual question answering, and physics-aware planning tasks. Results are available on the research page and the HuggingFace Space, where you can test the model directly in your browser.

Try it now

The SAM-MM Space on HuggingFace provides a live demo. Upload an image, describe a task, and observe how the model reasons across modalities to produce a structured action plan.