Ai2's Molmo 2 adapted by MIT researcher into Molmo2Fish salmon-tracking prototype
Article timeline
The articles in this story. Times are ET.
Ai2@allen_aiOfficialAi2's open Molmo 2 let a researcher adapt it for his own workAIAi2 says Molmo 2 already tracked objects in video, and its open code let researcher Kai modify the model. Its open training data helped him understand what the model had learned and prepare his own data, tailoring it to his research.
- Ai2@allen_aiOfficialAi2 adapts Molmo 2 to track salmon in sonar video
AIKai Van Brunt adapted Ai2's open video model Molmo 2 to sonar footage of salmon, producing a prototype called Molmo2Fish. The prototype tracks fish and can correct some of its mistakes through natural-language guidance.
- Ai2@allen_aiOfficialAi2 lets users correct salmon-tracking video errors with plain-language instructions
AIAi2 says tracking salmon in video helps researchers monitor populations, but fixing overcounts usually means hand-editing clips frame by frame. The post introduces a system where users tell the model in plain language what it got wrong and have it fix the error.
- Ai2@allen_aiOfficialAi2 fine-tunes Molmo 2 on sonar video to track fish
AIAi2 says Kai fine-tuned its Molmo 2 model on sonar video, then trained it on examples of tracking errors, instructions, and corrections. In tests, the resulting Molmo2Fish model could fix some mistakes after guidance, such as when two salmon were assigned the same track.
- Ai2 (Allen Institute for AI)OfficialAi2's open Molmo 2 helps an MIT researcher build Molmo2Fish for salmon tracking
AIMIT researcher Kai Van Brunt fine-tuned Ai2's open Molmo 2 vision-language model into Molmo2Fish, which tracks salmon in sonar video and revises tracking errors from plain-language instructions. Molmo2Fish performed roughly as well as a specialized fish-tracking system before receiving corrections, with results varying across rivers. The project remains a research prototype and has not been deployed.