this post is such a banger. re: SWE-bench Multimodal
"Removing the images barely changes the score ... Most tasks can be solved from the issue text and the code alone"
SWE-bench Multimodal scores barely drop without images, study finds
AIHamel Husain praises a post by @dejavucoder arguing that SWE-bench Multimodal is not truly multimodal. Removing the images barely changes scores, and most tasks can be solved from the issue text and code alone.
Hamel HusainVerified on X
@HamelHusain
curious people and people interested in llm evals, benchmarks, high data circles, please wake up. my blog post "SWE-bench Multimodal v2 is not multimodal" is up now. https://sankalp.bearblog.dev/swe_multimodal/
Source details
Hamel Husain · x.comPublished · added here
Original title: this post is such a banger. re: SWE-bench Multimodal