Replit says top AI models' builds fail where features interact

AIReplit says about three-quarters of top AI models' failed builds pass every single-feature check and break only where features meet. It gives four examples: a draft pull request hiding its merge conflict, comments unable to attach to text inside table cells, undo erasing a teammate's change, and a server owner without roles unable to moderate.

Replit ⠕Official

@Replit

View on X

Part of a thread · earlier post

Every feature works on its own. Together, they break.

~3/4 of the top models' failed builds pass every single-feature check. They break where features meet:
▪️ Code hosting: a draft PR never shows its merge conflict.
▪️ Document editor: nobody can comment on a word inside a table cell.
▪️ Design tool: undo can erase a teammate's change, not just yours.
▪️ Chat: a server owner with no roles can't moderate anyone.

Source details

Replit ⠕ · x.comPublished Oct 28, 5:22 PM ET

Original title: Every feature works on its own. Together, they break.

Topics

AI agents AI coding Papers & research Evals & benchmarks