James Shore proposes randomized with-AI and without-AI tests to measure AI's software productivity

1. 6:54 AM
   O'Reilly RadarBlog

   Measuring AI's impact on software delivery speed requires controlled comparisons

   AIJames Shore argues that delivery speed is only a component of software productivity, which he defines as value produced divided by money spent. He says estimates and story counts cannot show whether AI makes teams faster, so teams should be compared to themselves using randomized, interleaved with-AI and without-AI tasks. This is part 2 of his series, published on O'Reilly Radar.

1. 6:54 AM
   O'Reilly RadarBlog

   Are you getting your money's worth from AI? James Shore outlines how to measure it

   AIJames Shore's essay series argues engineering leaders need hard data on AI's costs and benefits before cutting staff budgets. He cites a CIO who measured a 2.1x productivity gain while top AI users spent $21,000 per month on tokens. The series covers delivery speed, unintended consequences, business outcomes and a CFO note, and warns against measuring lines of code or pull requests.

The trend compares only continuously observed participants, so its scope may be smaller than the current heat total. Hover or tap for hourly heat; use arrow keys with a keyboard.