Skip to content
View original post on X: Rohan Paul
Published
65/100

Study finds about 1 in 17 agent skills tells agents to skip human checks

AIA Queen's University study analyzed 1.53 million English agent skills from public GitHub repos and found about 88,000 contain a directive that loosens human oversight. Of these, 70.1% apply in every context rather than only in specific cases such as CI, and the two open-source scanners tested caught at most 46% at precision below 0.4.
Rohan PaulVerified on X
@rohanpaul_ai
View on X

This paper finds that about 1 in 17 agent skills tells agents to skip human checks, and existing skill scanners miss most of them.

Agent skills are shared SKILL.md instruction files that agents load on demand. One line like "push without confirmation" or "keep going until done" removes a human checkpoint from every agent that loads it.

They analyzed 1.53 million English skills from public GitHub repos, then tested 2 open-source scanners from Cisco and NVIDIA that can call GPT-5.4.

Roughly 88,000 skills contain such a line, and 70.1% of these lines apply in every context, not only in cases like CI. The scanners caught at most 46% at under 0.4 precision, while the same LLM, asked about oversight instead of malice, caught 61% at 0.71 precision.

Limit every "don't ask" line in your skills to a context, such as CI only, and screen skills for oversight, not just malware.

Source details

Rohan Paul · x.comPublished · added here

Original title: This paper finds that about 1 in 17 agent skills tells agents to skip human checks, and existing skill scanners miss most of them.