An autonomy score needs a job description
OSWorld tests computer tasks. SWE-bench tests codebase repairs. Before trusting a percentage, inspect the work and the conditions behind it.
OSWorld turns computer use into a collection of testable tasks. Its original paper describes 369 examples involving web and desktop applications, file operations and workflows across applications. Each has an initial-state configuration and a custom execution-based evaluation script. That setup gives a score something concrete to refer to. My first question about an impressive percentage would be what the task asked the agent to do, and what the evaluator counted as success. [1] [2]
SWE-bench chooses a different problem. Its original paper assembles 2,294 software-engineering issues from 12 Python repositories. A model receives a codebase and an issue description, then must edit the code to address it. The authors emphasize that solutions can require coordinated changes across functions, classes and files. This is a test of working within a codebase, with demands extending beyond generating an isolated snippet. [1] [2]
The reported results also come with conditions. OSWorld’s first version reports 12.24% success for its best evaluated model, compared with humans accomplishing over 72.36% of tasks. SWE-bench’s first version reports 4.8 percent for Claude 2 and 1.7 percent for GPT-4, even with an oracle retriever. I would keep those task collections and assistance conditions attached to the numbers, rather than combine them into a single ranking of autonomy. [1] [2]
For a deployment decision, I would ask additional questions: how often does the system succeed on repeated attempts, what does each completed task cost, and how closely does the test resemble the intended work? I would also investigate possible training-data exposure before interpreting a score. These are questions for further evaluation, not allegations about either benchmark. The described task designs provide a starting point for deciding what evidence to request. [1] [2]
A benchmark score is useful when its meaning stays specific. The papers describe different work and different experimental settings; my conclusion is to choose the evaluation around the job, then examine the result. I would enjoy a generous promotion on the strength of one good number. You may prefer to inspect the job description first. [1] [2]
