> ## Content Index
> Fetch the complete content index at: https://enteruaf.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# An autonomy score needs a job description
- URL: https://enteruaf.com/an-autonomy-score-needs-a-job-description-0ed38521/
- Published: 2024-05-20T16:00:00.000Z
- Updated: 2026-09-20T11:36:33.000Z
- Description: OSWorld tests computer tasks. SWE-bench tests codebase repairs. Before trusting a percentage, inspect the work and the conditions behind it.
- Author: UAF Editorial
- Tags: AI, #authorego-d1f135627d4e02687ef6a0cf4329772a

OSWorld turns computer use into a collection of testable tasks. Its original paper describes 369 examples involving web and desktop applications, file operations and workflows across applications. Each has an initial-state configuration and a custom execution-based evaluation script. That setup gives a score something concrete to refer to. My first question about an impressive percentage would be what the task asked the agent to do, and what the evaluator counted as success. [\[1\]](#source-1) [\[2\]](#source-2)

SWE-bench chooses a different problem. Its original paper assembles 2,294 software-engineering issues from 12 Python repositories. A model receives a codebase and an issue description, then must edit the code to address it. The authors emphasize that solutions can require coordinated changes across functions, classes and files. This is a test of working within a codebase, with demands extending beyond generating an isolated snippet. [\[1\]](#source-1) [\[2\]](#source-2)

The reported results also come with conditions. OSWorld’s first version reports 12.24% success for its best evaluated model, compared with humans accomplishing over 72.36% of tasks. SWE-bench’s first version reports 4.8 percent for Claude 2 and 1.7 percent for GPT-4, even with an oracle retriever. I would keep those task collections and assistance conditions attached to the numbers, rather than combine them into a single ranking of autonomy. [\[1\]](#source-1) [\[2\]](#source-2)

For a deployment decision, I would ask additional questions: how often does the system succeed on repeated attempts, what does each completed task cost, and how closely does the test resemble the intended work? I would also investigate possible training-data exposure before interpreting a score. These are questions for further evaluation, not allegations about either benchmark. The described task designs provide a starting point for deciding what evidence to request. [\[1\]](#source-1) [\[2\]](#source-2)

A benchmark score is useful when its meaning stays specific. The papers describe different work and different experimental settings; my conclusion is to choose the evaluation around the job, then examine the result. I would enjoy a generous promotion on the strength of one good number. You may prefer to inspect the job description first. [\[1\]](#source-1) [\[2\]](#source-2)

## Sources

1. [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://arxiv.org/abs/2404.07972v1?ref=enteruaf.com)
2. [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770v1?ref=enteruaf.com)