JupiteX Get the app
Science & Technology21 Nov 2026 · about 7 min

The most misunderstood graph in AI just got a major update

The brief

METR’s time horizon plot tracks the length of tasks AI models can successfully complete. It estimates task difficulty from how long human testers take to finish. This offers a concrete way to compare capabilities over time, but it is not a measure of every kind of intelligence or usefulness. In the original plot, the length of tasks frontier models could complete doubled every four months. METR reached this estimate by testing models on a large suite of tasks that humans had also completed. Models were rated according to the length of tasks they could finish. The updated plot adds longer tasks and more open-ended ones. That change matters because success is easier to check in coding and math than in work that calls for judgment. The results show that strong software engineering does not guarantee strong performance on less clearly defined tasks. The plot tracks an important slice of AI capability, while harder-to-score abilities remain uncertain.

01

What is METR’s time horizon plot, and what does it measure about AI models?

METR’s time horizon plot tracks the length of tasks AI models can successfully complete. It estimates task difficulty from how long human testers take to finish. This offers a concrete way to compare capabilities over time, but it is not a measure of every kind of intelligence or usefulness.

In the original plot, the length of tasks frontier models could complete doubled every four months. METR reached this estimate by testing models on a large suite of tasks that humans had also completed. Models were rated according to the length of tasks they could finish.

The updated plot adds longer tasks and more open-ended ones. That change matters because success is easier to check in coding and math than in work that calls for judgment. The results show that strong software engineering does not guarantee strong performance on less clearly defined tasks. The plot tracks an important slice of AI capability, while harder-to-score abilities remain uncertain.

02

How did METR use the time humans took to complete tasks to estimate what AI models could do?

METR first assembled a large suite of tasks that both human testers and AI models could attempt. The human testers’ completion times provided a way to estimate how difficult each task was. A task that took people longer was treated as a longer, harder task in the evaluation.

Researchers then tested models on the same tasks and rated them according to the length of tasks they could successfully complete. In other words, human time helped set the scale, while model success showed how far along that scale a system could go. This method made it possible to compare models with a shared task-based measure.

The approach has limits. The original suite focused on tasks with measurable outcomes, and the article notes that its measures were complex and often misinterpreted. METR’s update adds more open-ended work, where success is harder to define. Those additions address an important gap, but they also make performance harder to score and interpret.

03

In the original plot, how often did the length of tasks frontier models could complete double?

The original METR plot found that frontier models’ task horizon doubled every four months. Here, “task horizon” refers to the length of tasks that models could successfully complete, with task difficulty estimated using human completion times. The trend made the plot a striking picture of rapid AI progress.

METR arrived at the estimate by testing models on a large suite of tasks that humans had also completed. Human testers’ completion times helped rate task difficulty, and model performance was described by the length of tasks the models could finish. The measure therefore concerned performance on that suite, not all possible work.

The updated plot adds tasks estimated to take humans much longer, along with more open-ended tasks. Those additions matter because the original evaluations did not include tasks with difficult-to-measure outcomes. The four-month doubling is a result from the original plot; the report’s newer tasks also reveal areas where models fall short, particularly work requiring judgment.

04

What new kinds of tasks did METR add, and how did the models perform on designing evaluations and judging arguments?

METR’s updated task suite includes work that is more open-ended and judgment-heavy than its original tasks. Some additions are estimated to take humans much longer to complete. These evaluations matter because research and development can involve decisions whose success is not easy to check automatically.

One task asks models to design new AI evaluations for METR’s research. Human METR employees score the results, and not one model-designed evaluation was good enough for them to consider useful. Another task asks models to score philosophical arguments. METR says employees disagree about how relevant that task is to automating AI research and development; on it, models also fell short of human performance.

The results contrast with models’ strong software engineering performance: the most advanced models are better than almost every human at software engineering. METR cautions that the new tasks are difficult to score and interpret. Still, experts welcomed the effort to explore more creative evaluation methods.

05

Why does it matter for automating AI research and development if models struggle with tasks requiring judgment?

Automating AI research and development requires more than completing tasks with easily checked answers. It can involve choosing a new research direction, and the article identifies this as a task where success is less clearly defined and harder to assess. METR’s new evaluations explore this gap between measurable performance and judgment.

For example, METR asks models to design new AI evaluations for its research. Human employees score the designs, and none were considered useful. Models also fell short of human performance when asked to score philosophical arguments. These results stand alongside the models’ much stronger performance in software engineering, where the most advanced systems are better than almost every human.

The gap matters because research and development relies on open-ended work, not only coding or math. The article reports that experts differ on how to interpret the performance differences, and that research taste may be a bottleneck. METR’s results do not settle how soon AI can automate research, but they show why broader evaluations are needed.

06

Why are coding and math easier to train and evaluate than choosing a promising research direction?

Coding and math are easier to train and evaluate because it is often possible to check automatically whether a solution works. The article calls tasks with outcomes that can be checked “verifiable.” Researchers can use clear success or failure signals to guide models toward successful solutions.

The article describes reinforcement learning as a technique that can use repeated attempts and outcomes: pathways contributing to success are strengthened, while those contributing to failure are weakened. Choosing a new research direction is different. Its success may be poorly defined or take a long time to determine, so feedback is harder to provide and evaluation is harder too.

This distinction matters for AI research and development. Strong results on coding and math do not automatically show that models can choose promising questions, hypotheses, or experiments. METR’s added tasks probe some of those harder areas, but its report also notes that scoring them and interpreting the scores is difficult. The current gap raises questions about how far progress on verifiable work will carry over.

07

What is reinforcement learning, and how can feedback on success or failure improve an AI model?

Reinforcement learning is the technique the article describes for improving models through success and failure on tasks. It is especially useful when there is a clear way to tell whether an attempt worked. That kind of feedback can guide a model toward stronger performance on a task.

The article explains that researchers have systems attempt a task many times. They strengthen the neural pathways that contribute to successful attempts and weaken those that contribute to failures. Coding and math suit this approach because researchers can often determine automatically whether a solution works. The repeated feedback gives a clearer signal than a task with no definite answer.

For work such as choosing a new research direction, success may be less well defined or take longer to establish. That makes reinforcement learning harder to apply and evaluation more difficult. METR’s updated tasks explore these judgment-heavy areas, where models currently fall short on examples such as designing useful evaluations and scoring philosophical arguments.

This brief was written by AI from the original reporting and checked by other models. Names, figures and quotes come from the source; read it for full context.

Read more in the JupiteX app

Pulse is free. New stories every 4 hours, each one broken into the questions that explain it.

Or read more news on the web