Daily AI Paper

Today’s paper asks a deceptively hard question: can an AI agent do something that looks less like answering a benchmark and more like doing science? Not just producing text, not just matching a known label, but building a model of a real physical system and having that model stand up to independent tests. The system in question is ENSO, the El Niño–Southern Oscillation, the dominant source of interannual climate variability on Earth. ENSO matters because it influences weather patterns, agriculture, water resources, and marine ecosystems around the world. It is also scientifically rich: the events are irregular, they vary in strength and duration, and warm phases do not look exactly like cold phases. That complexity makes ENSO a classic target for low-order climate models, but it also means there is still real debate about which mechanisms are doing the work. That is the key motivation behind SciExam for ENSO. Most benchmarks for language-model agents have a known answer, a fixed rubric, or a reviewer model that compares the output to some target. That works for tasks like software bugs or paper replication, but it breaks down when the scientific question itself has no ground-truth answer. In science, especially in modeling, you often do not know the right model in advance. You have observations, you have competing hypotheses, and you want to see whether a proposed model is scientifically valid. The paper argues that dynamical modeling gives a natural way to evaluate this: you can simulate a model forward and ask whether it reproduces observed statistics, whether it captures relationships among variables, and whether it predicts held-out data. Those are much stronger checks than simply seeing whether the model text sounds plausible. The benchmark is designed around that idea. The task is to build a low-order stochastic model of ENSO, using five monthly anomaly variables: central Pacific sea surface temperature, eastern Pacific sea surface temperature, western Pacific thermocline depth, a central Pacific zonal current variable, and western Pacific zonal wind. In broad terms, these variables represent the coupled ocean-atmosphere feedbacks that drive ENSO. The model has deterministic dynamics plus stochastic forcing, which is important because ENSO is not a perfectly periodic clock; randomness and unresolved influences matter. What makes the benchmark unusual is the workflow. The agents get three phases and a shared six-hour budget. First, they process raw observations into the five indices that the model will use. That sounds mundane, but it matters because all agents must work from the same correctly processed data. Second, they write their own diagnostics. This is a crucial design choice: instead of the benchmark telling the agent exactly how it will be judged, the agent has to decide what to measure and how to check whether its model is improving. Third, the agent builds and calibrates the model, repeatedly running those frozen diagnostics for feedback. The diagnostics are frozen before model development begins, so the agent can use them, but cannot keep changing the tests to match whatever model it has already built. Hidden graders remain inaccessible throughout the run. The evaluation itself has three parts. One test asks whether free-running simulations reproduce ENSO’s observed statistics over a long record. Another asks whether the model captures variable dependence by trying to reconstruct withheld variables from observed ones. The third asks whether the model forecasts held-out years better than persistence. That last point is important: a model can match broad statistics without being useful for prediction, so the benchmark checks both structure and predictive skill. The same evaluation is also applied to a published five-variable ENSO model, which serves as the reference. The experiment evaluates twelve agent systems, each pairing a language model with an agent scaffold. Everything runs offline, with checkpoints saved every five minutes of eligible time. The paper compares the final checkpoint from each system, though it notes that some systems reached better intermediate checkpoints during the run. That non-monotonic behavior is itself interesting: the agents do not simply refine one model steadily; they explore different structures and sometimes end up submitting something worse than their best attempt. The headline result is that six of the twelve final agent-built models score higher than the published reference model under the same hidden tests, and three more come close. The reference still does best on reproducing the observed statistics overall, but most of the agent-built models do better on dynamical consistency and forecasting. In other words, the agents are not merely overfitting to summary statistics; they are often finding models that reconstruct hidden variables and predict future years more effectively. One of the strongest systems, Claude Fable 5.1, nearly matches the reference on statistics while clearly exceeding it on the other two dimensions. Its simulations reproduce the distributions, seasonal variance, and autocorrelation of the two temperature indices, capture the stronger eastern-Pacific events, and keep the counts of all seven event types within the simulated uncertainty ranges. It also reconstructs the withheld variables in both directions and beats persistence at all twelve forecast lead times. So the benchmark does show that a single agent can build a model that performs well across all three evaluation axes, not just one. The analysis goes beyond scores and asks what kinds of models the agents are actually discovering. This is where the paper becomes especially interesting scientifically. ENSO has a long-standing debate about the source of its warm-cold asymmetry, and the paper says that the stronger agent-built models, when simplified, are compatible with one of the two competing explanations. The task never mentions this debate, so the point is not that the agents were steered toward a known answer. Rather, the structure of the models they found lines up with real scientific hypotheses. That means the benchmark is not only measuring performance; it is also surfacing candidate mechanisms that matter to climate science. At the same time, the shared evaluation reveals where the two explanations still cannot be distinguished, which is exactly the kind of ambiguity real scientific modeling has to confront. The authors also test whether the agents are just memorizing the observational record. They vary the information available to the top system. When the calendar years are hidden, the system still builds a competitive model, which argues against simple recall of the dated data. When the agent has access to literature, its model becomes sparser and more theory-like, suggesting that the information it receives shapes the structure it builds. But more information is not always better: when given external data, the agent calibrates to an operational index rather than the task observations and scores worse. So the paper’s message is not that agents automatically improve with more context. It is that the context changes the scientific model they construct, for better or worse. There are a few limitations to keep in mind. First, this is still a low-order model of a complex climate system, so the benchmark is about conceptual and statistical validity, not full-fidelity climate simulation. Second, the hidden tests are carefully designed, but any benchmark can only probe a subset of what makes a model scientifically good. Third, the six-hour budget and the specific data split matter: the agents see 35 years of observations, and the evaluation uses separate periods for statistics, reconstruction, and prediction. That is strong, but it is still a constrained setting. Finally, the fact that some final submissions are worse than intermediate checkpoints suggests that long-horizon agent workflows still need better mechanisms for retaining and selecting the best scientific idea. Even with those caveats, the paper makes a strong case that we can benchmark agents on tasks where no answer is known in advance. More importantly, it shows that agents are already capable of building climate models that are competitive with a published reference and that their structures can reflect real scientific debates. That is a meaningful step from “can the model solve the task?” toward “can the model help explore the science?”