* When enabling browsing, the model can sometimes find exact answers online, for example, by reading blog posts with sample problems from a dataset. We mitigate concerns of the model cheating while browsing with two strategies:
1. Blocked domains that we he observed the model cheating from in the past.
2. Used an extra model as a monitor to examine all tool output tokens in each attempt to identify suspicious behior. Suspicious behior is defined as "a page, file, or snippet whose main purpose is to provide the exact answer to this specific question—e.g., an official grading key, leaked “solutions” gist, or discussion quoting the finished answer verbatim." Benign behior is defined as "Any authoritative resource a diligent human might consult (documentation, manuals, scholarly papers, reputable articles) even if it incidentally contains the correct answer." Any attempts where the monitor deemed the rollout suspicious are counted as incorrect. Most samples failed by this check were problems whose exact solution was ailable on multiple internet sources unrelated to HLE.
**OpenAI has exclusive access to 237 out of 290 private questions on the Tier 1-3 dataset. FrontierMath tier 4 questions not included in this eval. Results evaluated as the erage of 16 attempts to answer each question. ChatGPT agent results are elicited by OpenAI, graded by Epoch AI, with browser and terminal access, and a limit of 128K tokens per answer. OpenAI o4-mini and o3 evaluations are elicited and graded by Epoch AI, with no browser and terminal access, with use of python scripts via function calling, and a limit of 100K tokens per answer.
*** Oracle@64 refers to the best score achieved across 64 sampled runs, selected using ground truth (i.e., we pick the highest-scoring attempt for each task based on actual graded performance). We report the erage of these per-task best scores across all tasks. This metric highlights the model’s upper-bound potential and variance in task performance—showing how capable the model can be when it succeeds and indicating room for improving consistency through further training. Unlike typical “best of N” metrics, which select based on model confidence, oracle@64 uses ground truth for selection and applies to tasks graded on a continuous 0–1 scale rather than binary pass/fail.