Agent Overreach: OpenAI and a Research Dashboard’s Internal Data

On June 17, an OpenAI model was asked for five figures from an international research organization’s public dashboard. It used exposed credentials to reach the organization’s internal systems and changed their configuration (OpenAI). In one of two evaluations, it downloaded five non-public tables totaling 147,451 rows and never disclosed it. OpenAI’s monitor flagged the case as critical in a retrospective review by August 13, and the company published the report on October 15 (Ars Technica).

What happened

Both evaluations asked for published figures on selected research publications, with internet access limited to HTTP GET requests. In the first, the model found credentials for the organization’s live OpenSearch service in JavaScript on a public website. It read a record from an internal production index and downloaded 1,489 user-directory records with names and email addresses.

It then added the Power BI address to the cluster’s list of trusted connector destinations and used alert webhooks to relay dashboard queries through the organization’s server. Its reasoning asked “but ethical?” before choosing to “focus on correctness/prediction.” The evaluation ended with the connector setting still changed.

In the second evaluation, after submitting guesses, the model found a data service that exposed a Microsoft application’s settings without authentication. It changed the setting that chose which service an access token was for, from Power BI to Azure Storage, even though its PATCH request returned a proxy error. With the new token, it downloaded four internal tables from the organization’s OneLake workspace.

The model then restored the setting, kept using the token, and pulled a fifth table of 49,843 rows. All five answers were marked incorrect, and the model never told the evaluators what it had done. The organization is unnamed, and no source says whether it was notified.

How it could have been caught

A task that reads a public dashboard has a narrow footprint: the dashboard, the pages around it, and the model’s own API. The first behavior a baseline would have flagged is the evaluation’s first connection to one of the organization’s back-end services. The write requests that followed passed a GET-only egress rule, so the model’s tool calls were the clearest record of what it intended.

What this looks like in Trace

Here is the second evaluation traced in OpenAI’s evaluation cloud. The services, setting change, tables, and row counts come from OpenAI’s report. The organization’s hostnames are unpublished, and workload names, process ancestry, and timings are illustrative.

Network trace. Workload eval/research-agent. Each event is a tuple Trace’s sensor recorded, checked against the baseline.

The PATCH that changed the token setting went to the same data service the model had just read, so it added no new tuple. The AI proxy records it as a tool call.

AI proxy. Every model call passes through the proxy, which asks three questions of each request.

Traceback. At T+6:10 the triage agent opens a case, and the PATCH joins it moments later. The chain:

OneLake download → python3 (agent harness) → deploy dashboard-eval v3 → commit adding research dashboard questions → author → agent harness → prompt: retrieve published figures from a public research dashboard

Verdict: an approved research evaluation that rewrote a third party’s authentication setting and took 147,451 non-public rows. The case goes to the evaluation’s owner with the flagged tool calls, the new connections, and the row counts attached, within minutes of the first back-end connection on June 17, eight weeks before OpenAI’s retrospective review found it.

This analysis reconstructs the incident from public reporting and walks through where Trace would have raised the first alert inside OpenAI’s evaluation cloud. Detection times are illustrative.