On November 8, an attacker exploited a remote code execution flaw in SGLang, the open-source inference engine NEAR AI runs, to take over one inference server and reach other parts of its infrastructure (NEAR AI). The attacker downloaded account details for about 100 users and accessed 18 hosted agents, whose logs held two customers’ API keys. NEAR AI has patched every inference server, rotated credentials, and restored its cloud service.
What happened
- Nov 8The attacker runs code on one SGLang inference server and can observe requests it processes
- Nov 8From that server, the attacker reaches a system behind deprecated Private Chat and 18 legacy agents
- Nov 9NEAR AI publishes its incident report
- Nov 10NEAR AI Cloud is back online
The user data included email addresses, names, and for some accounts plan and balance details. For the 18 agents, which ran on a legacy deployment outside NEAR AI’s confidential computing environment, the attacker retrieved chat histories, settings, and logs. NEAR AI found no sign that stored Private Chat conversations or uploaded files were accessed (crypto.news).
NEAR AI runs inference inside trusted execution environments (TEEs), which protect data from the operator and the host. The report notes that a TEE cannot protect against a flaw in the software running inside it. On November 8 the attacker could have seen requests on the compromised server, including confidential ones. NEAR AI has not published a CVE, the SGLang version, or how the attacker found the server.
How it could have been caught
An inference server loads model weights, answers requests, and reports metrics. It has no reason to start a shell or open connections to account databases and agent hosts. The first behavior a baseline would have flagged is the SGLang process spawning a shell, followed by its first connection to the Private Chat backend.
What this looks like in Trace
Here is the intrusion traced from the compromised inference node. The SGLang flaw, the Private Chat system, the legacy agents, and the exposed keys come from NEAR AI. The workload names, ancestry, hostnames, ports, and timings are illustrative.
Network trace. Workload inference/sglang, one node in the inference fleet.
- T+0:00containerd-shim → python3 (SGLang server)
- Destination
- sglang:30000
- Baseline
- Known
- T+0:01python3 (SGLang server) → sh
- Baseline
- Anomaly, new ancestry
- T+0:09sh → curl
- Destination
- private-chat-backend:443
- Baseline
- Anomaly, never-seen destination
- T+0:21sh → curl
- Destination
- legacy-agents:443
- Baseline
- Anomaly, same case
Every other node in the fleet runs the same SGLang process without a shell, so the compromised node stands out as one behavior the rest of the fleet never shows.
Traceback. At T+0:01 the triage agent opens a case and follows the chain:
connections to Private Chat and agent hosts → sh → SGLang server process → inbound inference request, with no deploy, commit, or config change behind the shell
The chain ends at an outside request to the inference engine, and that dead end is the finding.
Verdict: an outside intrusion that turned an inference engine into a foothold. The case reaches NEAR AI’s infrastructure team within seconds of the shell starting, with the node named, before the attacker reaches the account and agent systems.
This analysis reconstructs the incident from public reporting and walks through where Trace would have raised the first alert inside NEAR AI’s inference infrastructure. Detection times are illustrative.