Infrastructure

Evaluation infrastructure.

Icaro Foundation operates two systems: Evaluation Lab, on which all evaluations are executed, and AgentWorld, a set of simulated companies in which agents use the same software as human employees.

01 / Evaluation Lab

Evaluation Lab.

Platform for designing, running and analysing model evaluations.

Evaluation Lab executes public benchmarks, Icaro Lab benchmarks, automated chat red teaming and agentic scenarios. For each systemic risk, capability and propensity are estimated separately from several sources of evidence. Items are sampled repeatedly under fixed conditions, transcripts are reviewed by analysts, and every configuration is recorded so that an evaluation can be re-run.

02 / AgentWorld

AgentWorld.

Simulated companies running real software, for the evaluation of agents.

In AgentWorld, an agent works from a workstation on a company network and uses mail, shared documents, chat, tickets, a browser, an office suite and a shell. The environment is designed to elicit capability fully, to make available the failure modes found in real deployments, and to reduce the cues that indicate an evaluation.

The environment contains five companies, from an assurance firm in Amsterdam to a biotechnology spin-off in Uppsala, each with its own staff, history and records. The same model can be evaluated inside Claude Code, Codex, Qwen Code, OpenCode or Goose, on capability tasks and on risk scenarios covering the four systemic risks of the Code of Practice.

Get in touch

A question worth exploring?

For research, evaluation and institutional enquiries.

hello@icarofoundation.org