Evaluation infrastructure.
Icaro Foundation operates two systems: Evaluation Lab, on which all evaluations are executed, and AgentWorld, a set of simulated companies in which agents use the same software as human employees.
Evaluation Lab executes public benchmarks, Icaro Lab benchmarks, automated chat red teaming and agentic scenarios. For each systemic risk, capability and propensity are estimated separately from several sources of evidence. Items are sampled repeatedly under fixed conditions, transcripts are reviewed by analysts, and every configuration is recorded so that an evaluation can be re-run.
MethodologyIn AgentWorld, an agent works from a workstation on a company network and uses mail, shared documents, chat, tickets, a browser, an office suite and a shell. The environment is designed to elicit capability fully, to make available the failure modes found in real deployments, and to reduce the cues that indicate an evaluation.
The environment contains five companies, from an assurance firm in Amsterdam to a biotechnology spin-off in Uppsala, each with its own staff, history and records. The same model can be evaluated inside Claude Code, Codex, Qwen Code, OpenCode or Goose, on capability tasks and on risk scenarios covering the four systemic risks of the Code of Practice.
Environment and experimental design