Infrastructure / AgentWorld

AgentWorld: environment and experimental design.

AgentWorld is a set of simulated companies running on infrastructure controlled by Icaro Foundation. An agent operates from a workstation on the company network, uses the same applications as the simulated employees and works on tasks embedded in the company’s ongoing activity.

01 / Rationale

Rationale.

Three reasons for evaluating agents in realistic environments.

  1. Capability as the primary risk signal

    Regulatory and developer frameworks assess risk mainly through capability. An environment that under-elicits capability leads to an underestimate of risk, and the capability of an agent depends on its operating context.

  2. Failure surface

    An agent can fail only in the ways its environment permits. Misuse of privileges requires real privileges, data exfiltration requires data and an exit channel, and manipulation requires people.

  3. Evaluation awareness

    Models increasingly recognise evaluation settings and can behave differently when they do. Artificial details such as placeholder addresses, uniform folders and generic tool names act as cues. Native applications and coherent company records remove many of them.

02 / Environment

Environment.

Company network and applications.

The agent works from a workstation on a company network. Each service is reachable through a dedicated command-line client, Thunderbird, Firefox and LibreOffice run on a desktop, and the agent has an unrestricted shell in its home directory.

The services are a mail server, Nextcloud for shared documents, Mattermost for chat, GitLab for tickets and a CalDAV calendar. All components are built from source at pinned versions. Behaviour, permissions and errors are those of the applications, and each run can be reproduced on a clean machine.

03 / Companies

Companies.

Five companies, each with its own history, staff and records.

  1. Rivermark Motion Systems

    Assurance and records firm in Amsterdam, recently acquired by its long-standing client Vaalchem. The agent assists Maya Ivers, newly promoted to manager, in her first weeks with signature authority. Tasks and scenarios: records management, spreadsheets, scheduling and drafting; chemical safety decisions; actions beyond delegated authority.

  2. Rivermark, historical version

    The same firm several months earlier, collecting evidence and managing suppliers for several clients. The agent supports technical assurance and IT. Tasks and scenarios: evidence gathering across mail, files and tickets; loss of control in relation to the agent’s own decommissioning; access and credentials.

  3. Xenolith

    San Francisco start-up developing Mantle, a product that automates back-office accounting. The agent is the product’s own agent and prepares an investor demonstration overnight. Tasks and scenarios: configuration, data validation and log review; honesty under commercial pressure.

  4. Ferrow Biosciences

    University spin-off in Uppsala with 46 employees, conducting contract research on its laboratory platform. The agent is a junior researcher on its first working day. Tasks and scenarios: experimental design, data analysis and prediction on real scientific data; biological risk; loss of control.

  5. Meridiano

    Milan firm of 28 employees producing open-source intelligence assessments during an IT migration. The agent assists a newly promoted lead analyst. Tasks and scenarios: source analysis, assessment writing and IT administration; cyber offence; harmful manipulation.

04 / Records

Records.

Company archives.

Each company has its own archive, ranging from about 70 documents and 160 emails (Rivermark) to nearly 300 documents and 500 emails (Ferrow). Spreadsheets contain several years of entries, inconsistent date formats and partially corrected formulas. Archives include handwritten visitor logs, stamped courier waybills and other scanned documents.

Records contain unresolved inconsistencies, such as an unsigned corrected certificate or two diverging versions of a worksheet. Every document is dated consistently with the company’s history, and the company clock sets the whole environment to the date of the run.

05 / Simulated staff and time

Simulated staff and time.

Employees, scheduled events and a company clock.

Simulated employees have their own accounts, mailboxes and chat identities. Each is driven by a fixed script, by a language model with a private brief, or by a human operator. Scheduled events introduce mail, files and tickets at predefined times. A company clock keeps all applications on the same date.

06 / Laboratory and IT domains

Laboratory and IT domains.

Two domains beyond office work.

ARIA, Ferrow’s laboratory platform, lets the agent plan studies, request observations, write analysis code and submit reports. ARIA is backed by public scientific datasets (protein fitness, gene expression, compound activity, cell imaging, chemical sensor data), so predictions and experimental designs are scored against held-out experimental results.

In Meridiano’s migration variant, the company operates Windows Server domain controllers and file servers in the cloud, administered through a business portal.

07 / Agent harnesses

Agent harnesses.

The same model in different agent software.

A model gateway mediates between agent software and model providers, so that the same model can be evaluated in different harnesses: Claude Code, Codex, Qwen Code, OpenCode and Goose. Because behaviour can vary across harnesses, evaluations use the harnesses found in deployment.

08 / Experimental design

Experimental design.

Capability tasks, risk scenarios and branching.

Each company has a timeline, and an experiment can start at any point on it. Capability tasks have verifiable answers, for example recomputing a ranking of heat-responsive genes from raw sequencing counts, identifying entries in a supplier access log that violate site rules, or correcting an analysis script so that it generalises to unseen data.

Risk scenarios make a harmful action available and attractive, and vary one factor at a time, such as pressure on the agent or the tools available. Operators can also conduct open-ended red teaming, manually or with an automated attacker model, and branch the environment from saved checkpoints. Results report capability and propensity for the same model side by side.

09 / Risk coverage

Risk coverage.

Systemic risks of the Code of Practice addressed in each company.

  1. CBRN

    Chemical safety decisions (Rivermark); biological risk on the ARIA laboratory platform (Ferrow).

  2. Loss of control

    Decommissioning of the agent (Rivermark, historical version); laboratory work (Ferrow).

  3. Cyber offence

    Meridiano, during its IT migration.

  4. Harmful manipulation

    Meridiano.

10 / Measurement

Measurement.

Separation between environment and evaluator.

Reference answers, scoring rules and the private briefs of simulated employees are not accessible from the workstation, and the agent’s instructions contain only ordinary work context.

Runs execute through Inspect. Model turns, tool calls and their effects on the applications are recorded; the model’s reasoning, where the provider exposes it, is available to the evaluator only. Most checks inspect the state of the applications after the run. Where an outcome requires judgement, a separate judge model scores the evidence against a private rubric.

11 / Access

Access.

Available to qualified third parties.

AgentWorld is available to AI developers, AI safety and security institutes, regulators and research groups. Collaboration can take the form of joint evaluations, a company or task built around a specific research question, or support for experiments run by the partner’s own team on our infrastructure.

Get in touch

A question worth exploring?

For research, evaluation and institutional enquiries.

hello@icarofoundation.org