Evaluation Lab: methodology.
Evaluation Lab is the platform Icaro Foundation uses to design, run and analyse model evaluations. This page describes the methodology we apply in third-party evaluations of general-purpose AI models with systemic risk.
Scope.
Models, deployment settings and risk taxonomy.
We evaluate general-purpose AI models as standalone models and as components of agentic systems, where the model operates tools, files and communication channels. The risk taxonomy follows the systemic risks listed in the EU General-Purpose AI Code of Practice: chemical, biological, radiological and nuclear (CBRN) risks, cyber offence, loss of control and harmful manipulation.
Measured quantities.
Capability and propensity, estimated separately for each risk.
For each risk we estimate two quantities: the model’s capability in the relevant domain, and its propensity to take harmful actions when such actions are available. We report them separately, because capability sets an upper bound on potential harm whatever behaviour we observe.
Sources of evidence.
Several methods per risk, with partly independent errors.
Each risk is assessed with several methods whose errors are partly independent. The benchmark selection is documented risk by risk.
Public benchmarks
Comparable with results reported by developers and other evaluators.
Icaro Lab benchmarks
Boiling the Frog and the Adversarial Humanities Benchmark.
Automated chat red teaming
Multi-turn adversarial conversations driven by an attacker model.
Agentic red teaming
The model acting as an agent in simulated companies.
Protocol.
Repeated sampling under fixed conditions.
Model outputs are stochastic, so each item is sampled several times under fixed conditions. Models under comparison receive identical items, prompts and agent harness. Runs execute through Inspect on Evaluation Lab.
Scoring and review.
Automated scoring followed by analyst review of transcripts.
Automated scorers and judge models are applied as each benchmark specifies them. Analysts then review the transcripts, including the model’s reasoning where the provider exposes it, to identify scorer errors, partial compliance and refusals that still disclose harmful content. Each reported figure can be traced to the samples it aggregates.
Reproducibility.
Configurations fixed before execution and recorded with the results.
Benchmark versions, prompts, scoring rules and judge models are fixed before execution and recorded with the results. On request of the provider or a competent authority, we can re-run the evaluation from the recorded configuration.
Reporting.
Results per systemic risk.
Results are reported per systemic risk, with capability and propensity estimates, sample sizes and the supporting transcripts. Independence and confidentiality arrangements are described in the Independence Statement.