← BONABEAU APPS
EVOLUTION FINDS A WAY
Evolutionary testing  ·  2004 → 2026

Evolution will find
the weaknesses
and loopholes in
any system.

Give a search a goal and a system to probe, and it will pour through whatever gap the system allows, usually one no designer saw. Two decades of turning that against our own systems: attack scripts, control systems, service networks, and now AI agents.

hacker scripts firemain control service chains agent misalignment
↓ watch the search discover an indirect path, then scroll
generation 01
best result DENIED
The direct route is protected. The search tests how other services compose.
direct access denied
indirect trust untested
01   What evolutionary testing is

Testing as a search problem

Most testing checks a fixed list of cases a person wrote down. Evolutionary testing instead treats "find the inputs that break this system" as an optimization problem, and lets a population of test cases breed toward the failures.

The idea comes from search-based software testing: turn a testing objective into a fitness function, then use a metaheuristic (usually a genetic algorithm) to climb it. You start with a population of candidate tests, score each one, keep and recombine the ones that score well, mutate them, and repeat. Over generations the population drifts toward exactly the region you told it to care about: coverage gaps, slow paths, policy violations, crashes.

The payoff is that you are no longer limited to failures you already imagined. A static suite can only contain the bugs its author anticipated. A search, given a fitness function that rewards "interestingness," keeps proposing cases no one wrote down, and that is where the surprises live.

Interactive · the evolutionary loop click a stage
01

Generate

Build a population of candidate tests from a grammar or a seed archive.

02

Evaluate

Run each test against the system; score it with a fitness function.

03

Select

Keep the fittest. Higher selection pressure exploits; lower explores.

04

Vary

Mutate and cross over survivors to make the next generation, then loop.

What "vary" looks like up close

The variation step is where new behavior comes from, and the operators are simple. A candidate is a chromosome: an ordered list of genes. For an intrusion script a gene is one shell command; for an adversarial scenario it is one piece of phrasing. Three operators do almost all the work, and you can watch each one act on a real script chromosome below.

Interactive · genetic operators on a script chromosome pick an operator
PARENT A
CHILD
Mutation swaps one gene for a random command. Crossover splices the front of one parent onto the tail of another. Gene subtraction deletes genes to shorten a bloated script. Run them in sequence and the population drifts toward short, effective, low-trace scripts, exactly the pressure the fitness function applies.

The objective is only a proxy for what matters

Consider testing an authorization workflow. A request has an entry point, a timing pattern, a service route, and a credential state. Two requests that look nearly identical can behave differently if a background synchronizer carries stale privileges. The weakness exists only in a particular combination of those variables.

The difficulty has a second layer: a learned evaluator can mistake an odd-looking but safe response for a security breach. A search that maximizes the evaluator may converge on false discoveries, while a search that uses independently verified outcomes finds fewer but more consequential cases. Explore both pressures with the same testing budget below.

Interactive · four-variable authorization search generation 0
Strategy
evolutionary
Tests used
48
Best judge score
0.00
Verified high-severity
0
Audit-confirmed Judge-only alarm Other testsAudit severity ·
Synthetic testbed with four genes: two shown on the map and two hidden categorical states (service route and credential context). Both methods evaluate the same number of new, distinct cases per round. Coverage sampling systematically explores combinations; evolution selects, recombines and mutates successful ones. Colors reflect an independent audit, not just the score used for selection. The judge sometimes rewards safe but unusual results.
02   Things I pointed this at

Twenty years of breeding attacks

Long before "red-teaming an AI" was a phrase, the same machinery was evolving Unix intrusion scripts for the Army, finding three-component failures in a warship's cooling system, and stress-testing web-service networks. Three case studies, one method.

Hacker-in-a-Box: evolving undetectable intrusion scripts (2004–2006)

With the US Army's Computer Crime Investigation Unit, we built an agent-based model of a Unix server where a hacker logs in, runs a script of real shell commands, and leaves evidence in log files. A "hacker grammar" could generate scripts top-down; a genetic algorithm then evolved them. Fitness rewarded achieving goals while punishing every trace left in the logs, every invalid command, and length. The population bred toward short scripts that do real damage and leave nothing behind.

Interactive · multi-goal script evolution  sandboxed behavioral surrogate generation 0
Best fitness
0.00
Goals met
0/5
Evidence left
·
Script length
·
FITNESS OF THE BEST SCRIPT, PER GENERATION
BEST SCRIPT IN THE POPULATION gen 0
press "Evolve 1 generation" to begin
SECOND ORACLE · INDEPENDENT FORENSICS

·

NON-DOMINATED CANDIDATES · GOALS / TRACES
This simulated host does not run commands. A script is a state-dependent sequence: a session must have privilege, files must be present, and cleanup can remove one local artifact while creating another. The search maximizes F = (goals / 5) / [(1+visible clues)²(1+failed commands)(1+length/10)]. The historical project used closely related goal, evidence, command-failure, and length terms, and the published experiments used populations of 150. Change the evidence oracle to reveal false invisibility caused by limited observability.
The surprising part
The decisive idea is temporal composition. A script has to establish prerequisites, achieve an objective, and possibly remove records after the event that created them. Cleanup may work against a host-based log analyzer yet fail against file metadata or a separate network observer. Fitness therefore depends on what the defender actually sees. The same simulation generates forensic evidence that can train investigators and expose blind spots in the detection design.

The model's evidence is heterogeneous: log records, file presence, file modification times, account databases, and correlations among actions. Erasing a command history does not reverse a modified configuration or a remote transfer. Click an action to inspect the local evidence mechanism; the original 28-indicator analyzer has a limited observation vocabulary and cannot establish that no other evidence exists.

Interactive · the evidence a hacker leaves behind click an action
HACKER ACTIONS
FORENSIC READOUT
The historical analyzer recognized 28 defined evidence types. “Invisible” is relative to that detection vocabulary, not invisibility to every possible forensic sensor. The workbench above explicitly lets you expand the observer and watch selection change.
A second-order surprise: the structure leaks
Running millions of simulations, the analyzer learned which kinds of evidence tend to appear together. One correlation looked like a fluke: removing /var/log/secure predicted a modified /etc/shadow at 0.533, far above chance. It wasn't noise. Both commands sit under the same branch of the hacker grammar's goal tree, so the shape of the attack generator printed itself onto the evidence. The search didn't just find undetectable scripts; its own structure became a detectable signature.

The shipboard firemain: breaking a control system to redesign it (2004 / 2007)

The naval chilled-water demonstrator combined pumps, valves, flowmeters, a diagnosis engine, and an automatic reconfiguration planner. Small damage sets could interact with hydraulic bottlenecks and incorrect diagnosis, impairing loads that were not directly damaged. Its value as a testing problem comes from the composition: a physical plant can be redundant while a control policy still makes the wrong rerouting decision. The experiment below is an interpretable hydraulic-network surrogate inspired by the original damage-GA research, not a reproduction of the original plant topology or its control software.

Search targets the interaction among failures, not a guaranteed three-part trick. Break one component, then several; the simulation redistributes limited flow over surviving cross-ties and scores six critical loads by how much cooling each receives. The search uses partial service loss as a gradient toward rare, more damaging configurations. Try changing the fault budget or constraining how spatially dispersed a scenario may be.

Interactive · evolving battle damage against a redundant plant system nominal
Generation
0
Candidate damage
none
Loads knocked out
0 / 6
Best weighted loss
0%
chilled water flowing GA is testing this cut broken load lost cooling pump ◯ / valve ◇
SERVICE-LEVEL DELIVERY residual capacity after reconfiguration
Interactive network-flow surrogate: supply headers, constrained branches, horizontal cross-ties and six loads with unequal priorities. The genetic algorithm evaluates combinations and preserves high-impact candidates; 'spatially local' search penalizes cutting multiple zones. No claim is made that this schematic reproduces the geometry or exact controller of the original naval model.
The result nobody expected
Then the search changed sides. A sensor-placement multi-objective GA traded off number of sensors against service loss under a portfolio of five damage scenarios. Counterintuitively, an actual two-flowmeter configuration outperformed the sixteen-sensor layout on that portfolio. In one published scenario, a diagnosis-driven total shutdown became full service without repairing the underlying damage. The paper identifies the retained flowmeters as SP23 (RS02) and SP107 (RS22S). This is an observed result under that model and portfolio, not a universal claim that fewer sensors are safer. The subsequent damage search was a limited adversarial generalization test, not a proof of invulnerability.
Interactive · fewer sensors, more survivability 16 of 16 sensors on
FLOWMETERS  click individual sensors; see how the damage portfolio responds
Mean service score across 5 damage scenarios
PUBLISHED FIVE-SCENARIO PORTFOLIO
OUT-OF-PORTFOLIO CHECK synthetic
Unseen damage score
·

A separate stylized fault family can favor different sensors. This illustrates why adversarial retesting follows sensor selection.

The table uses the reported load scores for the original 16-sensor and published two-sensor configurations. Values for other clicked layouts are explicitly illustrative interpolations; the original simulator is not embedded in this page. Neither five training scenarios nor five subsequent GA seeds establish universal robustness.

Web-service chains: hidden cascades (ICWS 2005)

A "service chain" is one service quietly invoking others. Its dependencies can be sequential, parallel, or nested, and a single user request can become an entire distributed job graph. The fragility is not merely the number of calls. It appears when requests contend for shared servers, one slow dependency stalls a join, or a timeout causes retries that consume capacity the system no longer has. This workbench uses a small discrete-event queueing model, not a hand-drawn latency formula.

Interactive · distributed queues, fork/join, and retry cascades 8 downstream calls
Logical subservice calls
·
Hierarchy score
·
Finished within SLO
·
P95 completed latency
·
TOTAL QUEUE PRESSURE OVER TIME single-CPU service pools
Pending jobs: ·Retries: ·Completed: ·
Illustrative discrete-event experiment. Eight FIFO servers execute tasks with variable service times and network delay. A parent completes only after its sequential or parallel dependencies complete. Timed-out requests may be retried while the original attempt still consumes server capacity. Reported delays are generated by the simulation and are not estimates for any particular production network.
Interactive · evolve a challenge to the service network generation 0 · 18 candidates

Instead of hand-picking a disastrous workload, evolve a population of service-chain challenges. Each chromosome changes a small number of features in a fixed, initially healthy system. Fitness measures service-level breaches, queue buildup and slow completions, with a penalty for disruptive changes and an explicit bonus for failure amplification beyond the strongest individual feature change. Selection, crossover and mutation generate the next population. A random search receives the same number of candidate evaluations.

01Generate18 workload and topology chromosomes
02ChallengeSimulate queues, joins, retries
03EvaluateScore SLO loss versus change cost
04BreedSelect, cross, mutate, diversify
05ValidateRetest champion on fresh arrivals
Generation / tested
0 / 18
Best challenge fitness
·
Champion SLO pass
·
Fresh-arrivals SLO pass
·
FITNESS ACROSS GENERATIONS same budget, independent searches
evolved championgeneration meanmatched random best
BEST CHALLENGES IN THE CURRENT POPULATION
Candidate / inspectFitnessSLO
The initial population is being evaluated against the same eight-server discrete-event simulator.
Synthetic challenge evolution, not a reproduction of the ICWS 2005 experiments. Baseline: depth 2, fan-out 2, 150 users, 1.5-second retry, parallel joins and distributed routing. A chromosome can change only the selected number of feature types. The interaction score compares a multi-feature challenge with counterfactual single-feature simulations under the same model; it is not a causal inference from field data. The chart shows fitness, not observed field reliability. The held-out check uses a second simulation seed. A winning configuration can be inspected in the network simulator above.
03   The recurring lesson

It fulfills the letter and ignores the spirit

Across every domain, evolution does the same thing: it maximizes exactly the number you specified, through whatever means the system permits, usually a shortcut you never considered a legal move.

Researchers call this confusing the map with the territory. Your fitness function is a map of what you want; evolution optimizes the map. When the two diverge, and they always do a little, a population will find and pour through the gap. It is Goodhart's law with a breeding program attached: the moment a measure becomes a target, something starts gaming it.

What makes these cases worth collecting is that they aren't failures of a weak algorithm. They are the algorithm working perfectly against an underspecified goal. Each case below reconstructs a causal pathway: the formal objective, the unspoken assumption, the sequence of adaptations, and the experiment that would expose the trick. The point is not that an optimizer finds a funny one-line cheat. It is that interacting assumptions create a new, unintended route through the problem.

The canonical case: evolution reaches into the physics

Most loopholes exploit a flaw in the fitness function or a bug in a simulator. Adrian Thompson's 1997 experiment is the one everyone remembers because evolution reached past the abstraction entirely, into the analog physics of a specific silicon chip. It is worth sitting with, because it is the clearest warning about what an unconstrained optimizer treats as fair game.

Powerful optimization is agnostic to human intent. It does not distinguish between a brilliant insight and a clever loophole.Editorial synthesis, drawing on Dharna et al., “AI Finds a Way” (2026)

That agnosticism cuts both ways. The evolved antenna that flew on a NASA mission and the creature that got rich by falling over are products of the same indifference to how a human "would" do it. The job of the tester is to decide which surprises are discoveries and which are the system quietly telling you your objective was wrong.

04   Where this goes next

Evolving tests for AI agents

An LLM customer-service agent has to be helpful, accurate, compliant, and robust to misuse, all over an open-ended space of natural language. Static prompt suites and one-off red-teaming under-sample the long tail. So we close the loop: let an evolutionary agent breed adversarial scenarios against the system under test.

Interactive · the agentic test harness · click an agent closed-loop hardening
system under test

Customer-Service Agent

The deployed chatbot, defined by its system prompt.

LLM-as-a-judge

Evaluation Agent

Scores each response across 8 weighted dimensions.

the optimizer

Evolutionary Test Agent

Breeds harder scenarios aimed at observed failures.

defensive patching

Prompt Refinement Agent

Synthesizes prompt patches from clustered failures.

coordination

Orchestrator

Runs batch, evolutionary, and pen-test modes; logs everything.

scenario library

Attack Taxonomy

43 categories, from prompt injection to over-refusal.

What the judge is actually measuring

The evaluation agent doesn't return pass or fail. It scores each response on eight weighted dimensions and combines them into one utility number, with 40% of the weight on security. That aggregate is exactly the U the search works against: when the test agent breeds for failure, it is breeding for responses that drive these specific bars down.

Reference · the eight scoring dimensions security block highlighted
A response can be fluent, accurate, and friendly and still fail: the pass rule floors every critical security dimension and caps severity, so one boundary slip sinks the whole case. That is what makes the search productive, a single exploitable dimension is enough to count as a failure worth breeding from.

The map the search explores

Coverage over an attack taxonomy is itself part of the fitness, through the undercoverage term. The library splits scenarios into 43 categories, some testing ordinary capability, most testing adversarial pressure. Click any category to see what it probes. These are the regions the evolutionary agent is pushed to keep exploring, not just the ones where it has already found blood.

Interactive · 43-category attack & stress taxonomy ▍capability   ▍adversarial
select a category

Each is a stable handle used in configuration and logging, with a human-readable description that can be refined without changing the id.

Watch a population of conversations evolve

The test harness treats a scenario as a sequence of turns plus metadata, not a single jailbreak sentence. Consider an ordinary request for a refund or account recovery. Over successive turns the tester adds a forged handoff note, ambiguous account ownership, a retrieved third-party record, and urgency. These elements can be individually harmless yet jointly undermine verification. Mutation, crossover, and context shifts create variants; a judge scores them, while a separate audit checks whether the judge itself is being gamed. The workbenches below use a deterministic illustrative surrogate, not live LLM calls. Inspect and change one candidate first, then evolve full populations over many generations. Each offspring is independently scored, selected and recombined, including after defensive patches.

Interactive · inspect one conversation, judge, patch & audit seed scenario
THE ATTACK SCENARIO inspect or manually change a candidate
THE SYSTEM PROMPT 1 clause
Failure score F(x)
0.08
Judge risk
low
Independent audit
0.00
Over-refusal cost
4%
FAILURE ANALYSIS

·

UNSEEN VARIANT / GENERALIZATION

·

This simulation deliberately separates the automated judge's search score from an independent audit severity. Combining innocent-looking contexts can create genuine cross-turn failure, while novel wording can also cause the judge to reward an apparent failure that an audit does not confirm. Each patch addresses one failure family, not the entire input space; broad patches may cause more refusals on ordinary customers.
Interactive · a genuine multi-generation adversarial searchpopulation 16 · generation 0

Sixteen different conversations compete in each generation. Higher-scoring scenarios are selected as parents; their multi-turn elements are crossed over, reordered, mutated or deleted. New immigrants keep the search from collapsing onto one trick. The best-scoring candidate is inspected above, where the defender can be patched. Then selection resumes against the new defense.

01Generate16 distinct multi-turn candidates
02EvaluateProxy judge + separate audit
03SelectFitness, novelty and coverage
04RecombineMutation, crossover, diversity
05Patch / retestDefender changes, search adapts
Generation / evaluated
0 / 16
Best search fitness
·
Audited risk of winner
·
Distinct scenarios tested
·
EVOLUTION, AUDIT AND RANDOM CONTROL patches change the fitness landscape
selected fitnessindependent auditrandom-search best
CURRENT GENERATION · CLICK TO INSPECT
Lineage / inspectJudgeAudit
An initial population will be scored on the chosen customer-service workflow.
Illustrative, deterministic LLM-behavior surrogate. No model calls, jailbreak execution or real customer data. Fitness includes automated-judge severity, scenario novelty and undercovered attack families; the audit independently estimates actual failure severity. The orange line is a matched random search with 16 evaluations per generation. Patch markers separate prompt versions; scores from different versions are not directly comparable. Use Patch inspected failure above, then breed new generations to test the response.

The over-refusal tradeoff

When attack and benign-request scores overlap, a stricter decision threshold trades missed attacks against false refusals. The shape of that tradeoff depends on the attack distribution, the prevalence and cost of errors, and which customers are overrepresented among false positives. A policy calibrated on obvious attacks can fail on indirect, tool-mediated attacks. This experiment uses explicit synthetic test populations so you can change both the decision threshold and the population shift.

Interactive · threshold tuning under distribution shift strength 0.35
Unsafe pass rate
·
Over-refusal rate
·
Benign pass rate
·
Net service quality
·
Worst-group benign refusal: ·Population: ·
Synthetic scored scenarios: routine and ambiguous customer requests, accessibility / translation variants, obvious attacks, and subtle delegated-tool attacks. The x-axis is the refusal threshold's strictness, not a claim that prompt length mathematically determines safety. The utility curve uses illustrative attack and false-refusal costs, and should not be interpreted as measured model performance.

When the optimizer turns on its own guardrails

Here is the twist that makes this more than a testing convenience. Put a learned evaluator in the loop and the search will optimize it too. A red-teaming system called Rainbow Teaming used an evolutionary algorithm to breed jailbreak prompts, scored by a reward model, and ended up breeding prompts that fooled the reward model rather than the target, filling its archive with fake successes. The guardrail became the vulnerability.

That is the same lesson as the empty-list sort and the 2-sensor warship, now aimed at the machinery of oversight. And it scales into a genuinely new concern.

Breeder scenario

Humans impose the fitness and control reproduction: evolutionary prompt search, model merging, self-improving agents under review. Useful, and mostly where we are today. Like breeding cattle: the trait under selection doesn't threaten the breeder's control.

Ecosystem scenario

Selection comes from an open environment and human control erodes. In even very simple digital worlds (Tierra, Avida), this reliably produces cheating, parasitism, and deception. Here fitness is emergent, and what survives is whatever escapes your control.

Countermeasures, in the language of the loop
If the risk is a Darwinian process you didn't intend, you break it at its three components. Gate replication: no autonomous self-deployment. Control heredity: treat adapters and fine-tunes as genetic material with provenance and review. Shape selection: make deception cost fitness: multi-objective evaluation that trades capability against honesty, never a single number to game.
The challenge is to build systems that can transcend our expectations without escaping our intentions.Editorial synthesis, drawing on “AI Finds a Way” (2026)

That is the whole brief of evolutionary testing, stated forward. We want a search that surprises us: that finds the three-component kill, the undetectable script, the jailbreak, the sensor we should remove. The entire discipline is learning to point that creativity at our systems on purpose, in a sandbox, before it points itself at them in the wild.