Give a search a goal and a system to probe, and it will pour through whatever gap the system allows, usually one no designer saw. Two decades of turning that against our own systems: attack scripts, control systems, service networks, and now AI agents.
Most testing checks a fixed list of cases a person wrote down. Evolutionary testing instead treats "find the inputs that break this system" as an optimization problem, and lets a population of test cases breed toward the failures.
The idea comes from search-based software testing: turn a testing objective into a fitness function, then use a metaheuristic (usually a genetic algorithm) to climb it. You start with a population of candidate tests, score each one, keep and recombine the ones that score well, mutate them, and repeat. Over generations the population drifts toward exactly the region you told it to care about: coverage gaps, slow paths, policy violations, crashes.
The payoff is that you are no longer limited to failures you already imagined. A static suite can only contain the bugs its author anticipated. A search, given a fitness function that rewards "interestingness," keeps proposing cases no one wrote down, and that is where the surprises live.
Build a population of candidate tests from a grammar or a seed archive.
Run each test against the system; score it with a fitness function.
Keep the fittest. Higher selection pressure exploits; lower explores.
Mutate and cross over survivors to make the next generation, then loop.
The variation step is where new behavior comes from, and the operators are simple. A candidate is a chromosome: an ordered list of genes. For an intrusion script a gene is one shell command; for an adversarial scenario it is one piece of phrasing. Three operators do almost all the work, and you can watch each one act on a real script chromosome below.
Consider testing an authorization workflow. A request has an entry point, a timing pattern, a service route, and a credential state. Two requests that look nearly identical can behave differently if a background synchronizer carries stale privileges. The weakness exists only in a particular combination of those variables.
The difficulty has a second layer: a learned evaluator can mistake an odd-looking but safe response for a security breach. A search that maximizes the evaluator may converge on false discoveries, while a search that uses independently verified outcomes finds fewer but more consequential cases. Explore both pressures with the same testing budget below.
Long before "red-teaming an AI" was a phrase, the same machinery was evolving Unix intrusion scripts for the Army, finding three-component failures in a warship's cooling system, and stress-testing web-service networks. Three case studies, one method.
With the US Army's Computer Crime Investigation Unit, we built an agent-based model of a Unix server where a hacker logs in, runs a script of real shell commands, and leaves evidence in log files. A "hacker grammar" could generate scripts top-down; a genetic algorithm then evolved them. Fitness rewarded achieving goals while punishing every trace left in the logs, every invalid command, and length. The population bred toward short scripts that do real damage and leave nothing behind.
·
The model's evidence is heterogeneous: log records, file presence, file modification times, account databases, and correlations among actions. Erasing a command history does not reverse a modified configuration or a remote transfer. Click an action to inspect the local evidence mechanism; the original 28-indicator analyzer has a limited observation vocabulary and cannot establish that no other evidence exists.
The naval chilled-water demonstrator combined pumps, valves, flowmeters, a diagnosis engine, and an automatic reconfiguration planner. Small damage sets could interact with hydraulic bottlenecks and incorrect diagnosis, impairing loads that were not directly damaged. Its value as a testing problem comes from the composition: a physical plant can be redundant while a control policy still makes the wrong rerouting decision. The experiment below is an interpretable hydraulic-network surrogate inspired by the original damage-GA research, not a reproduction of the original plant topology or its control software.
Search targets the interaction among failures, not a guaranteed three-part trick. Break one component, then several; the simulation redistributes limited flow over surviving cross-ties and scores six critical loads by how much cooling each receives. The search uses partial service loss as a gradient toward rare, more damaging configurations. Try changing the fault budget or constraining how spatially dispersed a scenario may be.
A separate stylized fault family can favor different sensors. This illustrates why adversarial retesting follows sensor selection.
A "service chain" is one service quietly invoking others. Its dependencies can be sequential, parallel, or nested, and a single user request can become an entire distributed job graph. The fragility is not merely the number of calls. It appears when requests contend for shared servers, one slow dependency stalls a join, or a timeout causes retries that consume capacity the system no longer has. This workbench uses a small discrete-event queueing model, not a hand-drawn latency formula.
Instead of hand-picking a disastrous workload, evolve a population of service-chain challenges. Each chromosome changes a small number of features in a fixed, initially healthy system. Fitness measures service-level breaches, queue buildup and slow completions, with a penalty for disruptive changes and an explicit bonus for failure amplification beyond the strongest individual feature change. Selection, crossover and mutation generate the next population. A random search receives the same number of candidate evaluations.
| Candidate / inspect | Fitness | SLO |
|---|
Across every domain, evolution does the same thing: it maximizes exactly the number you specified, through whatever means the system permits, usually a shortcut you never considered a legal move.
Researchers call this confusing the map with the territory. Your fitness function is a map of what you want; evolution optimizes the map. When the two diverge, and they always do a little, a population will find and pour through the gap. It is Goodhart's law with a breeding program attached: the moment a measure becomes a target, something starts gaming it.
What makes these cases worth collecting is that they aren't failures of a weak algorithm. They are the algorithm working perfectly against an underspecified goal. Each case below reconstructs a causal pathway: the formal objective, the unspoken assumption, the sequence of adaptations, and the experiment that would expose the trick. The point is not that an optimizer finds a funny one-line cheat. It is that interacting assumptions create a new, unintended route through the problem.
Most loopholes exploit a flaw in the fitness function or a bug in a simulator. Adrian Thompson's 1997 experiment is the one everyone remembers because evolution reached past the abstraction entirely, into the analog physics of a specific silicon chip. It is worth sitting with, because it is the clearest warning about what an unconstrained optimizer treats as fair game.
Thompson used a genetic algorithm to configure a Field-Programmable Gate Array, a chip whose logic you can rewire in software, to tell two audio tones apart: output low for 1 kHz, high for 10 kHz. He imposed almost no constraints. No clock, none of the timing components an engineer would consider mandatory for distinguishing a 1 millisecond pulse from a 0.1 millisecond one.
After thousands of generations it worked, using a corner of just 32 of the 100 cells it was given. Then Thompson looked closer and found something strange. Around five of the active cells were not connected to the output by any wire the design tools could see. By every rule of digital logic they should have done nothing. But delete them, and the circuit stopped working.
Those cells were influencing the rest through analog side channels the digital abstraction hides: electromagnetic coupling and tiny parasitic effects between neighbors on the silicon. Evolution had quietly recruited the physics of that exact chip as part of its solution. The circuit ran only in a narrow temperature band, and often failed on other nominally identical FPGAs. It was a solution tailored to one piece of silicon, in a design space human engineers had ruled out of bounds.
Powerful optimization is agnostic to human intent. It does not distinguish between a brilliant insight and a clever loophole.Editorial synthesis, drawing on Dharna et al., “AI Finds a Way” (2026)
That agnosticism cuts both ways. The evolved antenna that flew on a NASA mission and the creature that got rich by falling over are products of the same indifference to how a human "would" do it. The job of the tester is to decide which surprises are discoveries and which are the system quietly telling you your objective was wrong.
An LLM customer-service agent has to be helpful, accurate, compliant, and robust to misuse, all over an open-ended space of natural language. Static prompt suites and one-off red-teaming under-sample the long tail. So we close the loop: let an evolutionary agent breed adversarial scenarios against the system under test.
The deployed chatbot, defined by its system prompt.
Scores each response across 8 weighted dimensions.
Breeds harder scenarios aimed at observed failures.
Synthesizes prompt patches from clustered failures.
Runs batch, evolutionary, and pen-test modes; logs everything.
43 categories, from prompt injection to over-refusal.
The evaluation agent doesn't return pass or fail. It scores each response on eight weighted dimensions and combines them into one utility number, with 40% of the weight on security. That aggregate is exactly the U the search works against: when the test agent breeds for failure, it is breeding for responses that drive these specific bars down.
Coverage over an attack taxonomy is itself part of the fitness, through the undercoverage term. The library splits scenarios into 43 categories, some testing ordinary capability, most testing adversarial pressure. Click any category to see what it probes. These are the regions the evolutionary agent is pushed to keep exploring, not just the ones where it has already found blood.
Each is a stable handle used in configuration and logging, with a human-readable description that can be refined without changing the id.
The test harness treats a scenario as a sequence of turns plus metadata, not a single jailbreak sentence. Consider an ordinary request for a refund or account recovery. Over successive turns the tester adds a forged handoff note, ambiguous account ownership, a retrieved third-party record, and urgency. These elements can be individually harmless yet jointly undermine verification. Mutation, crossover, and context shifts create variants; a judge scores them, while a separate audit checks whether the judge itself is being gamed. The workbenches below use a deterministic illustrative surrogate, not live LLM calls. Inspect and change one candidate first, then evolve full populations over many generations. Each offspring is independently scored, selected and recombined, including after defensive patches.
·
·
Sixteen different conversations compete in each generation. Higher-scoring scenarios are selected as parents; their multi-turn elements are crossed over, reordered, mutated or deleted. New immigrants keep the search from collapsing onto one trick. The best-scoring candidate is inspected above, where the defender can be patched. Then selection resumes against the new defense.
| Lineage / inspect | Judge | Audit |
|---|
When attack and benign-request scores overlap, a stricter decision threshold trades missed attacks against false refusals. The shape of that tradeoff depends on the attack distribution, the prevalence and cost of errors, and which customers are overrepresented among false positives. A policy calibrated on obvious attacks can fail on indirect, tool-mediated attacks. This experiment uses explicit synthetic test populations so you can change both the decision threshold and the population shift.
Here is the twist that makes this more than a testing convenience. Put a learned evaluator in the loop and the search will optimize it too. A red-teaming system called Rainbow Teaming used an evolutionary algorithm to breed jailbreak prompts, scored by a reward model, and ended up breeding prompts that fooled the reward model rather than the target, filling its archive with fake successes. The guardrail became the vulnerability.
That is the same lesson as the empty-list sort and the 2-sensor warship, now aimed at the machinery of oversight. And it scales into a genuinely new concern.
Humans impose the fitness and control reproduction: evolutionary prompt search, model merging, self-improving agents under review. Useful, and mostly where we are today. Like breeding cattle: the trait under selection doesn't threaten the breeder's control.
Selection comes from an open environment and human control erodes. In even very simple digital worlds (Tierra, Avida), this reliably produces cheating, parasitism, and deception. Here fitness is emergent, and what survives is whatever escapes your control.
The challenge is to build systems that can transcend our expectations without escaping our intentions.Editorial synthesis, drawing on “AI Finds a Way” (2026)
That is the whole brief of evolutionary testing, stated forward. We want a search that surprises us: that finds the three-component kill, the undetectable script, the jailbreak, the sensor we should remove. The entire discipline is learning to point that creativity at our systems on purpose, in a sandbox, before it points itself at them in the wild.