I’ve promised this post twice now, so here it is: the enemy commander. The thing on the other side of the battlefield that buys tanks, sends infantry after your houses, and flies the other helicopter.
It turned into the hardest problem in the project, and not for the reason I expected. Making a computer opponent that wins is easy. It can see everything, it never misclicks, and it can react in a sixtieth of a second. The hard part is making one that loses well on easy, fights dirty on hard, and feels like a person in between.
Before I get into what I built, a quick tour of where computer opponents came from. It changed how I thought about the problem.
A Short History of the Guy You’re Playing Against
The first computer opponents weren’t trying to be smart. The paddle in Pong just tracked the ball, with deliberate lag so you could beat it. The aliens in Space Invaders marched in a fixed pattern. The famous speed-up as you cleared the screen wasn’t designed; the hardware simply had fewer sprites to draw. Players read intent into it anyway. That turns out to be the whole story of game AI in one sentence.
Pac-Man was the first one that really understood this. The four ghosts each have a personality: one chases you, one tries to get ahead and ambush you, one is erratic, one wanders off when it gets close. None of them is clever. But because each one is consistent, you learn them, and beating them feels like outwitting somebody rather than beating a pattern.
Through the 90s and 2000s, action games settled on state machines: an enemy is patrolling until it sees you, then attacking until it’s hurt, then fleeing. Bungie’s postmortem on Halo found that players rated the enemies as smarter when they could see what they were thinking: grunts panicking when their elite fell, enemies calling out what they were doing. Their other finding was more surprising. When they made the same enemies tougher without changing their behavior at all, play-testers said they were more intelligent. Difficulty and intelligence get mixed up in a player’s head.
F.E.A.R. in 2005 gave enemies actual planning: goals, actions, and a search for the sequence of actions that gets there. It produced squads that flanked and flushed you out of cover, and it’s still the high-water mark for “the AI did something I didn’t expect and it made sense.” The Sims went a different way with what’s now called utility AI: every possible action gets a score from the current situation, and the highest score wins. No plan, no states, just a lot of weighing. Left 4 Dead’s Director stopped controlling the zombies and started controlling the pacing, easing off when you were struggling and piling on when you got comfortable.
And then the machines started learning. Forza’s Drivatars learned to drive like you. Killer Instinct’s Shadow AI learned to fight like you. And in 2019, DeepMind’s AlphaStar beat professional StarCraft II players after training against copies of itself for the equivalent of two hundred years.
That last one is the tempting path, and I’ll come back to why I didn’t take it.
What the Research Says Now
Somewhere in July, I stopped tweaking numbers and spent a few days actually reading: GDC talks, the Game AI Pro books, a stack of papers. I’d summarize what I took away like this.
Dumb down decisions, never reflexes. If an easy opponent aims badly and reacts slowly, it reads as broken. If it aims fine but occasionally picks the wrong thing to do, it reads as a beginner.
Any single answer gets farmed. If the enemy always does X when you do Y, players will find it and do Y forever. Every response needs at least a couple of variants, and the variant should be picked with dice, not a schedule.
The top difficulty has to mean it. Resident Evil 4 quietly adjusts its difficulty behind your back on normal, and on Professional it stops.
Do the expensive thinking offline. Nothing heavier than a scoring function should run on a phone in the middle of a battle. Search, learning, tuning: all of that belongs on a desk overnight, and the result ships as data.
Now, the game.
What I Started With
I remember, back in the heyday of C, there was something called CRobots. It was a framework for a “robot” that battled other robots. You had a program that ran in cycles, and you could do things like scan, move, aim, fire. You had to find the best balance of running through those. Spend too long scanning and you’re a sitting duck. Spend too much time on the move and you look like a chicken with its head cut off. Basically, it was a priority list.
So my first go-around was a priority list. If the helicopter is down, buy it back. If the moment is right, send the van. If the player is flying, buy anti-air. Otherwise infantry for a house, then tanks with whatever is left. Walk down the list every couple of seconds, buy the first thing that applies. On top of that sat a hand-written pilot for the enemy helicopter, with states for rearming, dogfighting, hunting balloons and airlifting troops, and about fifteen numbers per difficulty tier: how well it aims, how often it fires a missile, how close it’s willing to get.
Five tiers, from Recruit to Ace, each a different set of those fifteen numbers. It worked. It also felt exactly like what it was.
Two Commanders Walk Into a Lab
Here’s where the decision I mentioned in the first post paid for itself. The whole battle in Raid & Rescue is a deterministic simulation ticking sixty times a second; the graphics just watch it. Which means you can take the graphics away.
So I built a lab. It’s a command-line tool with no screen that runs the exact same simulation code the phone runs, with a computer commander on both sides, at about four hundred times real time. It went in two days after the first build of the game, which tells you how early I suspected the commander would be the problem. Point it at every mission, every difficulty tier, every enemy profile, and it grinds through the whole grid overnight and hands back a summary of each battle: who won, how long it took, what each side bought, how many helicopters went down. The summary is the same shape as the one a real player’s phone produces at the end of a battle, which turned out to matter later.
The lab’s first lesson was about the lab, honestly: would it work? The mission clock fails the player when time runs out, because that’s who it was written for. Put a computer on both sides and every battle that went the distance was decided by which seat each commander happened to be sitting in. I had a hundred and twenty-six battles in a row ending at exactly ten minutes, and a win rate that was a coin flip about chair assignment. Now the lab runs a longer clock, swaps the sides every battle to cancel out the map, and reports a stalemate as a stalemate.
The first big sweep after that was six thousand battles. The results were not what I was hoping for.
Same two commanders from the first post. Different table, same steam.
Across the entire ladder, Recruit to Ace, all fifteen of those carefully tuned skill numbers moved the enemy’s win rate from 30% to 42%. Twelve points, for the whole difficulty range. Then I sorted the same battles by a single yes-or-no question: did the enemy capture at least one house? Battles where it didn’t, it won 4% of the time. Battles where it did, 55%.
The knobs I’d been turning for months didn’t matter. What the commander decided to pursue mattered. Strategy, not skill.
The lab also found the bug behind that number. The anti-air rule near the top of the priority list fired forever: the player’s helicopter always respawns and launchers self-destruct when they fire, so the commander bought a launcher, fired it, bought another, and never got to the infantry. On the losing side, over half of everything the enemy bought was anti-air. Smaller things fell out too. The game offered 1,301 supply crates across the sweep and the enemy pilot claimed none of them. I thought I’d taught it crates exist. Apparently not.
And it explained something the testers had already found. The enemy pilot, when it needed to rearm, flew home in a straight line at a constant altitude, and players had learned to park over its route and drop a bomb on it every time. One deterministic answer, farmed exactly as the research said it would be.
Handing Over the Keys
I fixed the four obvious things first: only buy anti-air when there’s actually something in the air, put infantry ahead of launchers, make the pilot weave on the way home, and teach it to notice a bomb falling on it. Four switches, in code. That commander, which I called Mark I (an Iron Man reference), won 52% of its battles. Better than Ace, which had been the hardest tier, and the only profile that lost fewer helicopters than it shot down.
Progress. But four hand-written switches don’t scale, so the rewrite turned the whole idea into data.
The priority list became a utility system, the same idea The Sims used, apparently. Every few seconds the commander scores each thing it could do: fortify, push, buy infantry for that house, save for the van, hold back, run. Each score comes from a handful of the sixteen readings the commander takes off the battlefield (is their helicopter over my ground right now, how many houses are there for the taking, do I have a spearhead, how full is my treasury) pushed through a response curve and multiplied together. The highest score wins. I wanted to be able to tweak enemy commanders on the fly, so the game actually downloads them from the server. No app update needed.
The pilot kept its hand-written states, because they’re good and they’re debugged. But every situation that used to have one answer now has two or three, picked with weighted dice. Withdrawing home is now sometimes a straight run, sometimes a weaving run with the gun firing backward, and sometimes a high road out of your reach.
Then I let the lab tune it. I didn’t write the weights for the next commander; an evolutionary algorithm did, proposing a table, running a batch of battles against the old commanders, keeping what worked and mutating it, for a few hundred generations. Mark II came out of that at 53% wins, timeouts cut in half, and it won about half a minute faster than Mark I.
It also came out with a hole in it. Mark II had learned to almost never buy anti-air. Less than one percent of its purchases. It had discovered, correctly, that none of its sparring partners punished it for that, and evolution is very good at finding what you forgot to test. The fix was to score each candidate by its worst result across several different opponents rather than its average, and the retuned table bought anti-air again. I’d taught it to beat the opponents I had, not the ones it would meet, and the lab is the only reason I caught it before a player did.
Three Commanders
With doctrine as data, the difficulty ladder stopped being one brain with bigger multipliers and became three commanders.
The Rusher buys tanks early and sends the van the moment it has a spearhead. Houses can wait. You beat it by blunting the push and out-earning it.
The Warlord fights the house campaign: infantry-heavy, fortifies what it takes, airlifts troops. You beat it by contesting the houses or starving.
The Air Baron lives in the sky: anti-air budget, balloon denial, an aggressive pilot. You win the air war or you hide from it.
Blunt the push. Contest the houses. Win the air war.
I’d originally pictured something more like Warcraft, rock-paper-scissors: mage beats fighter, fighter beats thief, thief beats mage, so you know what to be. The weighted dice took that away, and I decided I liked it better. Each commander sometimes makes a bad call. But it makes bad calls in character, and that’s what you learn to read.
Each one was tuned separately in the lab, with a guard so evolution couldn’t quietly converge all three onto the same best answer. It didn’t. The Warlord spends sixty percent of its money on infantry, the Rusher eighty-seven percent on tanks, and the Air Baron is the only one that buys anti-air in any quantity.
I gave each of the three personalities a “level”, even though I hate that term, so I call it a band. The bands differ exactly the way the research said they should. The low band commits to one plan and occasionally, on purpose, picks its second-best option, a beginner’s mistake rate of about a third. Its aim is fine. The mid band turns on the mixed responses. The high band never makes a deliberate mistake, switches plans when it sees you countering, and gets the tier’s stat multipliers on top. The top tier gets no softening of any kind.
The Air Baron got one more thing after a tester’s report. I’ve been watching Lanterns, so I call it nerve. Its pilot no longer holds a fixed distance from you. Every tick it weighs how hurt it is against how hurt you are. Take damage and its nerve drops and it backs off out of gun range. Land a hit on you and it comes back in. Getting hurt moves it more than hurting you does, deliberately, or a winning enemy snowballs and never lets up.
The same group of testers found the next gap a week later. When they landed to refuel, the enemy pilot just hovered at its hold distance and waited for them to come back. Everything it should have done already existed in code; it just never got there. So that became another band difference. A low-band pilot uses your pit stop to go home and top up, which also shows a new player that the pad repairs. A high-band pilot ignores you and goes to do damage. No free breathers: push the advantage and keep you busy.
Why Not Just Let It Learn Everything?
Back to AlphaStar. Self-play from scratch is the glamorous answer, and it’s the wrong one for this game. Neural networks pay off when the decision space is enormous, and a commander with five unit types and one pilot in a ten-minute battle doesn’t have that. An evolutionary tuner over a few dozen weights hits the same ceiling for a fraction of the effort.
The tuned numbers also only decide which of the moves I wrote the commander picks, and how often. So I can still promise that a tier-one commander will never rush your base in the first minute. A network that learned everything from scratch makes no such promise, and I’d find out what it had decided when a new player did.
And a battle here is completely determined by its starting number and the player’s inputs, which is how a suspicious leaderboard time gets audited: replay it, confirm it comes out the same. Put a network in the enemy’s head and the AI chips in different phones round the arithmetic differently, one decision drifts, and the check fails. So the learning happens on my desk, and what ships is a table of numbers. No AI engine in the battle. Yet.
What the Summaries Are For
One last piece. I said the lab’s battle summary is the same shape as the one a real player’s phone produces, and that was on purpose.
Lay them side by side and you get a table: for each mission and tier, how real players fared, and how each of the lab’s commanders fared against the same enemy. The column I care about is closest: which computer commander’s results look most like the humans at that tier. That’s the starting point for the next band, and I can tune it toward a target like “beats a mid-tier player about half the time, inside par,” instead of guessing. When a mission is too hard or too easy for the people actually playing it, the fix is an update on the server that the app picks up automatically, and nobody has to install a new version. I still wanted offline play, with no round trips to a server mid-battle, so the enemy commander has to live on your phone.
That’s the enemy commander. It’s still learning. So am I.
So, a question for you: what’s the best enemy AI you’ve ever played against, and what made it feel smart? Box below, as always.
More soon. Next time: the App Store review. I pressed the button in the first post. It turns out that was the easy part.
Credits and Further Reading
None of the ideas above are mine. The people who did the work:
The history
- Pong (Atari, 1972), Space Invaders (Taito, 1978), Pac-Man (Namco, 1980). The ghost personalities are documented in loving detail in Jamey Pittman’s The Pac-Man Dossier.
- Jaime Griesemer and Chris Butcher, The Illusion of Intelligence: The Integration of AI and Level Design in Halo, GDC 2002. The “tougher enemies read as smarter” finding is theirs.
- Jeff Orkin, Three States and a Plan: The A.I. of F.E.A.R., GDC 2006.
- Richard Evans on The Sims’ motive-scoring AI, and Dave Mark’s work on utility AI generally: Behavioral Mathematics for Game AI (2009) and, with Kevin Dill, Improving AI Decision Modeling Through Utility Theory, GDC 2010. The “compensated product” trick the commander uses to score its options comes straight from Dave Mark.
- Michael Booth, The AI Systems of Left 4 Dead, AIIDE 2009, on Valve’s publications page.
- Drivatar: Microsoft Research, Forza Motorsport (Turn 10, 2005 onward). Shadow AI: Iron Galaxy, Killer Instinct (2015).
- Oriol Vinyals et al., Grandmaster level in StarCraft II using multi-agent reinforcement learning, Nature, 2019.
- Resident Evil 4 (Capcom, 2005), whose hidden difficulty adjustment is the standard example of doing it invisibly and then switching it off at the top.
- Tom Poindexter, CROBOTS (1985). Still on GitHub, still fun.
The research
- Steve Rabin (ed.), Game AI Pro, volumes 1 through 3. Free online, and the best single shelf on the subject.
- Marcus Ponsen et al. on tuning rule-based bots with evolution strategies (2011), and the CMA-ES family generally (Nikolaus Hansen). The lab’s tuner is a plain (μ, λ) evolution strategy, the simplest cousin.
- Yutian Chen et al., Bayesian Optimization in AlphaGo (2018), for the idea that tuning a game AI’s parameters offline against win rate is worth a lot of points.
- Daniel Hernandez et al., Metagame Autobalancing for Competitive Multiplayer Games (2020), for tuning toward a target win rate rather than the maximum, which is how a difficulty band tunes itself.
- David Churchill and Michael Buro on Portfolio Greedy Search, and Nicolas Barriga et al. on Puppet Search, for the idea of searching over strategies instead of actions. I didn’t ship it, but it shaped how the doctrine tables are laid out.
And
- Iron Galaxy’s write-up of Killer Instinct’s Shadow AI is where I got “degrade decisions, never reflexes.”
- My testers, who found the straight-line retreat, the hovering pilot, and everything else I’d missed, usually within a day.