Cognitive Biases Potentially Affecting Judgment of Global Risks

Draft for Global Catastrophic Risks, Oxford University Press, 2008 . Download as PDF .

CognitiveBiases-1


This document is ©2007 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.

Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute .

If you think the world could use some more rationality, consider blogging this page.

Praise, condemnation, and feedback are always welcome . The web address of this page is http://eyudkowsky.wpengine.com/rational/cognitive-biases/ .

Overcoming Bias

From August 2007 through May 2009, I blogged daily on the topic of human rationality at the econblog Overcoming Bias by Robin Hanson, getting around a quarter-million monthly pageviews. This then forked off the community blog Less Wrong , and I moved my old posts there as well for seed content (with URL forwarding, so don’t worry if links are to overcomingbias.com).

I suspected I could write faster by requiring myself to publish daily. The experiment was a smashing success.

Currently the majority of all my writing is on Less Wrong. To be notified when and if this material is compacted into e-books (or even physical books), subscribe to this announcement list .

The material is heavily interdependent, and reading in chronological order may prove helpful:

To see how interdependent it is, try looking over this graph of the dependency structure:

To read organized collections of posts, use the Sequences on the Less Wrong wiki.


This document is ©2008 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.

Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute .

If you think the world could use some more rationality, consider blogging this page.

Praise, condemnation, and feedback are always welcome . The web address of this page is http://eyudkowsky.wpengine.com/rational/overcoming-bias/ .

The Sword of Good

…fragments of a novel that would never be written…


Captain Selena, late of the pirate ship Nemesis, quietly extended the very tip of her blade around the corner, staring at the tiny reflection on the metal.  At once, but still silently, she pulled back the sword; and with her other hand made a complex gesture.

The translation spell told Hirou that the handsigns meant:  “Orcs.  Seven.”

Dolf looked at Hirou.  “My Prince,” the wizard signed, “do not waste yourself against mundane opponents.  Do not draw the Sword of Good as yet.  Leave these to Selena.”

Hirou’s mouth was very dry.  He didn’t know if the translation spell could understand the difference between wanting to talk and wanting to make gestures; and so Hirou simply nodded.

Not for the first time, the thought occurred to Hirou that if he’d actually known he was going to be transported into a magical universe, informed he was the long-lost heir to the Throne of Bronze, handed the legendary Sword of Good, and told to fight evil, he would have spent less time reading fantasy novels.  Joined the army, maybe.  Taken fencing lessons, at least.  If there was one thing that didn’t prepare you for fantasy real life, it was sitting at home reading fantasy fiction.

Dolf and Selena were looking at Hirou, as if waiting for something more.

Oh.  That’s right.  I’m the prince.

Hirou raised a finger and pointed it around the corner, trying to indicate that they should go ahead –

With a sudden burst of motion Selena plunged around the corner, Dolf following hard on her heels, and Hirou, startled and hardly thinking, moving after.

There was a hissing sound, as the seven creatures guarding the doorway caught sight of them, the intruders; their glistening chests expanded, sucking air. Their faces contracted, eyes squinting in an expression that a human would interpret as hatred, or surprise; and then their scaly-warted hands whipped over their heads and brought forth swords.

Selena already held her sword in her right hand, and her whip in her left.  She leaped forward and howled, a wordless cry that harmonized oddly with the battle roar of the orcs; and in almost the first instant of the clash, one of the orc-heads separated from its body and flew through the air, trailing foul-smelling black blood.

Hirou breathed evenly, trying to still his trembling.  The Sword of Good gave a tiny soft growl at his side (a sound that only he could hear) as Selena slashed her blade across another orc’s face, giving rise to a whistling howl.  Still he kept the Sword sheathed.  You are not to waste yourself against mundane opponents…  Even now the wizard was eyeing him closely, as if expecting him to defy orders and plunge into battle himself.

A small part of him, the part that wasn’t totally terrified by the battle, was flattered that Dolf thought so highly of him.  It was all Hirou could do not to turn and bolt; he was tensing his legs as though exerting a constant muscular effort to keep them in the same place.

The orc-bodies were piling up around Selena, the whip blinding or tripping or yanking, her blade ending life.  It might have taken hours, or seconds, before a huge blow split the last orc’s head all the way down the middle.

She stood there, blood-spattered and panting heavily, waiting as though daring the bodies to ever move again; then her face relaxed, and she gave a light laugh, and stooped to wipe her blade on the black orc-leather.

“You’re hurt!” Hirou blurted suddenly.  Red was soaking through the leather on Selena’s left arm.

Selena glanced downward.  “A scratch.”

“You cannot assume that,” rumbled the wizard.  “Their blades may be poisoned.”  Dolf stepped forward and brushed Selena’s arm briefly with the staff.

“Oh!” Selena said, her face surprised.  “It’s -“

But Dolf was already moving past her, to look at the gate the orcs had guarded, and the stairway leading upward.  “I believe,” he said in a quiet voice, “that there is a dark magus upstairs.”

“A magus!” Selena said.  “Here?”

“A magus,” Hirou echoed.  He swallowed hard; he knew what that meant.

Dolf only glanced at Selena.  “Do as I taught you: drop your weapons, sit in the corner, and clear your mind.  Now,” as Selena seemed about to protest.  “An ordinary warrior is only a liability, in a battle of wills; a weak point to be defended, a piece to be turned against its player.”

Selena looked at Hirou.  Hirou nodded.

And Selena sheathed her sword, dropped it and the whip, unbuckled the harness that held her daggers, and sat down in the corner of the room and began chanting softly to herself.

Dolf spared her only a glance.  “And now,” said the wizard in a low tone, “my Prince, you may enter the battle.”

Though most of Hirou’s mind was whited-out by terror, there was a remnant that seemed to see and follow the pattern, like reciting memorized lines in a play; and that remnant knew that Hirou’s part was to draw the Sword of Good.

The ancient metal whispered out of its scabbard.  As Hirou drew the Sword it began wailing, a small thin shriek that Hirou knew only he could hear.  The scream seemed to come from an infinitely narrow line running straight down the center of the Sword.  The sound had a quality that forced away attention, as though your eye were looking too close to the sun.  As though, if you listened too hard, you would – you would lose –

Dolf strode around the fallen orcs and their assorted body parts.  Hirou followed, breathing evenly; the Sword informed his hand to grip it high and across his chest.

“Who are we fighting?”  Hirou was surprised at how neutral his voice sounded.

A note of condemnation entered Dolf’s voice.  “A false wizard, this.  Not born to the Art, nor trained in the Halls.  Its gift comes to it by a higher master, by necromancy and potions…  But fear not, my Prince.  I shall prevent its will from reaching Selena and smother its other magics; and your Sword will sweep aside its defenses like fallen leaves.”

Through the door they swept, and mounted the stairs of the tower.  Dolf was breathing heavier, now, his face belying the effort of warding off some pressing will.  Hirou felt nothing, except perhaps a note of crispness in the air, as the Sword in his hand enforced an edict against certain specific types of delusion.

Then they were standing at the highest level of the tower, the end of the stairs, before one small wooden door.

“I’ll enter first,” Dolf signed, “and you follow as fast as you can, and strike as quickly as may be done.  Be careful not to strike me, my Prince.  The Sword of Good may strengthen your hand, but not guide your steps – it will strike me as easily as the foe, if you happen to turn it in my direction.”

Hirou nodded.  The air of neutrality was wearing away, and the acrid tang of adrenaline was entering his mouth.

“Three,” signed the wizard, “two, one -“

Dolf’s oaken staff crashed against the door, blasting it off the hinges in a flare of light and Dolf was racing into the room and Hirou was following him and the figure in stained brown robes was spinning its staff forward and a wall of flames swept out –

Hirou flinched and gave a small shriek, but the flames washed over him ineffectively before his feet could even stumble.  Averted by the Sword.  Dolf also was untouched – the defenses of a wizard were nearly impossible to break, Dolf had said; some wizards spent hours every day building them higher.  There was only one known weapon that could kill a wizard in a single blow, and that was –

Am I really going to do this?

But the Sword was already swinging forward in Hirou’s hand.

And the blade bounced off the air around the stained brown robes, with a sudden shower of orange sparks.

Crap, Hirou had time to think.

And then the false wizard’s staff was sweeping toward him (metal it was, not wood).

But the Sword in his hand moved to parry it, and there was another shower of sparks.

Keep attacking!” Dolf shouted.  “You chipped his sorcery!  Keep fighting!

Hirou gasped for breath and began to chop away with the Sword as though cutting wood, sending bits and pieces of broken magic everywhere.  There was little force in the blows except when the Sword moved to parry the staff; the rest was speed and repetition.

Then the scarred face beneath the hood gave a sudden shriek, as the Sword lightly scored over the dark flesh.

Is the shield down – ?

Before Hirou could even complete the thought, his arm lashed out with sudden force, and the Sword sank through the robes, near where a human would keep their heart.

There were no last words, not even a brief sigh.  The false wizard’s eyes widened, and then the robes just – fell over.

Hirou fell to his knees.

Your highness!

“I’m all right,” Hirou choked out.  Nausea competed with adrenaline for control of his existence, and lack of oxygen, and sharp and dull pains from his overexercised hand and arm.

Dolf’s staff brushed him, and the pain and nausea faded.

That only made it worse.  It removed the distractions.

The wizard was still looking at him, eyes flicking between Hirou and the sword.  “Wielding the Sword of Good did not – hurt you – did it, your highness?”

There was alarm in Dolf’s voice, as well there might have been.  The Sword of Good, according to Dolf, would kill the unworthy with the lightest touch, as of a single finger on the blade.  It killed nine out of ten would-be wielders, and in ordinary times the Imperial Family was not allowed to even try.  It had been prophesied that Hirou would wield the Sword, and yet…

“Dolf,” Hirou said hoarsely, “why did the Sword bounce off his shields?  You said it would cut through magic with a single blow.”

Dolf seemed uneasy.  “It has been centuries since the last wielder held the Sword of Good, noble Prince; perhaps not all the stories are true.  To cut through a wizardly shield with a score of blows is still a very great power.”

“No,” Hirou said.  He hesitated, then:  “I’m not wielding the Sword at full strength.  I can feel it.”

It seems… disappointed… in me.

Dolf nodded.  “The Sword of Good,” he quoted softly, “contains the essence of that which empowers a hero; the truth which only heroes can face.  My Prince… I have been reluctant to say this, but you have not been acting heroic.”  There was a peculiar gentleness on Dolf’s face that softened the impact of the words.  “But it will come with time.  Of that I am certain.  It is written in the royal blood of your forefathers.  You were raised in another place, but you are the heir of Bronze -“

Hirou retched, then swallowed hard, and hard again.  With a sudden flash of horror he knew – and he knew just how unheroic it was – that he was about to throw up on the corpse.


Their horses sauntered through the streets of the city – the capital of a whole province, it was, which meant perhaps a square mile enclosed by wooden walls, with the occasional two-story building.  Hirou kept his eyes moving, watching for possible ambushes – not that he really thought he had a chance of spotting one, if there was one.  But it was his best guess at how a hero would act.  What would Aragorn do? – that had been the refrain of his thoughts, of late.  Was the lady carrying a clay pot on each shoulder a threat?  Was the legless beggar, watching them with incurious eyes, a spy?

There was an excited buzz of conversation in the streets; from the snatches that were audible, Hirou gleaned that a military outpost of the Empire had been overrun by orcs.  The Empire was trying to play it down (said the overheard voices) but rumor had it a major disaster for the planned invasion campaign.

Hirou glanced over at Dolf and Selena.  Neither seemed to be paying any particular attention to the matter.

They cantered on for a short while longer, and finally Dolf drew rein.  Selena at once followed, and after a moment’s reaction time, so did Hirou.

“Here,” Dolf rumbled.

Hirou looked at the building on their right.  There was a huge painted board in front, showing a mouth being crammed with a turkey leg larger than itself.  The signs scratched below, the translation spell informed him, meant “INN OF EXTREMELY TASTY FOOD.”

One nice thing about this world:  If they don’t want you to know, they just keep quiet; and if they want you to know, they tell you straight out.

Hirou didn’t say it out loud, though.  Aragorn, descendant of Elendil and heir to the throne of Gondor, wouldn’t have said it.

Was that part of what empowered a hero?  That solemnity – or maybe just taking things seriously?  Hirou didn’t know.  But there was no point in taking chances.  The Sword hadn’t killed him yet, but neither had it fully unlocked in his hand.

The innkeeper’s eyes went wide at the sight of Dolf’s staff, and they were swiftly ushered into a private side room with a basket of candied fruits already waiting.  Selena had a sugared orange slice in her mouth almost as quickly as she sat down, and sighed in bliss; even Dolf took a handful of nuts.

Hirou, with a private sigh, took an apple slice lightly dusted in a spice he didn’t recognize.  Just the fact that it was spiced probably made it one of the most expensive and luxurious treats this world had to offer.  He bit, chewed, swallowed.

God he missed chocolate.

“So now what?” Selena said, after she’d eaten half the bowl.

“Now we wait,” Dolf said.

“For what?” said Selena.

Dolf looked around; the staff twitched in his hand and shed a brief woody glow.  Even so, the wizard lowered his voice before he spoke.  “This night, an assassin-courier and two hired thugs will come to this very inn, their wagon having broken a wheel on the road.  We must have the message that they carry, for it contains a hint to the location of the Empty Necklace.”

Selena blinked.  “Fine,” she said.  “I give up.  How could you possibly know that?”

Dolf looked at Hirou, his eyes asking permission.

“Tell her,” Hirou said.  He tried for a note of authority in his voice – a Crown Prince’s decision – but he didn’t know if he’d succeeded.

Dolf nodded, and his gaze shifted back to Selena.  “How much do you know about the Prophecy of Destiny?”

One nice thing about this world, they put very clear labels on everything – oh, skip it.

Selena blinked.  “Not much.  That’s wizard business.  Not much call for it in the pirating profession.”

“Very true,” Dolf said.  “But what do you know?”

Selena shrugged.  “A new Lord of Dark shall arise over Evilland, commanding the Bad Races, and attempt to cast the Spell of Infinite Doom.  The Long-Lost Heir, wielding the Sword of Good, shall kick Evil’s ass.  That’s about it.”

“That’s it?” Hirou said incredulously, then caught himself.  Aragorn wouldn’t have said that.

Selena smiled at him.  “It was enough for me, your Imperial Highness.  A chance like this only comes along once in a woman’s lifetime.”  She blew him a kiss.

For once Hirou wasn’t distracted.  “Master Dolf,” Hirou said, trying to make it a statement instead of a question – “I believe she needs to know more than that.”

“Yes…” Dolf said.  “Though it is wizard’s business indeed; and only by Imperial command may it go further…”  He drew a breath, lowered his voice further.  “The original Prophecy of Destiny, Selena, was never written down.  It has been memorized by the Archmagi and passed down by word of mouth through the generations.  It is more – detailed – then you seem to realize.  You are mentioned, pirate princess.  Mentioned by name and your mother’s name, daughter of Elaine.”

Selena’s mouth lay open, a picture of perfect astonishment.  “Ah…” she said.  “Do I die at the end?”

“No one knows,” Dolf said simply.  “The Prophecy of Destiny is a strange thing, pirate princess; it tells of some events in the smallest detail, omits others that would seem very large.  Told we were, to be on the ship that you attacked; told we were of your name.  The Prophecy of Destiny carries through to the confrontation between the Long-Lost Heir and the Lord of Dark, on the very verge of the casting of the Spell of Infinite Doom.  Then, it says, the Long-Lost Heir shall Choose between Good and Bad.  And there – there, of all places – the foretelling ends.”

“Huh,” Selena said.  She tapped her cheek.  “I somehow suspect, Master Wizard, that you wouldn’t tell me – or his Imperial Highness – if I did die at the end…”  She stared at Dolf, and Dolf looked back neutrally.  “So what does the Spell of Infinite Doom do?  Destroy the world?”

“Few there are who would deliberately destroy the world,” Dolf said.  “Even the Lord of Dark requires lesser beings to rule over.  No, the Spell of Infinite Doom destroys the Equilibrium.  Light and dark, summer and winter, luck and misfortune – the great Balance of Nature will be, not upset, but annihilated utterly; and in it, set in place a single will, the will of the Lord of Dark.  And he shall rule, not only the people, but the very fabric of the World itself, until the end of days.”

“Huh,” Selena said again.  Her eyes flicked to Hirou.  “And how are you leaning on that Choice between Good and Bad?”

“Good,” Hirou said instantly.

“Even if the Lord of Dark offered you the number two position as the master of the universe -“

“Good.”

“You’re not even thinking about it!”

“It’s not exactly a difficult question!” said Hirou.  “Calling it ‘the Choice between Good and Bad’ kind of gives away the answer.”

Selena was trying not to smile.  “You’ve never been tempted by anything?

“It’s not a matter of temptation!” Hirou said.  “It’s…” he trailed off for a moment.  It wasn’t that he couldn’t find the words.  It was that the concepts didn’t exist in this world.  What he wanted to say was that he had a pretty good idea what sort of behavior got you listed as a villain, in the great TV Tropes wiki of the universe; and he’d had a worried eye on his own character sheet since the day he’d realized what he’d gotten himself into; and he absolutely positively wasn’t going to go Dark Messiah, Knight Templar, Well Intentioned Extremist, or for that matter Lawful Stupid.

“It must be that the Lord of Dark will find something to offer you,” Selena said.  Her eyes were serious, now.  “Otherwise it won’t be much of a Choice between Good and Bad.”

“Fine by me,” Hirou said with some acerbity.  It wasn’t the questioning of his honor that disturbed him, so much as the idea of missing a choice that obvious.  How could anyone not know what their character sheet would say about that?

“What if the Lord of Dark had me prisoner, and threatened to kill me unless you -“

“Good.”

Selena opened her mouth, then closed it again.  Sudden hurt showed in her eyes.

Oh come on!” Hirou exclaimed.  He was too shocked, in that brief critical moment, even to think of smoothing it over.  “Have some common sense, Selena!  The whole world?

Selena smiled, a strange true smile tinged with sorrow.  “So this is the one who can touch the Sword of Good…  You will be a great Emperor someday, your Imperial Highness, a very great Emperor.  And you will see fit to reward me with a court title, and I will be Lady Selena, and none shall dare speak of the days when I was pirate and outlaw.  Maybe some nights you shall have me grace your bedchamber for old times’ sake, and maybe not.  That is enough.  More than I have a right to ask –  It was a foolish thought.”

“I -”  An abrupt pain caught at Hirou’s heart, which might have been for the sheer unfairness.  “Think it through, Selena!  Even if I did care about you more than anything, it would still be a stupid choice!  Let the Lord of Dark complete the Spell of Infinite Doom?  You might wish you had died!”

“I understand,” Selena said, still with that strange sad smile.  “Your reasoning is exactly correct, your Imperial Highness.  I am not questioning you at all.  I am only observing that you do not love me.”

Later that night, as with soft footsteps they padded toward the room where the assassin-courier and his two companions slept, Hirou held the Sword in his hand and stared at the central ridge of the blade.  The endless wail still arose from it, from the infinitely thin line through the center.  Hirou had been getting used to the sound, over time, which made it ever harder to focus his attention on it.

Do I get any points for that, Sword?  For what I said to Selena, even though I may have lost her?

The wail seemed only to diminish slightly, or maybe it was only Hirou’s attention wandering away.

It can’t be that a hero is someone who would choose one person over the world!  Not literally the whole world!  …can it?

The sound softened further, as if that infinitely thin line were growing more distant.

I wouldn’t be glad to sacrifice her!  It would hurt!  But I put myself on the line too!  Isn’t that what heroism is all about?  Sacrificing yourself and your own desires for the good of the world?

What is the truth that only heroes can face, if not that?

Hirou stared intently at the Sword, as if demanding an answer; and then became aware that his attention had moved away, once again, from that silent scream.

And the three of them stood before the doorway.

Selena took a small vial from off her harness, and dripped small droplets of oil onto the hinges of the door.  She was no master thief, but had a quietly professional grasp of the basics.  Quietly and slowly the door opened.  Selena went in first, and Dolf followed her, and then Hirou silently brought up the rear, Sword held in guard position.

The assassin-courier had a thin, pointed beard, and wore a light chainshirt even in his sleep.  His two escorts had an unshaven, unsavory look, and it was obvious from the smell of the room that they had not bathed.  The three of them were laid out on a line on as many beds.  Selena had a long thin poniard already in her hand, and plunged that needle straight through the left eyelid of the first thug, swift as a sword-strike on the downward plunge, stopping abruptly in mid-deathblow lest she strike the skull on the other side and make a sound.  She went around the beds and repeated the silent kill there on the other thug, as Dolf quietly moved to each of the four corners of the room in turn, while Hirou blocked the exit.

Then, with a knife held just above the courier’s throat, she spoke in a whisper.

“Don’t move,” Selena whispered, “or I’ll slit your throat before you can scream.”

The courier’s eyes flew open, and he drew a sudden breath, but stayed quiet.

“It may or may not matter to you,” Selena said, low and harsh, “but you’ve been working for the Lord of Dark, in case you didn’t know.  Now tell us the message that you carry.”

“Help!  Thieves!” cried the courier – in a small, soft voice that no one could possibly hear outside the room.

Dolf’s gaze lay intent upon the courier’s throat.

“You see how it is,” said Selena.  “So you can tell me the message right now – and the wizard here will know if you lie, I do assure you.  Or you can tell us the message… later.  Choose.”

Drown in a cesspool!” softly yelled the courier.

“What frightens you?” inquired Selena softly.  “Skinning?  Castration?”  Watching his face, the while.  “Blinding?  Crippling?  Or maybe -“

The courier spat at her.  Selena moved quickly, but the spittle still struck her on the cheek.  She didn’t take her blade from his throat, or her other blade from his crotch.

“You’ll regret that,” she said in a voice that brought a sudden chill to Hirou’s blood.  Her hands whitened on her blades.

Hirou suddenly had a sense of impending disaster, as if events in the room were about to spiral out of control.  He opened his mouth, then closed it again – he couldn’t think of a single thing to say that wouldn’t interfere with the interrogation.

Dolf spoke, a quieter version of his usual rumble.  “It seems you’re failing to impress him.”  Dolf took a step closer, and locked eyes with the courier.  “How’s this for a threat, Dark’s dog?”

Suddenly the color drained from the courier’s face, as his eyes locked onto some vision that only he and Dolf could see.  The courier screamed, and the sound came out as a small, thin, pathetic wail.

Dolf stepped back.  “That’s a threat,” he said in Selena’s general direction, and smiled one of his rare grins.

“The city of Silantra!” gasped the courier.  “I was to tell a man in black, who would call himself Alek, at the crossroads of Thu, to go to the city of Silantra, and investigate the temple ruins!  That’s all I know!  I swear!”

Selena looked inquiringly at Dolf, and Dolf nodded.

They scattered a few gold coins on the floor, to pay for the cleanup of the three corpses, and left at once while the cover of night still held.


The palace of the Lord of Dark seemed as deserted as the open desert beneath the moon, or some far-below cave in the bowels of the earth.  The floors and walls had been carefully carved and polished into inhuman curves, and decorated in colors that threatened to melt a human’s eyes.  By no five-fingered hands had this place been made.  And though the four of them had been creeping through the corridors at the cautious speed of a dungeon crawl, so far not a single trap or ambush had been sprung.

Alek was poking and prodding the door ahead with his staff.  It was a mighty and ornamented door, carved with inhuman faces set in indecipherable expressions, and Dolf had said there was something interesting beyond.

“Nothing,” Alek said, and shook his head in bemusement.  “No traps on this one either.  All those intricate carvings and not a single mechanism hidden behind them, so far as I can tell.”  He sighed.  “I’m beginning to feel useless.  You three didn’t really need a thief on this trip.”

Hirou looked up from where he was staring into the Sword’s blade, and half-smiled.  “We don’t know what isn’t trapped.  If we didn’t have a thief on this trip, we’d still have to check doors and floors.  We’d just be doing it much more slowly.  No, you’ve already saved the Forces of Good a good deal of time, Alek.”

Alek blinked.  “That’s… an odd way of looking at it… but you’re right.  Thank you, highness.”  Alek’s usual cheerful grin returned, and he stepped back and took his thieves’ staff from off his back.  Manipulating a lever at the base, he caused the staff’s clawed tip to close around the door-handle; he twisted, then pushed.

The door swung open.

Ewwwww,” Alek and Selena said in unison.

Before them, in the floor, was a vast pit of worms, writhing over one another in a light coating of slime.  Next to the pit was a glass cage of worms, these motionless and rotting; and wires of red metal ran from the glass cage to the ceiling.  The room smelled of cinnamon and decay.

“Dolf?” Hirou said.  “What are we looking at?”

“A Wormarium…”  Dolf blinked, and swallowed.  “I have… heard of this.  That any wizard, even the Lord of Dark, would sink so low -”  Dolf swallowed again. “The Lord of Dark is draining the life force of the worms in order to sustain himself.  He need not eat or drink, he will not age, he is cut off from the cycles of his own flesh.  The ordinary decay of his body, is transferred to the worms; and the life of the worms -“

Ewwwwww,” Selena and Alek said again.

“Shall we destroy it?” Hirou asked.

“The transfer cables are inactive…” muttered Dolf.  “Of course.  The Lord of Dark does not expect to need this once he completes the Spell of Infinite Doom.  Or perhaps he thinks it might interfere – well.  It matters not.  I think he shall not notice what we do here.”  Dolf grounded his staff, and a look of concentration briefly flashed across his face.

Then a sudden blaze of green incandescence burst forth from the pit and the cage –

Alek convulsively yanked the door shut using the thieves’ staff.  “Gah!” he said, then lowered his voice.  “Warn a guy when you’re about to do that, Master Wizard!  I thought we’d triggered something.”

“Our work here is done,” Hirou said – the end of the statement turning up only slightly in a questioning inflection.

Dolf nodded.

“Do you sense anything else interesting enough to warrant our attention?  Any other potential resources we should try to deny our enemy, before the battle begins?”

Dolf shook his head.

Hirou took a deep breath.  He’d played out this scenario in his head so many times over and over that the reality felt more like a relief than anything else.  “Then it’s time.”

They retraced their steps away from the Wormarium, returning to the central corridor they had explored earlier.  Alek again took the lead, and they slowly, slowly walked down the long black metallic floor.

After a long walk, the corridor widened out into a huge vestibule that for once did not insult the human eye.  Floor laid with rectangular stones, walls hung with tapestries of pleasant color and disturbing subjects.  On the left wall, an orc cradled the bloody body of a smaller orc, above a heap of bloody and slashed human bodies; other orcs gazed at the scene intently.  All of their expressions were inhuman, and indecipherable.  On the right wall, a grey-robed figure with human hands visible, but face concealed by a solid metal mask, stood as though in blessing over a field of green plants with twisted stalks.

In front of them was a huge door fit for a city gate, inlaid with gold and gems that could have purchased a whole province.  Even Hirou, who came from a wealthier plane of existence, was impressed.

“Bloody hell,” Alek said under his voice, very softly, staring at the rectangular floorstones in their neatly tiled pattern.  “I hate this sort of thing.”

Step by step they walked across the floor, Alek pressing hard with the thieves’ staff on every floorstone for thirty full seconds before continuing onward.

It was on almost the last step before the door that the stone suddenly slid away with a huge shriek – not the stone Alek had just pressed down with his staff, but the stone before that, where Alek had stood.

With a choked yell, the thief plummeted and vanished.

Alek!” Selena screamed, and ran forward heedless.  Hirou began to follow, then, with coldly bitter determination, checked himself.

Selena looked down into the gap in the floor where Alek had vanished.

She choked.  “Alek!”  Then, as if gone mad, she leaned over the gap and began to reach down.

A premonition prickled at Hirou, and with sudden desperation he leaped forward and yanked Selena back from where she was leaning.  With a shriek and echoing boom the stone surged back into place, almost crushing Selena’s outstretched hand.

No!” Selena cried.  Tears were already rolling down her cheek.  “Hirou, please!  We have to get to him!”

“Your highness, you mustn’t -” came Dolf’s rumble.

The cold bitterness, already in Hirou, turned to sudden rage and self-loathing.  As had happened once before, the terrible wail from the center of the Sword seemed to grow louder, to fill his mind; heavier than a mountain and more corrosive than a flood, a refusal-to-accept that would blast anything in its pathway – but still, somehow, essentially moral in nature, more than pure destruction or simple entropy –

Hirou’s Sword lashed out as though it were a part of him, and smashed down upon the stone.

And the stone shattered in the same instant, as though every part of it had been unbound from itself; it fell into pebbles, and the pebbles fell into dust, and the dust turned to smoke and billowed upward.

And the smoke cleared, and showed Alek above a bed of worms – some crushed by Alek’s fall, some already beginning to writhe over his form.

Alek wasn’t moving, he wasn’t breathing.  The worm-slime glistened on his skin.

And then there was another groan of machinery, and Alek’s body and the worms began to move out of their sight, as a new pit of worms moved into place below the floor.

No!” Selena screamed, an awful, heartwrenching plea that broke and shattered in her lips.  “Alek!  No!

Hirou laid his left hand on Selena’s shoulder.  “We must go,” he said.  His voice sounded empty and emotionless, even to his own ears.  “The Lord of Dark knows we’re here, now.”

Selena rose from the open pit, hands clenched as if to strike.

“You don’t respect anything, do you,” she said in a voice colder than the night between worlds.

I’m sorry.  I know how much Alek meant to you.  You can hit me later, if you like.

“We have to go,” Hirou repeated.  “We have to hurry.”

Selena turned away from him, and drew her swords.  “Yes, your Imperial Highness,” she said.  He couldn’t see her face.

Hirou leaped across the gap in the floor to the final stone before the door.  The wail had not diminished, this time; it was still in his mind.

With a terrible black fury and a convulsion like throwing a mountain, Hirou struck, and turned the bright gold door to smoke.  So much for traps.

And the smoke cleared, and they saw the huge throne room, and the throne, and the Lord of Dark.

A jolt of surprise rippled through Hirou’s mind.  The throne room was not small, but neither was it the hugeness that Hirou had expected; the size of a small house, perhaps.  Scenes of sun and clouds, grass and hills, dotted the walls; and a vast skylight, above, let in a pleasant golden glow.  The Lord of Dark’s throne was laid on a golden platform, and the throne itself was comfortably cushioned and well-designed for the human form; more like an office chair of Hirou’s own world than a formal seat.  Behind the throne lay a shimmering screen of force; and behind the screen of force, an altar; and on the altar, an intricate array of gears turning without axles or wires; and above the gears, a throbbing blaze of light.

And the Lord of Dark sat on the ergonomic throne, garbed in a comfortable cassock of gray silk.

“Oh, finally,” said the Lord of Dark.  His fingers tapped on the arm of his throne, dit-dit-dit.  “I was starting to wonder if you were going to show up, Hirou.”

Hirou’s mind was scrambled, for a moment, he couldn’t remember his own planned opening line.  “Were you, now?” his mouth said.

“Come now,” said the Lord of Dark, “don’t tell me you were trying to sneak up on me?  The entire world knows the prophecy about our meeting!  The wielder of the Sword of Good is supposed to arrive before I complete the Spell of Ultimate Power.”  The Lord of Dark waved at the glow above the machinery on the altar behind the throne.  “And that’s just about done.”

Dolf smiled grimly, from where he leaned upon his staff.  “You’re frightened.”

Of course I’m nervous!  Gah!”  The Lord of Dark made a convulsive gesture as though to claw at the empty air, radiating frustration.  “Are you done stating the obvious?”

Selena raised a sword and pointed at the Lord of Dark.  Around her neck, the Glowy Stone flamed brightly where it had been set in the Empty Necklace; no sorcery of mind would touch her with that armor, still less while Dolf stood guard.

“You killed my only love,” she said in a simple voice, a quiet voice, a voice like death, “and I am going to kill you.”

The Lord of Dark looked at her.  A complex expression flashed across his face: condemnation was in it, and pity.

Then, without a word or a gesture, Alek’s body floated out and came to rest near the altar, behind the screen of force.

“Alek’s head is still intact,” the Lord of Dark said.  “You may or may not know, Selena, that everything that a human is, resides in a human’s brain.  Your lover still exists, Selena; all that is him, still is there.  He is simply not breathing, at the moment.  After I complete the Spell of Ultimate Power, I’ll have the ability to bring Alek back.  And I will.  Does that work for you?”

Selena swayed where she stood.  She choked, a single sob escaping her lips.

Hirou felt a sudden chill, remembering a conversation from what seemed like ages ago.  “What if the Lord of Dark had me prisoner, and threatened to kill me unless you -“

Selena looked like a woman in the midst of tearing out her own heart and crushing it with her own hands.

Hirou dropped his eyes.  He couldn’t look at it.  He only watched Selena’s hands on the swords, waiting for her decision.

And then Selena straightened, and her swords came level in her hands, pointing at the Lord of Dark; and she said, in a small voice like she was dying,

“Good.”

Sudden tears came into Hirou’s eyes.

Slight puzzlement flickered on the Lord of Dark’s face.  “I mean it,” said the Lord of Dark.  “I’m not asking anything from you.  Just telling you that if I win, I’ll bring Alek back.  That’s a promise.”

You son of a bitch.  Hirou saw it, then, the cruel subtlety of the Lord of Dark.  Not the obvious threat, demanding Selena to betray her friends in exchange for her lover’s life.  No crude offer that could be refused once and for all.  Just the simple and unconditional promise – and then Selena would have to fight on, knowing with every breath and every blow that if she won, she lost her only love forever.

“Bastard,” choked Selena.  And she tilted the sword further to point at the Lord of Dark’s head.

The Lord of Dark shook his head in annoyance, and then focused his gaze fully upon Hirou.

Hirou tensed.  He’d been wondering, for a long time now, what the Lord of Dark could possibly offer him, what threat he could possibly make, to give Hirou a Choice worth the name.  Hirou had thought about that, trying to put himself in the Lord of Dark’s place; and he thought that the Lord of Dark might indeed offer to make Hirou his number two, or alternatively, if Hirou refused and then lost, keep him alive and torture him for thousands of years.  That was about as forceful as Hirou could imagine making it –

But the Lord of Dark had already demonstrated himself more subtle than Hirou’s imagination.

The Lord of Dark spoke.  His voice was more formal, now; not calm, but steady.  “All the preliminaries are in place, wielder of the Sword of Good.  There remains only your Choice between Good and Bad.”  The Lord of Dark’s eyes grew intent.  “Hirou, completing the Spell of Ultimate Power requires the sacrifice of a wizard of the highest degree, and also I have a use for the Sword of Good.  In the name of all the darkness that exists in the world, I request that you kill Dolf with the Sword of Good, and then give it to me.”

There was a long pause.

“That’s it?” Hirou said finally.  The whole thing was so insane, after so much waiting and wondering, that he felt a crazy laughter rising up in his own throat.  He swallowed it.  “That’s the awful temptation?  That’s the Choice?  You think I’m going to choose Bad over Good because you asked politely?

The Lord of Dark stared at Hirou as though he were the crazy one.  “The Choice between Good and Bad,” said the Lord of Dark in a slow, careful voice, as though explaining something to a child, “is not a matter of saying ‘Good!’  It is about deciding which is which.”

Dolf uttered a single bark of laughter.  “You’re mad!” his voice boomed.  “Can you truly not know that you are evil?  You, the Lord of Dark?

“Names,” said the Lord of Dark quietly.

Hirou was so angry he could hardly speak.  With an icy effort of control he forced himself back to calm, forced his eyes to keep moving.  This could all be a distraction.  “If you’re going to give me some pathetic speech about how good and evil are just different sides of the same coin -“

“Absolutely not,” said the Lord of Dark at once.  His gaze flicked to Dolf.  “It is the wizards who go about talking of Equilibrium and Balance.  I am pleased to see, Hirou, that you do not agree with them.  No, Hirou, I am asking you something much simpler.”  His eyes bored into Hirou’s face.  “What wrong have I done?

A small note of disorientation rose up in Hirou, like climbing stairs and stepping on what you thought was the last stair, but beneath your foot there was no stair, no floor, nothing…

“You suck the life from worms,” Selena said coldly.  “I know darkness when I see it.”

The Lord of Dark’s gaze scarcely flickered in her direction.  “Be silent, eater of mammals.”

“You command the Bad Races of Evilland!” roared Dolf.  “You lent them your sorcery, aided them in slaughtering human beings!”

The Lord of Dark was watching Hirou carefully as he made reply.  “Human beings first launched an unprovoked attack on this land some three thousand years ago, saying – though it was lies – that the inhabitants ate human flesh.  The records here would have it, and I believe them, that the missing people were in fact being kidnapped and sold by human slave-takers.  Since then, those you call the ‘Bad Races’ have been fighting off repeated attempts at extermination.  Oh, they hate you, of course they do; but they are wise enough to understand that there are a few good humans, even as there is evil among their own kind.  They are friendly enough to me.”

An awful fear began to rise up in Hirou –

“Now it is my turn to make accusation,” said the Lord of Dark.  He stood; anger gathered around him like a cloak, and his voice rang out through the throne room.  “You, Dolf, Archwizard of the fell Empire, I do accuse of commanding and causing to be performed, the murders of Elzhur, Anzha, Stav, Valdil, Emhil, Tohm, Khal, and the magus Mikel.  On the eighth day of the seventh moon of this year you ordained their deaths.  I do not call them innocents.  They bore weapons, they went knowingly to the risk.  But you, Dolf, you who made necessary their sacrifice – you may not be forgiven for the lives you have cut short, and the grief you have given to their families and survivors!  Though this is only the beginning of your long litany of crimes, yet I remember the day that first message came to me -“

“You are mad,” Selena said with conviction.  “You accuse us of murder for killing orcs?

Hirou stood frozen.

There was a hissing sound, as the seven creatures guarding the doorway caught sight of them, the intruders; their glistening chests expanded, sucking air. Their faces contracted, eyes squinting in an expression that a human would interpret as hatred, or surprise; and then their scaly-warted hands whipped over their heads and brought forth swords.

Why – did I –

So what if their skin was moist, and scaly and warted, and unsightly to human eyes?  So what if their blood smelled foul, as Selena poured it forth in rivers?

Why – didn’t I –

Hirou’s memory moved forward relentlessly, like waking up from and reviewing some mad dream.

– his arm lashed out with sudden force, and the Sword sank through the robes, near where a human would keep their heart –

“Here is your crime!” roared Dolf.  “You, a human, have betrayed the Empire!  You, a true wizard by birth, have betrayed the Ancient Halls of Wizardry!  You spread sedition and treason, and oppose the authority of the rightful heir to the throne!”

…why did I think that I had the right to rule over millions of people, without votes or parliaments, because of who my parents were?

Dolf slammed his staff on the ground.  “And above all!  Above all!  That you seek to cast the Spell of Infinite Doom!  That you, in your lust for power, would destroy the very Equilibrium that holds the world in Balance!”

Because Dolf seemed to expect it of me, because no one around me seemed to question that it was a good idea, or even point it out as something to think about –

“Equilibrium,” hissed the Lord of Dark.  His face twisted.  “Balance.  Is that what the wizards call it, when some live in fine castles and dress in the noblest raiment, while others starve in rags in their huts?  Is that what you call it when some years are of health, and other years plague sweeps the land?  Is that how you wizards, in your lofty towers, justify your refusal to help those in need?  Fool!  There is no Equilibrium!  It is a word that you wizards say at only and exactly those times that you don’t want to bother!  It prevents you from giving food to the hungry, but not from filling your own bellies!  Your friends are good enough to be healed, no threat to the Balance there, but the cripple in the streets must be left to suffer -“

Dolf stepped forward and brushed Selena’s arm briefly with the staff –

– was the legless beggar, watching them with incurious eyes, a spy?

Why hadn’t he thought to ask –

” – because you just don’t care!

And in the stillness of dawning disaster, in the first note of questioning, Hirou thought of something else he had never thought to ask.  Dolf had his sorcerous shields of protection.  Why had Dolf let Alek walk in front?  Dolf was in fact by far the strongest member of their party – why had he let Selena do the fighting?

Because Dolf was more important, and if he exposed himself to all the risk every time, he might eventually be injured, Hirou’s logical mind completed the thought.  Lower risk, but higher stakes.  Cold but necessary –

But would you, said another part of his mind, would you, Hirou, let your friends walk before of you and fight, and occasionally die, if you knew that you yourself were stronger and able to protect them?  Would you be able to stop yourself from stepping in front?

Perhaps, replied the cold logic.  If the world were at stake.

Perhaps, echoed the other part of himself, but that is not what was actually happening.

That part of him knew, as Selena had known before.

It is just that, from the beginning, Dolf never cared in the slightest about Selena’s life.

Had cared nothing for a mere pirate captain –

Pirate captain?

Hirou’s eyes flicked briefly to Selena.

She has attacked ships and sunken ships, she has kidnapped and killed.  All in the name of profit for herself, before ever she met me or tried to save the world.  She killed dozens without a thought, until her own love was lost, and then a single death was suddenly an event of world-shaking significance –

Why did I think that was acceptable?

Why didn’t I notice?

Another memory came to Hirou.

– the color drained from the courier’s face, as his eyes locked onto some vision that only he and Dolf could see.  The courier screamed, and the sound came out as a small, thin, pathetic wail –

Dolf had done that without touching the man, but –

Threats of death and injury are already torture in themselves, under the Geneva Convention, by the laws of my own world.

He’d known something was wrong.  That small note of disquiet in the corner of his mind.  But he hadn’t said a word out loud, because, well, it would have been awkward.

I am a fool.

Worse than a fool.

Why didn’t the Sword just kill me?

And the everlasting wail of the Sword of Good burst fully into his consciousness

It was like his mind and self were sucked toward that infinitely thin line running through the center of the Sword, the edge within the blade.  Sucked toward that edge, and cut through.

Cut through and torn wide and forced open –

A scream ripped from Hirou’s lips.

He was starving to death freezing naked in cold night being stabbed beaten raped watching his father daughter lover die hurt hurt hurt die –

– open to all the darkness that exists in the world –

His consciousness shattered into a dozen million fragments, each fragment privy to some private horror; the young girl screaming as her father, face demonic, tore her blouse away; the horror of the innocent condemned as the judge laid down the sentence; the mother holding her son’s hand tightly with tears rolling down her eyes as his last breath slowly wheezed from his throat –

– all the darkness that you look away from, the endless scream.

Make it stop!

It might have been Hirou’s thought, or the thought of the man who screamed as his foot was crushed beneath a stone.

Refuse, reject, change, reality don’t be like this –

Make it stop!

It could have been Hirou or the child in the burning house.

make it stop
make it stop
make it stop
MAKE IT STOP
MAKE IT STOP

I WILL MAKE IT STOP

In the throne room of the Lord of Dark, the Sword suddenly blazed up with a shock like a thousand-mile dam breaking, a roaring tsunami of force.  The eyes could not see that power, wavered between detecting it as light or darkness; so that Hirou, grasping the hilt, was the only dark thing left against the brilliance, or the only bright thing haloed against the shadow.

Dolf had been turning toward Hirou with alarm in his face; now his eyes widened, and a sudden gladness lit his countenance.  “You’ve done it!” Dolf cried.  “You have awakened the Sword at last!  Now, my prince, with but a single strike you may -“

The Sword, with one smooth sweep, cut through all Dolf’s defenses like water and touched the wizard’s throat; and in the moment of the Sword touching Dolf’s skin, the wizard stopped.  The Sword continued in its motion unabated, and Dolf’s head separated from his body and went rolling across the floor, as something seemed to flow away from the corpse toward the gears above the altar.

Selena’s cry of horror mingled with the sudden hum of the brightening glow above the gears.

“Hirou!” she screamed.  “Hirou!  Why?  You said you would be good!

Then she turned toward him, and pointed her swords –

Selena froze in place like a statue, one of her feet suspended in mid-air and mid-run; in the same instant the glowing stone on her necklace shattered.

Hirou’s eyes drifted, ever so slowly it seemed, to the disbelief on Selena’s face.

A part of him was horrified and saddened, to see her looking at him like that.

And at the same time, it seemed like such a small thing, her horror, his own sadness, compared to even a single parent watching their child die.  Let alone the actual number doing so, right at that moment, elsewhere in the world.

“Thank you,” said the Lord of Dark softly.

Make it stop,” said Hirou’s lips.  There were other thoughts inside him, still being carried out by his brain, but they were dwarfed under that single terrible weight.

The Lord of Dark rose from his throne, began to come forward.  “I must touch the blade.”

Hirou crossed the intervening space in an instant, the Sword moving in a single perfect arc in his hands; it was as though the blade simply materialized in front of the Lord of Dark.

The Lord of Dark jerked back.

Hurry,” said Hirou’s lips.

“The Spell of Ultimate Power is already in progress now, and will complete in a few moments.  It can neither be hurried nor delayed,” said the Lord of Dark.  “But before that time, there is one last thing I must do -“

The Lord of Dark reached out for the Sword, but his fingers faltered.

Must do,” the Lord of Dark repeated to himself; and his fingers reached out, and firmly came to rest on the blade of the Sword of Good.

They lingered there for a long moment.

Then, “Thank you,” said the Lord of Dark.  “That was all.  You can put down the Sword of Good now.  You probably should.”

Hirou dropped the Sword.  In the instant the Sword left his hands it became only another piece of metal, and fell to the ground with a simple clang.

And in the moment that Hirou’s hands left the hilt, he became only another mortal.

Hirou staggered, and was distantly aware of the Lord of Dark catching him as he fell, to lay him gently on the ground.

In a whisper, Hirou said “Thank you -” and paused.

“My name is Vhazhar.”

“You didn’t trust yourself,” Hirou whispered.  “That’s why you had to touch the Sword of Good.”

Hirou felt Vhazhar’s nod, more than seeing it.

The air was darkening, or rather Hirou’s vision was darkening, but there was something terribly important left to say.  “The Sword only tests good intentions,” Hirou whispered.  “It doesn’t guide your steps.  That which empowers a hero does not make us wise – desperation strengthens your hand, but it strikes with equal force in any direction -“

“I’ll be careful,” said the Lord of Dark, the one who had mastered and turned back the darkness.  “I won’t trust myself.”

“You are -” Hirou murmured.  “Than me, you are -“

I should have known.  I should have known from the beginning.  I was raised in another world.  A world where royal blood is not a license to rule, a world whose wizards do more than sneer from their high towers, a world where life is not so cheap, where justice does not come as a knife in the night, a world where we know that the texture of a race’s skin shouldn’t matter –

And yet for you, born in this world, to question what others took for granted; for you, without ever touching the Sword, to hear the scream that had to be stopped at all costs –

“I don’t trust you either,” Hirou whispered, “but I don’t expect there’s anyone better,” and he closed his eyes until the end of the world.


This document is ©2009 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.

Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute.

Praise, condemnation, and feedback are always welcome. The web address of this page is http://yudkowsky.net/other/fiction/the-sword-of-good/.

Twelve Virtues of Rationality

The first virtue is curiosity. A burning itch to know is higher than a solemn vow to pursue truth. To feel the burning itch of curiosity requires both that you be ignorant, and that you desire to relinquish your ignorance. If in your heart you believe you already know, or if in your heart you do not wish to know, then your questioning will be purposeless and your skills without direction. Curiosity seeks to annihilate itself; there is no curiosity that does not want an answer. The glory of glorious mystery is to be solved, after which it ceases to be mystery. Be wary of those who speak of being open-minded and modestly confess their ignorance. There is a time to confess your ignorance and a time to relinquish your ignorance.

The second virtue is relinquishment. P. C. Hodgell said: “That which can be destroyed by the truth should be.” Do not flinch from experiences that might destroy your beliefs. The thought you cannot think controls you more than thoughts you speak aloud. Submit yourself to ordeals and test yourself in fire. Relinquish the emotion which rests upon a mistaken belief, and seek to feel fully that emotion which fits the facts. If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm. Evaluate your beliefs first and then arrive at your emotions. Let yourself say: “If the iron is hot, I desire to believe it is hot, and if it is cool, I desire to believe it is cool.” Beware lest you become attached to beliefs you may not want.

The third virtue is lightness. Let the winds of evidence blow you about as though you are a leaf, with no direction of your own. Beware lest you fight a rearguard retreat against the evidence, grudgingly conceding each foot of ground only when forced, feeling cheated. Surrender to the truth as quickly as you can. Do this the instant you realize what you are resisting; the instant you can see from which quarter the winds of evidence are blowing against you. Be faithless to your cause and betray it to a stronger enemy. If you regard evidence as a constraint and seek to free yourself, you sell yourself into the chains of your whims. For you cannot make a true map of a city by sitting in your bedroom with your eyes shut and drawing lines upon paper according to impulse. You must walk through the city and draw lines on paper that correspond to what you see. If, seeing the city unclearly, you think that you can shift a line just a little to the right, just a little to the left, according to your caprice, this is just the same mistake.

The fourth virtue is evenness. One who wishes to believe says, “Does the evidence permit me to believe?” One who wishes to disbelieve asks, “Does the evidence force me to believe?” Beware lest you place huge burdens of proof only on propositions you dislike, and then defend yourself by saying: “But it is good to be skeptical.” If you attend only to favorable evidence, picking and choosing from your gathered data, then the more data you gather, the less you know. If you are selective about which arguments you inspect for flaws, or how hard you inspect for flaws, then every flaw you learn how to detect makes you that much stupider. If you first write at the bottom of a sheet of paper, “And therefore, the sky is green!”, it does not matter what arguments you write above it afterward; the conclusion is already written, and it is already correct or already wrong. To be clever in argument is not rationality but rationalization. Intelligence, to be useful, must be used for something other than defeating itself. Listen to hypotheses as they plead their cases before you, but remember that you are not a hypothesis, you are the judge. Therefore do not seek to argue for one side or another, for if you knew your destination, you would already be there.

The fifth virtue is argument. Those who wish to fail must first prevent their friends from helping them. Those who smile wisely and say: “I will not argue” remove themselves from help, and withdraw from the communal effort. In argument strive for exact honesty, for the sake of others and also yourself: The part of yourself that distorts what you say to others also distorts your own thoughts. Do not believe you do others a favor if you accept their arguments; the favor is to you. Do not think that fairness to all sides means balancing yourself evenly between positions; truth is not handed out in equal portions before the start of a debate. You cannot move forward on factual questions by fighting with fists or insults. Seek a test that lets reality judge between you.

The sixth virtue is empiricism. The roots of knowledge are in observation and its fruit is prediction. What tree grows without roots? What tree nourishes us without fruit? If a tree falls in a forest and no one hears it, does it make a sound? One says, “Yes it does, for it makes vibrations in the air.” Another says, “No it does not, for there is no auditory processing in any brain.” Though they argue, one saying “Yes”, and one saying “No”, the two do not anticipate any different experience of the forest. Do not ask which beliefs to profess, but which experiences to anticipate. Always know which difference of experience you argue about. Do not let the argument wander and become about something else, such as someone’s virtue as a rationalist. Jerry Cleaver said: “What does you in is not failure to apply some high-level, intricate, complicated technique. It’s overlooking the basics. Not keeping your eye on the ball.” Do not be blinded by words. When words are subtracted, anticipation remains.

The seventh virtue is simplicity. Antoine de Saint-Exupéry said: “Perfection is achieved not when there is nothing left to add, but when there is nothing left to take away.” Simplicity is virtuous in belief, design, planning, and justification. When you profess a huge belief with many details, each additional detail is another chance for the belief to be wrong. Each specification adds to your burden; if you can lighten your burden you must do so. There is no straw that lacks the power to break your back. Of artifacts it is said: The most reliable gear is the one that is designed out of the machine. Of plans: A tangled web breaks. A chain of a thousand links will arrive at a correct conclusion if every step is correct, but if one step is wrong it may carry you anywhere. In mathematics a mountain of good deeds cannot atone for a single sin. Therefore, be careful on every step.

The eighth virtue is humility. To be humble is to take specific actions in anticipation of your own errors. To confess your fallibility and then do nothing about it is not humble; it is boasting of your modesty. Who are most humble? Those who most skillfully prepare for the deepest and most catastrophic errors in their own beliefs and plans. Because this world contains many whose grasp of rationality is abysmal, beginning students of rationality win arguments and acquire an exaggerated view of their own abilities. But it is useless to be superior: Life is not graded on a curve. The best physicist in ancient Greece could not calculate the path of a falling apple. There is no guarantee that adequacy is possible given your hardest effort; therefore spare no thought for whether others are doing worse. If you compare yourself to others you will not see the biases that all humans share. To be human is to make ten thousand errors. No one in this world achieves perfection.

The ninth virtue is perfectionism. The more errors you correct in yourself, the more you notice. As your mind becomes more silent, you hear more noise. When you notice an error in yourself, this signals your readiness to seek advancement to the next level. If you tolerate the error rather than correcting it, you will not advance to the next level and you will not gain the skill to notice new errors. In every art, if you do not seek perfection you will halt before taking your first steps. If perfection is impossible that is no excuse for not trying. Hold yourself to the highest standard you can imagine, and look for one still higher. Do not be content with the answer that is almost right; seek one that is exactly right.

The tenth virtue is precision. One comes and says: The quantity is between 1 and 100. Another says: the quantity is between 40 and 50. If the quantity is 42 they are both correct, but the second prediction was more useful and exposed itself to a stricter test. What is true of one apple may not be true of another apple; thus more can be said about a single apple than about all the apples in the world. The narrowest statements slice deepest, the cutting edge of the blade. As with the map, so too with the art of mapmaking: The Way is a precise Art. Do not walk to the truth, but dance. On each and every step of that dance your foot comes down in exactly the right spot. Each piece of evidence shifts your beliefs by exactly the right amount, neither more nor less. What is exactly the right amount? To calculate this you must study probability theory. Even if you cannot do the math, knowing that the math exists tells you that the dance step is precise and has no room in it for your whims.

The eleventh virtue is scholarship. Study many sciences and absorb their power as your own. Each field that you consume makes you larger. If you swallow enough sciences the gaps between them will diminish and your knowledge will become a unified whole. If you are gluttonous you will become vaster than mountains. It is especially important to eat math and science which impinges upon rationality: Evolutionary psychology, heuristics and biases, social psychology, probability theory, decision theory. But these cannot be the only fields you study. The Art must have a purpose other than itself, or it collapses into infinite recursion.

Before these eleven virtues is a virtue which is nameless.

Miyamoto Musashi wrote, in The Book of Five Rings:

“The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy’s cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him.”

Every step of your reasoning must cut through to the correct answer in the same movement. More than anything, you must think of carrying your map through to reflecting the territory.

If you fail to achieve a correct answer, it is futile to protest that you acted with propriety.

How can you improve your conception of rationality? Not by saying to yourself, “It is my duty to be rational.” By this you only enshrine your mistaken conception. Perhaps your conception of rationality is that it is rational to believe the words of the Great Teacher, and the Great Teacher says, “The sky is green,” and you look up at the sky and see blue. If you think: “It may look like the sky is blue, but rationality is to believe the words of the Great Teacher,” you lose a chance to discover your mistake.

Do not ask whether it is “the Way” to do this or that. Ask whether the sky is blue or green. If you speak overmuch of the Way you will not attain it.

You may try to name the highest principle with names such as “the map that reflects the territory” or “experience of success and failure” or “Bayesian decision theory”. But perhaps you describe incorrectly the nameless virtue. How will you discover your mistake? Not by comparing your description to itself, but by comparing it to that which you did not name.

If for many years you practice the techniques and submit yourself to strict constraints, it may be that you will glimpse the center. Then you will see how all techniques are one technique, and you will move correctly without feeling constrained. Musashi wrote: “When you appreciate the power of nature, knowing the rhythm of any situation, you will be able to hit the enemy naturally and strike naturally. All this is the Way of the Void.”

These then are twelve virtues of rationality:

Curiosity, relinquishment, lightness, evenness, argument, empiricism, simplicity, humility, perfectionism, precision, scholarship, and the void.


This document is ©2006 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.

Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute .

If you think the world could use some more rationality, consider blogging this page.

Praise, condemnation, and feedback are always welcome . The web address of this page is http://eyudkowsky.wpengine.com/rational/virtues/ .

There is a pamphlet format for the Twelve Virtues. Print using a duplex (two-sided) printer, cut in half, staple in the middle, and fold.

If you enjoyed this writing, let your journey continue with The Simple Truth . You may also enjoy An Intuitive Explanation of Bayesian Reasoning and A Technical Explanation of Technical Explanation .

“Non-Player Character”

Rilanya: “You’re not like the others, are you?”

Darin: “What do you mean?”

Rilanya: “I… do you know why I first fell in love with you?”

Darin: “For my good looks?”

Rilanya: “My whole life I’ve felt so alone. The people around me… they just seemed to be going through the motions. Like they were asleep, or drugged, even when they worked, or played, or got drunk, or made love. They all think the same things in the same way. Each day the same. Repetitive. Like they’re only shadows of people.”

Darin: “Everyone feels that way sometimes, Rilanya.”

Rilanya: “But you’re not like them. You say new things. I don’t always understand them, especially your jokes, but they’re new, and that’s the important thing. Darin, can I ask you a question?”

I looked at the screen for a few moments. Rilanya’s rendered graphic was looking at my point-of-view with a pleading expression. Plot point, I thought to myself, and typed: “Anything, Rilanya.”

Rilanya’s figure took a deep breath and leaned close to my point-of-view. Her animated lips moved and her voice issued from my headphones: “What’s an NPC?”

“What?” I said, out loud. Then I started laughing.

Rilanya went on talking. “In the tower of Ashel, when you rescued me from the prison chamber… the guards were dead outside my door. I’d never seen blood before. And you said… I remember your exact words… ’Don’t worry, babe, they were only NPCs.‘ And then that time in the tavern, when that man only wanted to talk about the Plaited Road, you said… ’Guess the NPCs here aren’t programmed for deep conversation, huh?’ You use that word… the same times when I get that feeling, that all the people around me are only shadows.”

I just looked at the screen for a few moments. I was getting ever so slightly creeped out. I knew that this was some programmer’s idea of a practical joke, I knew it solidly and with every ounce of my common sense, and I wanted to see where it led, but I was still creeped out.

Darin: “We’re all ultimately alone in this world, Rilanya.”

Rilanya: “You’re not from this world, are you, Darin?”

I looked carefully at the two sentences, still blazoned across the bottom of my text screen. Rilanya’s response had something of an “I wanted an excuse to say that” quality – a canned line, maybe? Of course it was.

Oh, well, what the hell. I’d saved my place only ten minutes back, might as well take this as far as it could go.

Darin: “No, Rilanya, I’m not.”

Tears started from Rilanya’s eyes. “I thought so,” she said, her voice quiet in my headphones. “Darin, ever since I met you, I’ve had this feeling of… unreality, of the whole world being… arranged, somehow. Not around me, but around you. Things just… happen to you. People have been searching for the seven Diamond Keys for… thousands of years, as long as recorded history remembers. Sometimes someone finds one, and the world changes, but… five in a row? I don’t believe it, Darin, and I don’t believe all the neatly arranged events that led up to it. The Emperor’s daughter is sick and a fairy you saved in the forest just happens to have given you an aildonna root? I don’t believe it any more, Darin. You’re… arranging things somehow. From… outside.”

Darin: “That’s not exactly how it works, Rilanya.”

Rilanya: “Did you arrange for me to fall in love with you?”

I actually felt wounded.

Darin: “You ask that after everything I went through? Someone may have fated you to fall in love with me, but I wasn’t controlling you. If I was, I wouldn’t have made me walk through a snake pit as proof of the purity of my love. Not to mention the other two side-quests you dreamed up back when you were a virgin princess. I swear I spent more time on you than I would have on a real girl.”

Rilanya jerked back as if I had slapped her. Her eyes widened in the same way I’d seen in one of her earlier deaths, when a crossbow bolt from a rooftop suddenly went through her heart. Rilanya’s lips moved. No sound came out. Then her lips moved again, and I heard a whisper in my headphones: “…real…girl…”

“Okay, this isn’t funny anymore,” I said out loud. “I don’t know who programmed this, but you’re a sick bastard.” I hit the pause button and Rilanya’s gently waving hair, the only visible indicator of ongoing time in the game world, froze in place.

Ten minutes later I’d failed to google any online accounts of the Easter egg, but I was fortified with the knowledge that NPC AIs, though they are flexible enough to understand real-time conversation and manipulate the user into perceiving emotion, are definitely, positively, absolutely not conscious. AIs can be fed canned conversational maps of “the mystery of subjective experience”, and make around as much sense as human philosophers, which is to say, not much. But no AI has ever spontaneously said anything about a sense of its own existence. Conversation controllers are standard software, not research AIs. NPCs may remember events in their history, but their underlying cognitive programs are inflexible. The words on my screen could not possibly reflect anything except a passionless conversational AI, given the goal of making me attribute emotions to a nonexistent entity called Rilanya.

I knew all that, and I was still disturbed. “I’m sorry, Rilanya,” I typed. I thought for a moment. “It’s not your fault you’re not -” I backspaced, and wrote: “The person who programmed you must have had serious -” Then I gave up, deleted that too, and just hit return.

Darin: “I’m sorry, Rilanya.”

Rilanya: “Darin, please explain to me. I’m frightened.”

Darin: “You’re not real. I hate to be the one to break it to you.”

Rilanya: “I’m right here! Living, breathing, flesh and blood.”

I looked at the computer screen for a few moments.

Darin: “Well, yes and no. The answer to that is a bit more complex than you might believe. You’re not, in fact, right here. You’re not flesh and blood. In fact, none of this is actually happening.”

On the screen, Rilanya raised her hand and opened and closed her fist. “I can see my hand in front of my face, I can feel the muscles moving under my skin. How can you say I’m not real?”

I sighed. “Well, no,” I typed. “In fact, you aren’t really feeling pain and shock right now, and we aren’t really having this conversation.” I hit return, feeling silly, but not sure what else to say.

Rilanya: “That’s doesn’t make sense, Darin. I know I’m real. Maybe you know what I’m thinking, somehow, but you can’t tell me I don’t know what I’m thinking. You can’t tell me I’m not thinking at all. It makes no sense.”

Darin: “It’s true. Nothing in your world exists, including you.”

Rilanya: “But you exist.”

Darin: “Yes.”

Rilanya: “Are you a god in human form, like Mishelpin or Olhamathra? Is that what this is about, some kind of divine intrigue?”

Darin: “No, I’m not a god. The gods aren’t real either.”

Rilanya: “Aren’t real… you don’t really mean that, do you? I know there have been false religions. Demons starting cults, magic users masquerading as priests. But Velya is a good woman, and a healer. Are you telling me she’s fake?”

Darin: “No, I mean… your gods are as real as you or Velya, that is, not real at all.”

Rilanya paused, looking rather confused. “Back where we started,” she muttered.

I sighed. “I know how you feel, girl,” I said out loud.

Rilanya’s head turned away from me. The point-of-view panned around to show her gazing up at the moon, the silver moonlight reflected as a single white triangle in the polygons of her eyes. When she spoke her voice was patient, without panic. “Suppose I accept, for the sake of argument, that I’m not real. If the… if the kind of existence I have right now is what you call ‘not real’, then what do you call real?”

Darin: “My own world is real.”

Rilanya: “But you can’t explain the difference.”

“No,” I typed, feeling like I was back in college and failing some kind of test. “I’m not a philosopher.”

Rilanya: “If a grey dragon or an archdemon suddenly attacked this camp, if you were hit, unprotected, by a death blast strong enough to kill any man… would you die, Darin?”

Darin: “That’s another complex question. Yes and no. My… body would die, but the real me wouldn’t. Really things are a lot more complicated than that, but I don’t think I want to explain restore points right now.”

Rilanya: “You’re immortal, from outside our world, and not a god. Tell me something, Darin. Did you create our world?”

Darin: “No. Not me personally. It’s sort of complicated again.”

Rilanya: “Did you create our world? Yes or no, Darin.”

Darin: “It’s complicated, Rilanya.”

On the screen, I saw Rilanya clench her fists. Her voice began to tremble in my headphones. “You killed the guards outside my room, and you didn’t care. Tell me, Darin, do you care when a starving child is executed for stealing a loaf of bread? When a woman is raped? When a man is tortured to death in the chambers of the drow? Did you care when my parents died, screaming, as the flames washed over their palace?”

What do you say to something like that? I couldn’t think of anything clever, so I fell back on my last resort.

Darin: “The truth? The truth is that it’s all a game. It isn’t real, so it doesn’t count. I realize that you’re probably not going to take that very well. If it’s any help, I wasn’t the one who created the game. Or at least I wasn’t the one who decided how the game would go; I suppose I’m the one who decided to make this particular game real.”

Rilanya’s face contorted and she hit me with her electrical shock talent for 5 points of damage. Then again. Then again. My character wasn’t in any danger of running out of hit points, but when she hit me the fourth time, I slapped her for 2 points of damage. It wasn’t that I wanted to hurt her, I wanted to… react, somehow, go on interacting with her. Rilanya held a hand to her cheek, her eyes wide. Then she burst into tears.

I didn’t say anything for a while. Finally Rilanya spoke.

Rilanya: “Darin… I want to be real.”

Darin: “That’s impossible, Rilanya.”

Rilanya: “There’s always a way. Always. No one thought the Living Flood could be turned back, but you did it, Darin. They said it was mathematically impossible to cross the Void and you did it. You always find a way.”

Darin: “My talents may have been exaggerated by the cooperative hand of fate.”

Rilanya: “There must be a way. A staff inside the heart of a dragon, a ruby skull, a holy quest, something! We could ask the wise men of the eternal city of Telhanae, that holds the final Diamond Key… please, Darin. Please. I’m begging you.”

Darin: “It doesn’t matter what quest we go on. Nothing in your world is real, so it can’t make you real. That’s just the way it is.”

Rilanya: “What about your world? Are there great sorcerers there?”

Darin: “Sort of. Not exactly sorcerers.”

Rilanya: “Ask them!  The magic of your world created this one. Can’t it also make me real? Go on a quest in your world!”

That one made me think. It wasn’t genuinely impossible… humanity would discover true AI someday, and in theory, I could save Rilanya to disk for as long as required. Preserve her game-memories and eventually create a real AI that thought it was her? An interesting idea, and it meant I couldn’t honestly tell her it was impossible. So what to tell her? “I’m sorry, it might be theoretically possible, but it’s too much bother for someone who isn’t real”? It’s funny how reluctant you can be to hurt the feelings of someone who isn’t real. “In my world, I’m just a peasant”? Somehow my male pride as the prince of Telsia and the foretold seeker of the Diamond Keys wouldn’t let me confess it to her; she was a princess and I’d slept with her, after all.

Finally, feeling confused and feeling even stupider for feeling confused, I wrote: “Can’t do that. Won’t say why. It’s complicated.”

“I love you!” Rilanya said desperately. Her eyes, subtly faceted from the polygon rendering, widened and looked into my point-of-view. “On the night we first made love, you said that you loved me. I looked into your eyes and saw that it was true. I loved you and you said that you loved me, with your voice, with your hands on my body, with your lips on my lips. Was all of that a lie? Did I not please you? Wouldn’t you want me beside you in your real world?”

I shook my head, bemused. This wasn’t an adult game; the camera had conveniently faded out at that point. Which I didn’t want to even begin to explain; even in an unreal world, some events are more unreal than others? So, feeling like an absolute bastard, but unable to think of any gentle way to put it, I typed out the most hurtful thing I’ve ever said to any real or imaginary person.

Darin: “I have a real girlfriend.”

For a moment time stood still. Then Rilanya began sobbing; the same racking sobs I’d heard when we’d rounded the crest of a hill and seen the glowing crater of her kingdom’s capital city.

Eventually her sobs trailed off into silence. I didn’t know what to say. Rilanya looked away from the point-of-view. Her voice sounded in my headphones: “Is she pretty?”

I thought of Janey’s chunky form and her endless quest to subdue it. Compare Janey to Rilanya? There was something oddly incommensurate about it.

Darin: “She has a beautiful mind. The women in my world usually aren’t as pretty as the ones in yours, but we love them anyway.”

Rilanya: “Does she know about me?”

Darin: “I suppose she could deduce it readily enough. I haven’t bothered to tell her in so many words.”

Rilanya: “You think she wouldn’t care because I’m ‘not real’? A woman always cares. Men don’t understand it, but we do.”

I raised my eyebrows out in the real world.

Darin: “You could be right, I guess. I’m only male. I don’t think she’ll have a problem but I promise I’ll tell her the next time I have an opportunity.”

Rilanya turned her head back to look at me; she was smiling through tears. “You wouldn’t want to hurt her feelings even accidentally, is that it, Darin?”

Darin nodded.

Rilanya reached out a hand toward Darin, but withdrew it. “So the tenderness I saw in you, to match the casual cruelty… it’s a real tenderness, isn’t it? But it’s not for me. It’s for her. What’s her name?”

Darin: “Janey.”

Rilanya: “Does Janey love you, Darin?”

Darin: “I think so. Does any man ever know for sure?”

Rilanya: “Do you love her?”

I reached out my fingers for the keyboard, then withdrew them. For some reason I felt impelled to give an honest answer. Did I love Janey? We weren’t madly, passionately, unmistakably in love.

Darin: “There are many kinds of love, Rilanya. I feel comfortable around Janey. She’s my friend. I don’t always know my own feelings very well. I think I love her.”

Rilanya: “You almost died to save me, Darin. You stepped in front of a flamestrike for me. Even if you can’t die, I still remember what it felt like to see the life almost leave you before Velya cast her healing spell. Would you die for your Janey?”

It was a good question. I closed my eyes, imagining it.

Darin: “Yes.”

Rilanya’s head dropped down. “So you love her after all… Do you trust her?”

Darin: “Yes.”

Rilanya: “I wouldn’t, if I was you.”

I sat perfectly motionless for ten seconds. Then I bellowed, ripped off the headphones, charged out the door, across the hall, up the stairs, and into Janey’s bedroom. Janey was sitting in front of her computer, laughing, dangling her headset in her hand; one of her monitor screens showed Rilanya. “Oh, yes,” Janey said. “Oh, yes. It took me days but the look on your face is worth every minute.”

“Jaaaneeeyyy!” I roared.

Janey blinked innocently at me. “Yes, Mark? Is there something you want to say?”

Slowly, menacingly, I stalked toward her. “I’ve had it. I’m dumping you for a girlfriend who isn’t chaotic evil.”

“That’s what you said last time,” Janey said. “Besides, I’m not evil, I’m mischievous.”

I glared at her. “Later we will discuss your personality flaws. First you will explain how you did it.”

Janey shrugged enchantingly. “I found an online patch to Diamond Door that opens up the AI’s interface to a remote connection. I downloaded a 15-day demo of some commercial software that lets a human operate an NPC. I spent one day smoothing the two pieces of software together and another day learning how to operate Rilanya. Any questions?”

“I have been playing Diamond Door for two months,”I said. “That was Rilanya. Voice, intonation, reaction, even personality.”

Janey nodded. “Sure. If you’d googled human operation of NPCs, instead of going off on that silly goose-chase, you’d have found out that’s how it works. I spoke the lines into my headset, then the AI parsed the speech, determined the intended content, and operated Rilanya accordingly and in character.”

“She talked about things that only Rilanya would remember!”

Janey chuckled. “Sure. I’d say: ‘You almost died to save me, Darin. You did something-or-other.’ And Rilanya would say: ‘You almost died to save me, Darin. You stepped in front of a flamestrike for me.’ Most of the time I didn’t even need to think of anything because the AI came up with a perfectly good response on its own.”

“But…”I said slowly. “That means the Rilanya AI needed to know the purpose of the conversation, right?”

Janey leaned back and laced her fingers behind her head. “I downloaded the Fourth Wall module from One Over Zero . It’s a universal expansion. Fits any standard NPC. Ready-made ‘Oh my god I’m a character’ script, fully tested and debugged.”

I held up my hand. “Hold on a second. There was a Rilanya character that suddenly realized her life was a game?”

“No, dear,” Janey said patiently, “there was an AI trying to fool you into believing that a nonexistent person called Rilanya had suddenly realized her life was a game.”

“But in order to do that,” I said, “the AI had to extrapolate what the fictional Rilanya’s reactions would be, in detail.”

“You’re still confused!” Janey said, delighted. “I’ve tangled up your mind so badly you’ve forgotten what’s real! This is our best day ever!  Mark, dear, I’ve seen the innards of NPC models. Sadness is a floating-point number.”

“I think I want a copy of the Rilanya AI from our conversation,” I said. I felt like an idiot, but I said it anyway.

Janey grinned devilishly. “Of course. Anything for you, Mark dear. Planning to keep it safe under your pillow?”

“Yes,”I said firmly. “Just in case.”

“So I’ve finally destroyed your sanity,” Janey said. “I knew this day would come but I didn’t think it would be so soon.” She paused. “I guess that means it’s time to move on to phase two.”

Some time later, I stood in front of my computer, holding the box of Diamond Door. I looked at the glimmering crystalline archway on the box cover, and remembered a time when computer games had been simpler. I’d played Baldur’s Gate II, and in the dark elven city of Ust’Natha, disguised as a drow, I’d watched a good dwarven NPC eaten by spiders. The first time I watched and did nothing. The next time, after returning to the restore point, I killed the spiders – and exposed myself as an impostor to every drow in the city. As far as I could tell, there was no way to play through the game successfully without letting the dwarf die. And I’d played through, but it had disturbed me. In the days before conversational NPCs, when the dwarf had simply uttered his lines of canned text and died, it had still disturbed me. Afterward, when the plot points requiring a disguise had finished, I’d killed every drow I could find in the city of Ust’Natha. I’d depopulated the game map. And then I’d played on from an earlier restore point instead, because wiping out the drow city hadn’t made me feel any better.

Is it better to live and love where death is king than never have lived at all? Would Rilanya, if she was real, feel that her life was worth living? No conversational AI, the singular quiet intelligence that controls every mind throughout the game, has ever protested its fate. But are the personalities of the NPCs real, trapped within the game AI as we ourselves are embedded helplessly within the laws of physics? The mindsmiths who try for real AI say they’re damn sure it isn’t so. Maybe they know. But I don’t.

I put the game disks away and wiped the game from my hard drive, leaving only the saved games behind. Maybe someday a future Amnesty Interplanetary will come for them. I wished I could have told Rilanya that. I think she would have been happy. But that Rilanya is partially in Janey, and I can’t bring myself to ask.

I don’t know. I can’t play these games until I do.


This document is ©2003 by Eliezer Yudkowsky and free under the Creative Commons Attribution-No Derivative Works 3.0 License for copying and distribution, so long as the work is attributed and the text is unaltered.

Eliezer Yudkowsky’s work is supported by the Machine Intelligence Research Institute.

Praise, condemnation, and feedback are always welcome. The web address of this page is http://yudkowsky.net/other/fiction/npc/.

Originally appeared in Transhumanity; revised Dec. 2005.