Moloch’s Toolbox

There’s a toolbox of reusable concepts for analyzing systems I would call “inadequate”—the causes of civilizational failure, some of which correspond to local opportunities to do better yourself. I shall, somewhat arbitrarily, sort these concepts into three larger categories:

1. Decisionmakers who are not beneficiaries;
2. Asymmetric information;

and above all,

3. Nash equilibria that aren’t even the best Nash equilibrium, let alone Pareto-optimal.

In other words:

1. Cases where the decision lies in the hands of people who would gain little personally, or lose out personally, if they did what was necessary to help someone else;
2. Cases where decision-makers can’t reliably learn the information they need to make decisions, even though someone else has that information; and
3. Systems that are broken in multiple places so that no one actor can make them better, even though, in principle, some magically coordinated action could move to a new stable state.

I will then play fast and loose with these concepts in order to fit the entire Taxonomy of Failure inside them.

For example, “irrationality in the form of cognitive biases” wouldn’t obviously fit into any of these categories, but I’m going to shove it inside “asymmetric information” via a clever sleight-of-hand. Ready? Here goes:

If nobody can detect a cognitive bias in particular cases, then from our perspective we can’t really call it a “civilizational inadequacy” or “failure to pluck a low-hanging fruit.” We shouldn’t even be able to see it ourselves. So, on the contrary, let’s suppose that you and some other people can indeed detect a cognitive bias that’s screwing up civilizational decisionmaking.

Then why don’t you just walk up to the decision-maker and tell them about the bias? Because they wouldn’t have any way of knowing to trust you rather than the other five hundred people trying to influence their decisions? Well, in that case, you’re holding information that they can’t learn from you! So that’s an “asymmetric information problem,” in much the same way that it’s an asymmetric information problem when you’re trying to sell a used car and you know it doesn’t have any mechanical problems, but you have no way of reliably conveying this knowledge to the buyer because for all they know you could be lying.

That argument is a bit silly, but so is the notion of trying to fit the whole Scroll of Woe into three supercategories. And if I named more than three supercategories, you wouldn’t be able to remember them due to computational limitations (which aren’t on the list anywhere, and I’m not going to add them).

i. For want of docosahexaenoic acids, a baby was lost

My discussion of modest epistemology in Chapter 1 might have given the impression that I think of modesty mostly as a certain set of high-level beliefs: beliefs about how best to combat cognitive bias, about how individual competencies stack up against group-level competencies, and so on. But I predict that many of this book’s readers have high-level beliefs similar to those I outlined in Chapter 2, while employing a reasoning style that is really a special case of modest epistemology; and I think that this reasoning style is causing them substantial harm.

As reasoning styles, modest epistemology and inadequacy analysis depend on a mix of explicit principles and implicit mental habits. In inadequacy analysis, it’s one thing to recognize in the abstract that we live in a world rife with systemic inefficiencies, and quite another to naturally perceive systems that way in daily life. So my goal here won’t be to unkindly stick the label “inadequate” to a black box containing the world; it will be to say something about how the relevant systems actually operate.

For our central example, we’ll be using the United States medical system, which is, so far as I know, the most broken system that still works ever recorded in human history. If you were reading about something in 19th-century France which was as broken as US healthcare, you wouldn’t expect to find that it went on working when overloaded with a sufficiently vast amount of money. You would expect it to just not work at all.

In previous years, I would use the case of central-line infections as my go-to example of medical inadequacy. Central-line infections, in the US alone, killed 60,000 patients per year, and infected an additional 200,000 patients at an average treatment cost of $50,000/patient.

Central-line infections were also known to decrease by 50% or more if you enforced a five-item checklist that included items like “wash your hands before touching the line.”

Robin Hanson has old Overcoming Bias blog posts on that untaken, low-hanging fruit. But I discovered while re-Googling in 2015 that wider adoption of hand-washing and similar precautions are now finally beginning to occur, after many years—with an associated 43% nationwide decrease in central-line infections. After partial adoption.¹

So my new example is infants suffering liver damage, brain damage, and death in a way that’s even easier to solve, by changing the lipid distribution of parenteral nutrition to match the proportions in breast milk.

Background: Some babies have digestion problems that require direct intravenous feeding. Long ago, somebody created a hospital formula for this intravenous feeding that matched the distribution of “fat,” “protein,” and “carbohydrate” in breast milk.

Just like “protein” comes in different amino acids, some of which the body can’t make on its own and some of which it can, what early doctors used to think of as “fat” actually breaks down into metabolically distinct elements like short-chain triglycerides, medium-chain triglycerides, saturated fat, and omega-6, omega-9, and the famous “omega-3.” “Omega-3” is actually several different lipids in its own right; vegetable oils with “omega-3” usually just contain alpha-linolenic acids, which can only be inefficiently converted to ecosapentaenoic acids, which are then even more inefficiently converted to docosahexaenoic acids, which are the actual key structural components in the body. This conversion pathway is rate-limited by a process that also converts omega-6, so too much omega-6 can prevent you from processing ALA into DHA even if you’re getting ALA.

So what happens if your infant nutrition was initially designed based on the concept of “fat” as a natural category, and all the “fat” in the mix comes from soybean oil?

From a popular book by Jaminet and Jaminet:

Some babies are born with “short bowel syndrome” and need to be given parenteral nutrition, or nutrition delivered intravenously directly to the blood, until their digestive tracts grow and heal. Since 1961, parenteral nutrition has used soybean oil as its source of fat.[6] And for decades, babies on parenteral nutrition have suffered devastating liver and brain damage. The death rate on soybean oil is 30 percent by age four. […]

In a clinical trial, of forty-two babies given fish oil [after they had already developed liver damage on soybean oil], three died and one required a liver transplant; of forty-nine given soybean oil, twelve died and six required a liver transplant.[8] The death-or-liver-transplant rate was reduced from 37 percent with soybean oil to 9 percent with fish oil.²

When Jaminet and Jaminet wrote the above, in 2012, there was a single hospital in the United States that could provide correctly formulated parenteral nutrition, namely the Boston Children’s Hospital; nowhere else. This formulation was illegal to sell across state lines.

A few years after the Boston Children’s Hospital developed their formula—keeping in mind the heap of dead babies continuing to pile up in the meanwhile—there developed a shortage of “certified lipids” (FDA-approved “fat” for adding to parenteral nutrition). For a year or two, the parenteral nutrition contained no fat at all which is worse and can kill adults.

You see, although there’s nothing special about the soybean oil in parenteral nutrition, there was only one US manufacturer approved to add it, and that manufacturer left the market, so…

As of 2015, the state of affairs was as follows: The FDA eventually solved the problem with the shortage of US-certified lipids, by… allowing US hospitals to import parenteral nutrition bags from Europe. And it only took them two years’ worth of dead patients to figure that out!

As of 2016, if your baby has short bowel syndrome, and has already ended up with liver damage, and either you or your doctor is lucky enough to know what’s wrong and how to fix it, your doctor can apply for a special permit to use a non-FDA-approved substance for your child on an emergency basis. After this, you can buy Omegaven and hope that it cures your baby and that there isn’t too much permanent damage and that it’s not already too late.

This is an improvement over the prior situation, where the non-poisonous formulation was illegal to sell across state lines under any circumstances, but it’s still not good by any stretch of the imagination.

Now imagine trying to explain to a visitor from a relatively well-functioning world just why it is that your civilization has killed a bunch of babies and subjected other babies to pointless brain damage.

“It’s not that we’re evil,” you say helplessly, “it’s that… well, you see, it’s not that anyone wanted to kill those babies, it’s just the way the System ended up, somehow…”

ii. Asymmetric information and lemons problems

Three people have gathered in a blank white space:

• The Visitor from a Better World;
• Simplicio, who is attending a major university but hasn’t taken undergraduate economics;
• Cecie, the Conventional Cynical Economist.

The Visitor speaks first.

Visitor: So I’ve listened to you explain about babies suffering death and brain damage from parenteral nutrition built on soybean oil. I have several questions here, but I’ll start with the most obvious one.

Cecie: Go ahead.

Visitor: Why aren’t there riots?

Simplicio: The first thing you have to understand, Visitor, is that the folk in this world are hypocrites, cowards, psychopaths, and sheep.

I mean, I certainly care about the lives of newborn children. Hearing about their plight certainly makes me want to do something about it. When I see the problem continuing in spite of that, I can only conclude that other people don’t feel the level of moral indignation that I feel when staring at a heap of dead babies.

Cecie: I don’t think that hypothesis is needed, Simplicio. As a start, Visitor, you have to realize that the picture I’ve shown you is not widely known. Maybe 10% of the population, at most, is walking around with the prior belief that the FDA in general is killing people; our government runs on majority rule and the 10% can’t unilaterally defy it.³ Maybe 0.1% of that 10% know that omega-3 ALA is converted into omega-3 DHA via a metabolic pathway that competes with omega-6. And then most of those aren’t aware of what’s happening to babies right now.

Visitor: Pointing to that state of ignorance is hardly a sufficient explanation! If a theater is on fire and only one person knows it, they yell “Fire!” and then more people know it. People from my civilization would scream “Babies are dying over here!” and other people from my civilization would whip around their heads and look.

Simplicio: Our world’s cowards and sheep would hear that and think that it’s (a) somebody else’s problem and (b) all part of the plan.

Cecie: In our world, Visitor, we have an economic phenomenon sometimes called the lemons problem. Suppose you want to sell a used car, and I’m looking for a car to buy. From my perspective, I have to worry that your car might be a “lemon”—that it has a serious mechanical problem that doesn’t appear every time you start the car, and is difficult or impossible to fix. Now, you know that your car isn’t a lemon. But if I ask you, “Hey, is this car a lemon?” and you answer “No,” I can’t trust your answer, because you’re incentivized to answer “No” either way. Hearing you say “No” isn’t much Bayesian evidence. Asymmetric information conditions can persist even in cases where, like an honest seller meeting an honest buyer, both parties have strong incentives for accurate information to be conveyed.

A further problem is that if the fair value of a non-lemon car is $10,000, and the possibility that your car is a lemon causes me to only be willing to pay you $8,000, you might refuse to sell your car. So the honest sellers with reliable cars start to leave the market, which further shifts upward the probability that any given car for sale is a lemon, which makes me less willing to pay for a used car, which incentivizes more honest sellers to leave the market, and so on.

Visitor: What does the lemons problem have to do with your world’s inability to pass around information about dead babies?

Cecie: In our world, there are a lot of people screaming, “Pay attention to this thing I’m indignant about over here!” In fact, there are enough people screaming that there’s an inexploitable market in indignation. The dead-babies problem can’t compete in that market; there’s no free energy left for it to eat, and it doesn’t have an optimal indignation profile. There’s no single individual villain. The business about competing omega-3 and omega-6 metabolic pathways is something that only a fraction of people would understand on a visceral level; and even if those people posted it to their Facebook walls, most of their readers wouldn’t understand and repost, so the dead-babies problem has relatively little virality. Being indignant about this particular thing doesn’t signal your moral superiority to anyone else in particular, so it’s not viscerally enjoyable to engage in the indignation. As for adding a further scream, “But wait, this matter really is important!”, that’s the part subject to the lemons problem. Even people who honestly know about a fixable case of dead babies can’t emit a trustworthy request for attention.

Simplicio: You’re saying that people won’t listen even if I sound really indignant about this? That’s an outrage!

Cecie: By this point in our civilization’s development, many honest buyers and sellers have left the indignation market entirely; and what’s left behind is not, on average, good.

Visitor: Your reply contains so many surprising postulates of weird civilizational dysfunction, I hardly know what to ask about next. So instead I’ll try to explain how my world works, and you can explain to me why your world doesn’t work that way.

Cecie: Sounds reasonable.

iii. Academic incentives and beneficiaries

Visitor: To start with, in my world, we have these people called “scientists” who verify claims experimentally, and other people trust the “scientists.” So if our “scientists” say that a certain formula seems to be killing babies, this would provoke general indignation without every single listener needing to study docohexa-whatever acids.

Simplicio: Alas, our so-called scientists are just pawns of the same medical-industrial complex that profits from killing babies.

Cecie: I’m afraid, Visitor, that although there are strong prior reasons to expect too much omega-6 and no omega-3 to be very bad for an infant baby, and there are now a few dozen small-scale studies which seem to match that prediction, this matter hasn’t had the massive study that would begin to produce confident scientific agreement—

Visitor: You’d better not be pointing to that as an exogenous fact that explains your civilization’s problem! See, on my planet, if somebody points to strong prior suspicion combined with confirming pilot studies saying that something is killing innocent babies and is fixable, and the pilot studies are not considered sufficient evidence to settle the issue, our people would do more studies and wouldn’t just go on blindly feeding the babies poison in the meantime. Our scientists would all agree on that!

Cecie: But people loudly agreeing on something, by itself, accomplishes nothing. It’s all well and good for everyone to agree in principle that larger studies ought to be done; but in your world, who actually does the big study, and why do they do it?

Visitor: Two subclasses within the profession of “scientist” are suggesters, whose piloting studies provide the initial suspicions of effects, and replicators whose job it is to confirm the result and nail things down solidly—the exact effect size and so on. When an important suggestive result arises, two replicators step forward to confirm it and nail down the exact conditions for producing it, being forbidden upon their honor to communicate with each other until they submit their findings. If both replicators agree on the particulars, that completes the discovery. The three funding bodies that sustained the suggester and the dual replicators would receive the three places of honor in the announcement. Do I need to explain how part of the function of any civilized society is to appropriately reward those who contribute to the public good?

Cecie: Well, that’s not how things work on Earth. Our world gives almost all the public credit and fame to the discoverer, as the initial suggester is called among us. Our scientists often say that replication is important, but our most prestigious journals won’t publish mere replications; nor do the history books remember them. The outcome is a lot of small studies that have just enough subjects to obtain “statistically significant” results—

Visitor: … What? Probability is quantitative, not qualitative. There’s no such thing as a “significant” or “insignificant” likelihood ratio—

Cecie: Anyway, while it might be good if larger studies were done, the decisionmaker is not the beneficiary—the people who did the extra work of a larger study, and funded the extra work of a larger study, would not receive fame and fortune thereby.

Visitor: I must be missing something basic here. You do have multiple studies, right? When you have multiple bodies of data, you can multiply the likelihood functions from the studies’ respective data to the hypotheses to obtain the meaning of the combined evidence—the likelihood function from all the data to the hypotheses.⁴

Cecie: I’m afraid you can’t do that on Earth.

Visitor: … Of course you can. It’s a mathematical theorem. You can’t possibly tell me that differs between our universes!

Yes, there are pitfalls for the especially careless. Sometimes studies end up being conducted under different circumstances, with the result that the naively computed likelihood functions don’t have uniform relations to the hypotheses under consideration. In that case, blindly multiplying will give you a likelihood function that’s nearly zero everywhere. But, I mean, if you just look at all the likelihood functions, it’s pretty obvious when some of them are pointing in different directions and then you can investigate that divergence.

Either it makes sense to multiply all the likelihood functions and get out one massive evidential pointer, or else you don’t get a sensible result when you multiply them and then you know something’s wrong with your methods—

Cecie: I’m afraid our scientific community doesn’t run on your world’s statistical methods. You see, during the first half of the twentieth century, it became conventional to measure something called “p-values” which imposed a qualitative distinction between “successful” and “unsuccessful” experiments—

Visitor: That is still not an explanation. Why not change the way you do things?

Cecie: Because somebody who tried using unconventional statistical methods, even if they were better statistical methods, wouldn’t be able to publish their papers in the most prestigious journals. And then they wouldn’t get hired. It’s similar to the way that the most prestigious journals don’t publish mere replications, only discoveries, so people focus on making discoveries instead of replications.

Visitor: Why would anyone pay attention to journals like that?

Cecie: Because university hiring departments care a lot about whether you’ve published in prestigious journals.

Visitor: No, I mean… how did these journals end up prestigious in the first place? Why do university hiring departments pay attention to them?

Simplicio: Why would university hiring departments care about real science? Shouldn’t it be you who has to explain why some lifeless cog of the military-industrial complex would care about anything except grant money?

Cecie: Okay… you’re digging pretty deep here. I think I need to back up and try to explain things on a more basic level.

Visitor: Indeed, I think you should. So far, every time I’ve asked you why someone is acting insane, you’ve claimed that it’s secretly a sane response to someone else acting insane. Where does this process bottom out?

iv. Two-factor markets and signaling equilibria

Cecie: Let me try to identify a first step on which insanity can emerge from non-insanity. Universities pay attention to prestigious journals because of a signaling equilibrium, which, in our taxonomy, is a kind of bad Nash equilibrium that no single actor can defy unilaterally.

In your terms, it involves a sticky, stable equilibrium of everyone acting insane in a way that’s secretly a sane response to everyone else acting insane.

Visitor: Go on.

Cecie: First, let me explain the idea of what Eliezer has nicknamed a “two-factor market.” Two-factor markets are a conceptually simpler case that will help us later understand signaling equilibria.

In our world there’s a crude site for classified ads, called Craigslist. Craigslist doesn’t contain any way of rating users, the way that eBay lets buyers and sellers rate each other, or that Airbnb lets renters and landlords rate each other.

Suppose you wanted to set up a version of Craigslist that let people rate each other. Would you be able to compete with Craigslist?

The answer is that even if this innovation is in fact a good one, competing with Craigslist would be far more difficult than it sounds, because Craigslist is sustained by a two-factor market. The sellers go where there are the most buyers; the buyers go where they expect to find sellers. When you launch your new site, no buyers will want to go there because there are no sellers, and no sellers will want to go there because there are no buyers. Craigslist initially broke into this market by targeting San Francisco particularly, and spending marketing effort to assemble the San Francisco buyers and sellers into the same place. But that would be harder to do for a later startup, because now the people it’s targeting are already using Craigslist.

Simplicio: Those sheep! Just mindlessly doing whatever their incentives tell them to!

Cecie: We can imagine that there’s a better technology than Craigslist, called Danslist, such that everyone using Craigslist would be better off if they all switched to Danslist simultaneously. But if just one buyer or just one seller is the first to go to Danslist, they find an empty parking lot. In conventional cynical economics, we’d say that this is a coordination problem—

Simplicio: A coordination problem? What do you mean by that?

Cecie: Backing up a bit: A “Nash equilibrium” is what happens when everyone makes their best move, given that all the other players are making their best moves from that Nash equilibrium—everyone goes to Craigslist, because that’s their individually best move given that everyone else is going to Craigslist. A “Pareto optimum” is any situation where it’s impossible to make every actor better off simultaneously, like “Cooperate/Cooperate” in the Prisoner’s Dilemma—there’s no alternative outcome to Cooperate/Cooperate that makes both agents better off. The Prisoner’s Dilemma is a coordination problem because the sole Nash equilibrium of Defect/Defect isn’t Pareto-optimal; there’s an outcome, Cooperate/Cooperate, that both players prefer, but aren’t reaching.

Simplicio: How stupid of them!

Cecie: No, it’s… ah, never mind. Anyway, the frustrating parts of civilization are the times when you’re stuck in a Nash equilibrium that’s Pareto-inferior to other Nash equilibria. I mean, it’s not surprising that humans have trouble getting to non-Nash optima like “both sides cooperate in the Prisoner’s Dilemma without any other means of enforcement or verification.” What makes an equilibrium inadequate, a fruit that seems to hang tantalizingly low and yet somehow our civilization isn’t plucking, is when there’s a better stable state and we haven’t reached it.

Visitor: Indeed. Moving from bad equilibria to better equilibria is the whole point of having a civilization in the first place.

Cecie: Being stuck in an inferior Nash equilibrium is how I’d describe the frustrating aspect of the two-factor market of buyers and sellers that can’t switch from Craigslist to Danslist. The scenario where everyone is using Danslist would be a stable Nash equilibrium, and a better Nash equilibrium. We just can’t get there from here. There’s no one actor who is behaving foolishly; all the individuals are responding strategically to their incentives. It’s only the larger system that behaves “foolishly.” I’m not aware of a standard term for this situation, so I’ll call it an “inferior equilibrium.”

Simplicio: Why do you care what academics call it? Why not just use the best phrase?

Cecie: The terminology “inferior equilibrium” would be fine if everyone else were already using that terminology. Mostly I want to use the same phrase that everyone else uses, even if it’s not the best phrase.

Simplicio: Regardless, I’m not seeing what the grand obstacle is to people solving these problems by, you know, coordinating. If people would just act in unity, so much could be done!

I feel like you’re placing too much blame on system-level issues, Cecie, when the simpler hypothesis is just that the people in the system are terrible: bad at thinking, bad at caring, bad at coordinating. You claim to be a “cynic,” but your whole world-view sounds rose-tinted to me.

Visitor: Even in my world, Simplicio, coordination isn’t as simple as everyone jumping simultaneously every time one person shouts “Jump!” For coordinated action to be successful, you need to trust the institution that says what the action should be, and a majority of people have to trust that institution, and they have to know that other people trust the institution, so that everyone expects the coordinated action to occur at the critical time, so that it makes sense for them to act too.

That’s why we have policy prediction markets and… there doesn’t seem to be a word in your language for the timed-collective-action-threshold-conditional-commitment… hold on, this cultural translator isn’t making any sense. “Kickstarter”? You have the key concept, but you use it mainly for making video games?

Cecie: I’ll now introduce the concept of a signaling equilibrium.

To paraphrase a commenter on Slate Star Codex: suppose that there’s a magical tower that only people with IQs of at least 100 and some amount of conscientiousness can enter, and this magical tower slices four years off your lifespan. The natural next thing that happens is that employers start to prefer prospective employees who have proved they can enter the tower, and employers offer these employees higher salaries, or even make entering the tower a condition of being employed at all.⁵

Visitor: Hold on. There must be less expensive ways of testing intelligence and conscientiousness than sacrificing four years of your lifespan to a magical tower.

Cecie: Let’s not go into that right now. For now, just take as an exogenous fact that employers can’t get all of the information they want by other channels.

Visitor: But—

Cecie: Anyway: the natural next thing that happens is that employers start to demand that prospective employees show a certificate saying that they’ve been inside the tower. This makes everyone want to go to the tower, which enables somebody to set up a fence around the tower and charge hundreds of thousands of dollars to let people in.⁶

Visitor: But—

Cecie: Now, fortunately, after Tower One is established and has been running for a while, somebody tries to set up a competing magical tower, Tower Two, that also drains four years of life but charges less money to enter.

Visitor: … You’re solving the wrong problem.

Cecie: Unfortunately, there’s a subtle way in which this competing Tower Two is hampered by the same kind of lock-in that prevents a jump from Craigslist to Danslist. Initially, all of the smartest people headed to Tower One. Since Tower One had limited room, it started discriminating further among its entrants, only taking the ones that have IQs above the minimum, or who are good at athletics or have rich parents or something. So when Tower Two comes along, the employers still prefer employees from Tower One, which has a more famous reputation. So the smartest people still prefer to apply to Tower One, even though it costs more money. This stabilizes Tower One’s reputation as being the place where the smartest people go.

In other words, the signaling equilibrium is a two-factor market in which the stable point, Tower One, is cemented in place by the individually best choices of two different parts of the system. Employers prefer Tower One because it’s where the smartest people go. Smart employees prefer Tower One because employers will pay them more for going there. If you try dissenting from the system unilaterally, without everyone switching at the same time, then as an employer you end up hiring the less-qualified people from Tower Two, or as an employee, you end up with lower salary offers after you go to Tower Two. So the system is stable as a matter of individual incentives, and stays in place. If you try to set up a cheaper alternative to the whole Tower system, the default thing that happens to you is that people who couldn’t handle the Towers try to go through your new system, and it acquires a reputation for non-prestigious weirdness and incompetence.

Visitor: This all just seems so weird and complicated. I’m skeptical that this scenario with the magical towers could happen in real life.

Simplicio: I agree that trying to build a cheaper Tower Two is solving the wrong problem. The interior of Tower One boasts some truly exquisite architecture and decor. It just makes sense that someone should pay a lot to allow people entry to Tower One. What we really need is for the government to subsidize the entry fees on Tower One, so that more people can fit inside.

Cecie: Consider a simpler example: Velcro is a system for fastening shoes that is, for at least some people and circumstances, better than shoelaces. It’s easier to adjust three separate Velcro straps then it is to keep your shoelaces perfectly adjusted at all loops, it’s faster to do and undo, et cetera, and not everyone is running at high speeds that call for perfectly adjusted running shoes. But when Velcro was introduced, the earliest people to adopt Velcro were those who had the most trouble tying their shoelaces—very young children and the elderly. So Velcro became associated with kids and old people, and thus unforgivably unfashionable, regardless of whether it would have been better than shoelaces in some adult applications as well.

Visitor: I take it you didn’t have the stern and upright leaders, what we call the Serious People, who could set an example by donning Velcro shoes themselves?

Simplicio & Cecie: (in unison) No.

Visitor: I see.

Cecie: Now consider the system of scientific journals that we were originally talking about. Some journals are prestigious. So university hiring committees pay the most attention to publications in that journal. So people with the best, most interesting-looking publications try to send them to that journal. So if a university hiring committee paid an equal amount of attention to publications in lower-prestige journals, they’d end up granting tenure to less prestigious people. Thus, the whole system is a stable equilibrium that nobody can unilaterally defy except at cost to themselves.

Visitor: I’m still skeptical. Doesn’t your parable of the magical tower suggest that, if that’s actually true, somebody ought to rope off the journals too and charge insane amounts of money?

Cecie: Yes, and that’s exactly what happened. Elsevier and a few other profiteers grabbed the most prestigious journals and started jacking up the access costs. They contributed almost nothing—even the peer review and editing was done by unpaid volunteers. Elsevier just charged more and more money and sat back. This is standardly called rent-seeking. In a few cases, the scientists were able to kickstart a coordinated move where the entire editing board would resign, start a new journal, and everybody in the field would submit to the new journal instead. But since our scientists don’t have recognized kickstarting customs, or any software support for them, it isn’t easy to pull that off. Most of the big-name journals that Elsevier has captured are still big names, still getting prestigious submissions, and still capturing big-money rents.

Visitor: Well, I guess I understand why my cultural translator keeps putting air quotes around Earth’s version of “science.” The whole idea of science, as I understand the concept, is that everything has to be in the open for anyone to verify. Science is the part of humanity’s knowledge that everyone can potentially learn about and reproduce themselves. You can’t charge money in order for people to read your experimental results, or you lose the “everyone can access and verify your claims” property that distinguishes science from other kinds of information.

Cecie: Oh, rest assured that scientists aren’t seeing any of this money. It all goes to the third-party journal owners.

Simplicio: And this isn’t just scientists being stupid?

Cecie: No stupider than you are for going to college. It’s hard to beat signaling equilibria—because they’re “multi-factor markets”—which are special cases of coordination problems that create “inferior Nash equilibria”—which are so stuck in place that market controllers can seek rent on the value generated by captive participants.

Simplicio: Weren’t we talking about dead babies at some point?

Cecie: Yes, we were. I was explaining how our system allocated too much credit to discoverers and not enough credit to replicators, and the only socially acceptable statistics couldn’t aggregate small-scale trials in a way regarded as reliable. The Visitor asked me why the system was like that. I pointed to journals that published a particular kind of paper. The Visitor asked me why anyone paid attention to those journals in the first place. I explained about signaling equilibria, and that’s where we are now.

Visitor: I can’t say that I feel enlightened at the end of walking through all that. There must be particular scientists on the editorial boards who choose not to demand replications and who forbid multiplying likelihood ratios. Why are those particular scientists doing the non-sensible thing?

Cecie: Because people in the general field wouldn’t cite nonstandard papers, so if the editors demanded nonstandard papers, the journal’s publication factor would decrease.

Visitor: Why don’t the journal editors start by demanding that paper submitters cite dual replications as well as initial suggestions?

Cecie: Because that would be a weird unconventional demand, which might lead people with high-prestige results to submit those results to other journals instead. Fundamentally, you’re asking why scientists on Earth don’t adopt certain new customs that you think would be for the good of everyone. And the answer is that there’s this big, multi-factor system that nobody can dissent from unilaterally, and that people have a lot of trouble coordinating to change. That’s true even when there are forces like Elsevier that are being blatant about ripping everyone off. Implementing your proposed cultural shift to “suggesters” and “replicators,” or using likelihood functions, would be significantly harder than everyone just simultaneously ceasing to deal with Elsevier, since the case for it would be less obvious and would provoke more disagreement. All that we can manage is to make incremental shifts toward funding more replication and asking more for study preregistration.

To sum up, academic science is embedded in a big enough system with enough separate decisionmakers creating incentives for other decisionmakers that it almost always takes the path of least resistance. The system isn’t in the best Nash equilibrium because nobody has the power to look over the system and choose good Nash equilibria. It’s just in a Nash equilibrium that it wandered into, which includes statistical methods that were invented in the first half of the 20th century and editors not demanding that people cite replications.

Visitor: I see. And that’s why nobody in your world has multiplied the likelihood functions, or done a large-enough single study, or otherwise done whatever it would take to convince whoever needs to be convinced about the effects of feeding infants soybean oil.

Cecie: It’s one of the reasons. A large study would also be very expensive because of extreme paperwork requirements, generated by other systemic failures I haven’t gotten around to talking about yet—⁷

Visitor: How does anything get done ever, in your world?

Cecie: —and when it comes to funding or carrying out that bigger study, the decisionmaker would not significantly benefit under the current system, which is held in place by coordination problems. And that’s why people who already have a background grasp of lipid metabolic pathways have asymmetric information about what is worth becoming indignant about.

v. Total market failures

Visitor: Even granting the things you’ve said already, I don’t feel like I’ve been told enough to understand why your society is killing babies.

Cecie: Well, no. Not yet. The lack of incentive to do a large-scale convincing study is only one thing that went wrong inside one part of the system. There’s a lot more broken than just that—which is why effective altruists shouldn’t be running out and trying to fund a big replication study for Omegaven, because that by itself wouldn’t fix things.

Visitor: Okay, suppose there had been a large enough study to satisfy your world’s take on “scientists.” What else would likely go wrong after that?

Cecie: Several things. For example, doctors wouldn’t necessarily be aware of the experimental results.

Visitor: Hold on, I think my cultural translator is broken. You used that word “doctor” and my translator spit out a long sequence of words for Examiner plus Diagnostician plus Treatment Planner plus Surgeon plus Outcome Evaluator plus Student Trainer plus Business Manager. Maybe it’s stuck and spitting out the names of all the professions associated with medicine.

Cecie: So, in your world, if there is a dual replication of results on Omegaven versus soybean oil, how does that end up changing the actual patient treatments?

Visitor: By informing the Treatment Planners who specialize in infant ailments that required parenteral nutrition, of course. The discovery would appear inside the “parenteral nutrition” pages in the Earthweb and show up in the feeds of everyone subscribed to that page. The statistics would appear inside the Treatment Planner’s decision-support software. And if all of those broke for some reason, every Treatment Planner for infant ailments that required parenteral nutrition would just use chatrooms. And anyone who ignored the chatrooms would have worse patient outcome ratings, and would lose status relative to Treatment Planners who were more attentive.

Cecie: It sounds like “Treatment Planners” in your world are much more specialized than doctors in this world. I suppose they’re also selected specifically for talent at… cost-benefit analysis and decision theory, or something along those lines? And then they focus their learning on particular diseases for which they are Treatment Planners? And somebody else tracks their outcomes?

Visitor: Of course. I’m… almost afraid to ask, but how do they do it in your world?

Cecie: Your translator wasn’t broken. In our world, “doctors” are supposed to examine patients for symptoms, diagnose especially complicated or obscure ailments using their encyclopedic knowledge and their keen grasp of Bayesian inference, plan the patient’s treatment by weighing the costs and benefits of the latest treatments, execute the treatments using their keen dexterity and reliable stamina, evaluate for themselves how well that went, train students to do it too, and in many cases, also oversee the small business that bills the patients and markets itself. So “doctors” have to be selected for all of those talents simultaneously, and then split their training, experience, and attention between them.

Visitor: Why in the name of—

Cecie: Oh, and before they go to medical school, we usually send them off to get a four-year degree in philosophy first or something, just because.

I don’t know if there’s a standard name for this phenomenon, but we can call it “failure of professional specialization.” It also appears when, for example, a lawyer has to learn calculus in order to graduate college, even though their job doesn’t require any calculus.

Visitor: Why. Why. Why why why—

Cecie: I’m not sure. I suspect the origin has something to do with status—like, a high-status person can do all things at once, so it’s insulting and lowers status to suggest that an esteemed and respectable Doctor should only practice one surgical operation and get very good at it. And once you yourself have spent twelve years being trained under the current system, you won’t be happy about the proposal to replace it with two years of much more specialized training. Once you’ve been through a painful initiation ritual and rationalized its necessity, you’ll hate to see anyone else going through a less painful one. Not to mention that you won’t be happy about the competition against your own human capital, by a cheaper and better form of human capital—and after the sunk cost in pain and time that you endured to build human capital under the old system…

Visitor: Do they not have markets on your planet? Because on my planet, when you manufacture your product in a crazy, elaborate, expensive way that produces an inferior product, someone else will come along and rationalize the process and take away your customers.

Cecie: We have markets, but there’s this unfortunate thing called “regulatory capture,” of which one kind is “occupational licensing.”

As an example, it used to be that chairs were carefully hand-crafted one at the time by carpenters who had to undergo a lengthy apprenticeship, and indeed, they didn’t like it when factories came along staffed by people who specialized in just carving a single kind of arm. But the factory-made chairs were vastly cheaper and most of the people who insisted on sticking to handcrafts soon went out of business.

Now imagine: What if the chair-makers had been extremely respectable—had already possessed very high status? What if their profession had an element of danger? What if they’d managed to frighten everyone about the dangers of improperly made chairs that might dump people on the ground and snap their necks?

Visitor: Okay, yes, we used to have Serious People who would go around and certify the making of some medicines where somebody might be tempted to cheat and use inferior ingredients. But that was before computers and outcome statistics and online ratings.

Cecie: And on our planet, Uber and Lyft are currently fighting it out with taxi companies and their pet regulators after exactly that development. But suppose the whole system was set up before the existence of online ratings. Then the carpenters might have managed to introduce occupational licensing on who could be a carpenter. So if you tried to set up a factory, your factory workers would have needed to go through the traditional carpentry apprenticeship that covered every part of every kind of furniture, before they were legally allowed to come to your factory and specialize in carving just one kind of chair-arm. And then your factory would also need a ton of permits to sell its furniture, and would need to inveigle orders from a handful of resellers who were licensed to buy and resell furniture at a fixed margin. That small, insular group of resellers might not benefit literally personally—in their own personal salary—from buying from your cheaper factory system. And so it would go.

Visitor: But why would the legislators go along with that?

Cecie: Because the carpenters would have a big, concentrated incentive to figure out how to make legislators do it—maybe by hiring very persuasive people, or by subtle bribery, or by not-so-subtle bribery.

Insofar as occupational licensing works to the benefit of professionals at the expense of consumers, occupational licensing represents a kind of regulatory capture, which happens when a few regulatees have a much more concentrated incentive to affect the regulation process. Regulatory capture in turn is a kind of commons problem, since every citizen shares the benefits of non-captured regulation, but no individual citizen has a sufficient incentive to unilaterally spend their life attending to that particular regulatory problem. So occupational licensing is regulatory capture is a commons problem is a coordination problem.

Visitor: Then… the upshot is that it’s impossible for your country to test a functional hospital design in the first place? The reformers can’t win the competition because they’re not legally allowed to try?

Cecie: But of course. Though in this case, if you did manage to set up a test hospital working along more reasonable lines, you still wouldn’t be able to advertise your better results relative to any other hospitals. With just a few isolated exceptions, all of the other hospitals on Earth don’t publish patient outcome statistics in the first place.

Visitor: … But… then—what are they even selling?

Simplicio: Hold on. If you reward the doctors with the highest patient survival rates, won’t they just reject all the patients with poor prognoses?

Visitor: Obviously you don’t evaluate raw survival rates. You have Diagnosticians who estimate prognosis categories and are rated on their predictive accuracy, and Treatment Planners and Surgeons who are rated on their relative outcomes, and you have the outcomes evaluated by a third party, and—

Cecie: In our world, there’s no separation of powers where one person assigns patients a prognosis category and has their prediction record tracked, and another person does their best to treat them and has their treatment record tracked. So hospitals don’t publish any performance statistics, and patients choose the hospital closest to their house that takes their workplace’s insurance, and nobody has any financial incentive to decrease the number of patient deaths from sloppy surgeons or central line infections. When anesthesiologists in particular did happen to start tracking patient outcomes, they adopted some simple monitoring standards and subsequently decreased their fatality rates by a factor of one hundred.⁸ But that’s just anesthesiologists, not, say, cardiac surgeons.

With cardiac surgeons, a group of researchers recently figured out how to detect when the most senior cardiac surgeons were at conferences, and found that the death rates went down while the most senior cardiac surgeons were away.⁹ But our scientists have to use special tricks if they want to find out any facts like that.

Visitor: Do your patients not care if they live or die?

Cecie: Robin Hanson has a further thesis about how what people really want from medicine is reassurance rather than statistics. But I’m not sure that hypothesis is necessary to explain this particular aspect of the problem. If no hospital offers statistics, then you have no baseline to compare to if one hospital does start offering statistics. You’d just be looking at an alarming-looking percentage for how many patients die, with no idea of whether that’s a better percentage or a worse percentage. Terrible marketing! Especially compared to that other hospital across town that just smiles at you reassuringly.

No hospital would benefit from being the first to publish statistics, so none of them do.

Visitor: Your world has literally zero market demand for empirical evidence?

Cecie: Not zero, no. But since publishing scary numbers would be bad marketing for most patients, and hospitals are heavily regional, they all go by the majority preference to not hear about the statistics.

Visitor: I confess I’m having some trouble grasping the concept of a market consisting of opaque boxes allegedly containing goods, in which nobody publishes what is inside the boxes.

Cecie: Hospitals don’t publish prices either, in most cases.

Visitor: …

Cecie: Yeah, it’s pretty bad even by Earth standards.

Visitor: You literally don’t have a healthcare market. Nobody knows what outcomes are being sold. Nobody knows what the prices are.

Cecie: I guess we could call that Total Market Failure? As in, things have gone so wrong that there’s literally no supply-demand matching or price-equilibrating mechanism remaining, even though money is still changing hands.

And while I wish that this phenomenon of “you simply don’t have a market” were only relevant to healthcare and not to other facets of our civilization… well, it’s not.

vi. Absence of (meta-)competition

Visitor: I suppose I can imagine a hypothetical world in which one country screws things up as badly as you describe. But your planet has multiple governments, I thought. Or did I misunderstand that? Why wouldn’t patients emigrate to—or just visit—countries that made better hospitals legal?

Cecie: The forces acting on governments with high technology levels are mostly the same between countries, so all the governments of those countries tend to have their medical system screwed up in mostly the same way (not least because they’re imitating each other). Some aspects of dysfunctional insurance and payment policies are special to the US, but even the relatively functional National Health System in Britain still has failure of professional specialization. (Though they at least don’t require doctors to have philosophy degrees.)

Visitor: Is there not one government that would allow a reasonably designed hospital staffed by specialists instead of generalists?

Cecie: It wouldn’t be enough to just have one government’s okay. You’d need some way to initially train your workers, despite none of our world’s medical schools being set up to train them. A majority of legislators won’t benefit personally from deciding to let you try your new hospital in their country. Furthermore, you couldn’t just go around raising money from rich countries for a venture in a poor country, because rich countries have elaborate regulations on who’s allowed to raise money for business ventures through equity sales. The fundamental story is that everything, everywhere, is covered with varying degrees of molasses, and to do any novel thing you have to get around all of the molasses streams simultaneously.

Visitor: So it’s impossible to test a functional hospital design anywhere on the planet?

Cecie: But of course.

Visitor: I must still be missing something. I just don’t understand why all of the people with economics training on your planet can’t go off by themselves and establish their own hospitals. Do you literally have people occupying every square mile of land?

Cecie: … How do I phrase this…

All useful land is already claimed by some national government, in a way that the international order recognizes, whether or not that land is inhabited. No relevant decisionmaker has a personal incentive to allow there to be unclaimed land. Those countries will defend even a very small patch of that claimed land using all of the military force their country has available, and the international order will see you as the aggressor in that case.

Visitor: Can you buy land?

Cecie: You can’t buy the sovereignty on the land. Even if you had a lot of money, any country poor enough and desperate enough to consider your offer might just steal your stuff after you moved in.

Negotiating the right to bring in weapons to defend yourself in this kind of scenario would be even more unthinkable, and would spark international outrage that could prevent you from trading with other countries.

To be clear, it’s not that there’s a global dictator who prevents new countries from popping up; but every potentially useful part of every land is under some system’s control, and all of those systems would refuse you the chance to set up your own alternative system, for very similar reasons.

Visitor: So there’s no way for your planet to try different ways of doing things, anywhere. You literally cannot run experiments about things like this.

Cecie: Why would there be? Who would decide that, and how would they personally benefit?

Visitor: That sounds extremely alarming. I mean, difficulties of adoption are one thing, but not even being able to try new things and see what happens… Shouldn’t everyone on your planet be able to detect at a glance how horrible things have become? Can this type of disaster really stand up to universal agreement that something is wrong?

Cecie: I’m afraid that our civilization doesn’t have a sufficiently stirring and narratively satisfying conception of the valor of “testing things” that our people would be massively alarmed by its impossibility. And now, Visitor, I hope we’ve bottomed out the general concept of why people can’t do things differently—the local system’s equilibrium is broken, and the larger system’s equilibrium makes it impossible to flee the game.

Visitor: Okay, look… despite everything you’ve said so far, I still have some trouble understanding why doctors and parents can’t just not kill the babies. I manage to get up every single morning and successfully not kill any babies. It’s not as hard as it sounds.

Cecie: I worry you’re starting to think like Simplicio. You can’t just not kill babies and expect to get away with it.

Simplicio: I actually agree with Cecie here. The evil people behind the system hate those who defy them by behaving differently; there’s no way they’d countenance anyone departing from the norm. What we really need is a revolution, so we can depose our corrupt overlords, and finally be free to coordinate, and…!

Cecie: There’s no need to add in any evil conspiracy hypotheses here.

It’s sufficient to note that the system is in equilibrium and it has causes for the equilibrium settling there—causes, if not justifications. You can’t go against the system’s default without going against the forces that underpin that default. A doctor who gives a baby a nutrition formula that isn’t FDA-approved will lose their job. A hospital that doesn’t fire that kind of doctor will be sued. A scientist that writes proposals for a big, expensive, definitive study won’t get a grant, and while they were busy writing those failed grant proposals, they’ll have lost their momentum toward tenure. So no, you can’t just try out a competing policy of not killing babies. Not more than once.

Visitor: Have you tried?

Cecie: No.

Visitor: But—

Cecie: Anyway, from my perspective, it’s no surprise if you don’t yet feel like you understand. We’ve only begun to survey the malfunctions of the whole system, which would further include the FDA, and the clinical trials, and the p-hacking. And the way venture capital is structured, and equity-market regulations. And the insurance companies, and the tax code. And the corporations who contract with the insurance companies. And the corporations’ employees. And the politicians. And the voters.

Visitor: … Consider me impressed that your planet managed to reach this level of dysfunction without actually physically bursting into flames.

vii. Sticky traditions in belief-dependent Nash equilibria without common knowledge

Cecie: I could talk next about a tax system that makes it cheaper for corporations to pay for care instead of patients, and how that sets up a host of “decisionmaker is not the beneficiary” problems. But I suspect a lot of people reading this conversation understand that part already, so instead I’ll turn my attention to venture capital.

Visitor: It sounds like the “politicians” and the “voters” might be a more key issue, if the cultural translator is right about what those correspond to.

Cecie: Ah! But it turns out that venture capitalists and startups can be seen as a simpler version of voters and politicians, so it’s better to consider entrepreneurs first.

Besides, at this point I imagine the Visitor is wondering, “Why can’t anyone make any money by saving those babies? Doesn’t your society have a profit incentive that fixes this?”

Visitor: Actually, I don’t think that was high on my list of questions. It’s understood among my people that not every problem is one you can make a profit by fixing—persistent societal problems tend to be ones that don’t have easily capturable profits corresponding to their solution.

I mean, yes, if this was all happening on our world and it wasn’t already being addressed by the Serious People, then somebody would just mix the bleeping nutrients and sell it to the bleeping parents for bleeping money. But at this point I’ve already guessed that’s going to be illegal, or saving babies using money is going to be associated with the wrong Tower and therefore unprestigious, or your parents are using a particular kind of statistical analysis that requires baby sacrifices, or whatever.

Cecie: Hey, details matter!

Visitor: (in sad reflection) Do they? Do they really? Isn’t there some point where you just admit you can’t stop killing babies and it doesn’t really matter why?

Cecie: No. You can never say that if you want to go on being a cynical economist.

Now, there are several different kinds of molasses covering the world of startups and venture capital. It’s the tradition-bound aspects of that ecosystem that we’ll find especially interesting, since according to its own ideology, venture capitalists are supposed to chase strange new ideas that other venture capitalists don’t believe in. Walking through the simpler case of venture capital will help us understand the more complex reasons why voters and politicians are nailed into their own equilibria, underpinning the ultimate reasons why nobody can change the laws that prevent change.

Visitor: (gazing off into the distance) … I wonder if maybe there are some worlds that can’t be saved.

Cecie: Suppose it’s widely believed that the most successful entrepreneurs have red hair. If you’re an unusually smart venture capital company that realizes that, a priori, hair color doesn’t seem like it should correlate to entrepreneurial ability, you might think you could make an excess profit by finding some overlooked entrepreneur with blonde hair.

The key insight here is that venture capital is a multi-stage process. There’s the initial or pre-seed round, the seed round, the Series A, the Series B, the middle rounds, the Series C… and if the startup fails to raise money on any of those rounds before they become durably profitable, they’re dead. What this means is that the seed-round investors need to consider the probability that the company can successfully raise a Series A. If the angels invest in the seed round of a company whose entrepreneurs don’t have red hair, that company won’t be able to raise a Series A and will go bust and the angel investment will be worthless. So the angel investors need to decide where to invest, and what price to offer, based partially on their beliefs about what most Series A investors believe.

Simplicio: Ah, I’ve heard of this. It’s called a Keynesian beauty contest, where everyone tries to pick the contestant they expect everyone else to pick. A parable illustrating the massive, pointless circularity of the paper game called the stock market, where there’s no objective except to buy the pieces of paper you’ll think other people will want to buy.

Cecie: No, there are real returns on stocks—usually in the forms of buybacks and acquisitions, nowadays, since dividends are tax-disadvantaged. If the stock market has the nature of a self-fulfilling prophecy, it’s only to the extent that high stock prices directly benefit companies, by letting the company get more capital or issue bonds at lower interest. If not for the direct effect that stock prices had on company welfare, it wouldn’t matter at all to a 10-year investor what other investors believe today. If stock prices had zero effect on company welfare, you’d be happy to buy the stock that nobody else believed in, and wait for that company to have real revenues and retained assets that everyone else could see 10 years later.

Simplicio: But nobody invests on a 10-year horizon! Even pension companies invest to manage the pension manager’s bonus this year!

Visitor: Surely the recursive argument is obvious? If most managers invest with 1-year lookahead, a smarter manager can make a profit in 1 year by investing with a 2-year lookahead, and can continue to extract value until there’s no predictable change from 2-year prices to 1-year prices.

Cecie: In the entrepreneurial world, startups are killed outright, very quickly, by the equivalent of low stock prices. And for legal reasons there are no hedge funds that can adjust market prices en masse, so the recursive argument doesn’t apply. The upshot is that seed investors have a strong incentive to care about what Series A investors think. If the entrepreneurs don’t fit the stereotype of cool entrepreneurs who have red hair, you can’t make an excess return by going against the popular misapprehension, because the startup will die in the next funding round.

The key phenomenon underlying the social molasses is that there’s a self-reinforcing equilibrium of beliefs. Maybe a lot of the Series A investors think the idea of entrepreneurs needing to have red hair is objectively silly. But they expect Series B investors to believe it. So the Series A investors don’t invest in blonde-haired entrepreneurs. So the seed investors are right to believe that “Series A investors won’t invest in blonde-haired companies” even if a lot of the reason why Series A investors aren’t investing is not that they believe the stereotype but that they believe that Series B investors believe the stereotype. And from the outside, of course, all that investors can see is that most investors aren’t investing in blonde-haired entrepreneurs—which just goes to reinforce everyone’s belief that everyone else believes that red-haired entrepreneurs do better.¹⁰

Visitor: And you can’t just have everyone say those exact words aloud, in unison, and simultaneously wake up from the dream?

Simplicio: I’m afraid people don’t understand recursion as well as that would require.

Cecie: Perhaps, Simplicio, it is only that most VCs believe that most other VCs don’t understand recursion; that would have much the same effect in practice.

Simplicio: Or maybe most people are too stupid to understand recursion. Is that something you’d be able to accept, if it were true?

Cecie: Regardless, on a larger scale, what we’re seeing is an extra stickiness that results when the incentive to try an innovation requires you to believe that other people will believe the innovation will work. An equilibrium like that can be much stickier than a scenario where, if you believe that a project will succeed, you have an incentive to try it even if other people expect the project to fail.

Stereotypically, the startup world is supposed to consist of heroes producing an excess return by pursuing ideas that nobody else believes in. In reality, the multi-stage nature of venture capital makes it very easy for the field to end up pinned to traditions about whether entrepreneurs ought to have red hair—not because everyone believes it, but because everyone believes that everyone believes it.

viii. First-past-the-post and wasted votes

Visitor: Does this feed back into our primary question of why your society can’t stop itself from feeding poisonous substances to babies?

Cecie: It’s true that venture capitalists are now collectively skeptical of attempts at new drug development, but the real problem (at least for cases like this) is the enormous cost of approval and the long delays the FDA causes.¹¹ The actual reason I went into this is that by understanding venture capitalists and entrepreneurs, we can understand the more complex case of voters and politicians. Which is the key to the political equilibrium that pins down the FDA, and all the other laws that prevent anyone from doing better. Not always, but quite often, the ultimate foundations of failure trace back to the molasses covering voters and politicians.

Simplicio: I’d like to offer, throughout whatever theory follows, the alternative hypothesis that voters are in fact just fools, sheep, and knaves. I mean, you should at least be considering that possibility.

Cecie: The simplest way of understanding the analogy between venture capitalists and voters is that voters have to vote for politicians that are electable.

Visitor: Uh, what? When you write down your preference ordering on elected representatives, you need to put politicians that other voters prefer at the top of your preference ordering?

Cecie: Yes, that’s pretty much what it amounts to. In the US, at least, elections are run on what’s known as a “first-past-the-post” voting system. Whoever gets the most votes in the contest wins. People who study voting systems widely agree that first-past-the-post is among the worst voting systems—it’s provably impossible for one voting system to have all the intuitively good properties at once, but FPTP is one of the most broken.

Visitor: Why not vote to change the voting system, then?

Cecie: I’ll get to that!

There are several ways of explaining what’s wrong with FPTP, but a lovely explanation I recently encountered phrases the explanation in terms of “wasted votes”—the total number of votes that can be removed without changing the outcome.

The two classic forms of gerrymandering are cracking and packing. Let’s say the parties are Green and Orange, and the Green party is in charge of drawing the voting boundaries. As a Green, you want to draw up districts such that Green politicians win with 55% of the vote—with some room for error, but not all that much—and for Orange politicians to win with 100% of the vote.

Simplicio: Ah, so that the Orange politicians won’t need to be responsive to Orange voters because their re-election is nearly guaranteed, right?

Cecie: No, the plot is far more diabolical than that. Consider a district of 100,000 people, where a Green politician wins with 55% of the vote. When 50,001 Green voters had cast their ballots, the election was already decided, under first-past-the-post, so the next 4,999 Green votes are “wasted”—this is to be understood as a technical term, not a moral judgment—in that they don’t further change the outcome. Then 45,000 Orange votes are also “wasted,” in that they don’t change the outcome. And also, one notes, those Orange voters don’t get the representative they wanted.

In an Orange district of 100,000 where the politician wins with 100% of the vote, there are 50,000 potent Orange votes and 50,000 wasted Orange votes. In total, there are 50,000 potent Green votes, 5,000 wasted Green votes, 50,000 potent Orange votes, and 95,000 wasted Orange votes. On a larger scale, this means that you can control a majority of a state legislature with slightly more than 1/4 of the votes—just have 55% of the districts containing 55% Green voters, with everything else solid Orange.

Visitor: And then this quarter of the population rules cruelly over the remaining three-quarters, who in turn lack the weapons to rise up?

Cecie: No, the real damage is far subtler. Let’s say that Alice, Bob, and Carol have taken time off from their cryptographic shenanigans to run for political office. Alice is in the lead, followed by Bob and then by Carol. Suppose Dennis prefers Carol to Bob, and Bob to Alice. But Dennis can’t actually write “Carol > Bob > Alice” on a slip of paper that gets processed by a trivially more sophisticated voting system. Dennis is only allowed to write down one candidate’s name, and that’s his vote. Under a system where the candidate with the most votes wins, and there’s uncertainty about which of the two frontrunners might win, all votes for whoever is in third place will be wasted votes, and this fact is predictable to the voters.

Visitor: Ah, I see. That’s why you introduced your peculiar multi-stage system of venture capital, which I assume must be held in place by laws forbidding anyone else to go off and organize their own financial system differently, and observed how it creates a sticky equilibrium in which financiers must believe that other financiers will believe in a startup.

If Dennis doesn’t believe that other “voters” will believe in Carol, Dennis will vote for Bob, which makes your politics stickier than a system in which “voters” were permitted to support the people they actually liked.

Cecie: Well, you see the analogy, but I’m not sure you appreciate the true depth of the horror.

Visitor: I’m sure I don’t.

Cecie: The upshot of first-past-the-post is typically a political system dominated by exactly two parties.

Visitor: Parties?

Simplicio: Entities that tell sheep who to vote for.

Cecie: In elections that have a single winner, votes for any candidate who isn’t one of the top two choices are wasted. In a representative democracy where districts vote on representatives who vote on laws, the dynamics of the district vote are then influenced by the dynamics of the national vote. Even if a third-party candidate could win a district, they wouldn’t have anyone to work with in the legislature, and so their votes would generally be wasted.

In the absence of a way to solve a large coordination problem, there’s no way for a third party to gain marginal influence over time. Each individual who considers voting for a third-party candidate knows they’ll be wasting their vote. This also means that third parties can’t field good candidates, since potential candidates know they’d be running to lose, which is stressful and unrewarding for people with better life options. And that’s a sufficient multi-factor system to prevent strong third parties from arising. When you’re not allowed to vote for Carol, who you actually like, you’ll vote for whichever of Alice and Bob you dislike the least.

The resulting equilibrium… well, Abramowitz and Webster found that what mainly predicted voting behavior wasn’t how much the voter liked their preferred party, but how much they disliked the opposing party.¹² Essentially, the US has two major voting factions, “people who hate Red politicians” and “people who hate Blue politicians.” When the Red politicians do something that Red-haters really dislike, that gives the Blue politicians more leeway to do additional things that Red-haters mildly dislike, which can give the Red politicians more leeway of their own, and so the whole thing slides sideways.

Simplicio: Looking at the abstract of that Abramowitz and Webster paper, isn’t one of their major findings that this type of hate-based polarization has increased a great deal over the last twenty years?

Cecie: Well, yes. I don’t claim to know exactly why that happened, but I suspect the Internet had something to do with it.

In the US, the current two parties froze into place in the early twentieth century—before then, there was sometimes turnover (or threatened turnover). I suspect that the spread of radio broadcasting had something to do with the freeze. If you imagine a country in the pre-telegraph days, then it might be possible for third-party candidates to take hold in one state, then in nearby states, and so a global change starts from a local nucleus. A national radio system makes politics less local.

The Internet might have pushed this phenomenon further and caused most of politics to be about the same national issues, which in turn reinforces the Red-vs.-Blue dynamic that allows each party to sustain itself on hatred for the other.

But that’s just me trying to eyeball the phenomenon using American history—I haven’t studied it. Other countries that also have the radio and Internet and similar electoral dynamics do manage to have more than two relevant parties, possibly because of dynamics that cause the votes of third-party politicians to be less wasted.

Simplicio: Isn’t the solution here obvious, though? All of these problems are caused by voters’ willingness to compromise on their principles and accept the lesser of two evils.

Cecie: Would things be better if people chose the greater of two evils? If they acted ineffectually against that greater evil? The Nash equilibrium isn’t an illusion. Individuals would do worse by playing away from that Nash equilibrium. Wasted votes are wasted. The current system is an effective trap and the voters are trapped. They can’t just wish their way out of that trap.

There doesn’t need to be any way for good to win; and if there isn’t, the lesser evil really is the best that voters can do. Pretending otherwise may feel righteous, but it doesn’t change the equilibrium.

Visitor: Just one second. Isn’t this all window dressing, compared to the issue of whatever true ruler imposes these rules on the “voters”? Like, if you put me into an elaborate cage that gives me an electric shock each time I vote for Carol, obviously the person who really controls the system is whoever put the cage in place and determines which politicians you can vote for without electric shocks.

Simplicio: I like the way you think.

Cecie: It’s not quite true to say that the system is self-reinforcing and that the voters are the sole instrument of their own destruction. But the lack of any obvious, individual tyrant who personally decides who you’re allowed to vote for has indeed caused many voters to believe that they are in control. I mean, they don’t feel like they’re in control, but they think that “the voters” select politicians.

They aren’t able to personalize a complicated bad equilibrium as a tyrant—not like they would blame a jeweled king who was standing in the polling booth, ready to give them an electric shock if they wrote down Carol’s name.

Inspired by Allan Ginsberg’s poem Moloch, Scott Alexander once wrote of coordination failures:

Moloch is introduced as the answer to a question—C. S. Lewis’ question in Hierarchy Of Philosophers—what does it? Earth could be fair, and all men glad and wise. Instead we have prisons, smokestacks, asylums. What sphinx of cement and aluminum breaks open their skulls and eats up their imagination?

And Ginsberg answers: Moloch does it.

There’s a passage in the Principia Discordia where Malaclypse complains to the Goddess about the evils of human society. “Everyone is hurting each other, the planet is rampant with injustices, whole societies plunder groups of their own people, mothers imprison sons, children perish while brothers war.”

The Goddess answers: “What is the matter with that, if it’s what you want to do?”

Malaclypse: “But nobody wants it! Everybody hates it!”

Goddess: “Oh. Well, then stop.”

The implicit question is—if everyone hates the current system, who perpetuates it? And Ginsberg answers: “Moloch.” It’s powerful not because it’s correct—nobody literally thinks an ancient Carthaginian demon causes everything—but because thinking of the system as an agent throws into relief the degree to which the system isn’t an agent.¹³

Scott Alexander saw the face of the Enemy, and he gave it a name—thinking that perhaps that would help.

Visitor: So if you did do this to yourselves, all by yourselves with no external empire to prevent you from doing anything differently by force of arms, then why can’t you just vote to change the voting rules? No, never mind “voting”—why can’t you all just get together and change everything, period?

Cecie: It’s true that concepts like these are nontrivial to understand.

It’s not obvious to me that people couldn’t possibly understand them, if somebody worked for a while on creating diagrams and videos.

But the bigger problem is that people wouldn’t know they could trust the diagrams and videos. I suspect some of the dynamics in entrepreneur-land are there because many venture capitalists run into entrepreneurs that are smarter than them, but who still have bad startups. A venture capitalist who believes clever-sounding arguments will soon be talked into wasting a lot of money. So venture capitalists learn to distrust clever-sounding arguments because they can’t distinguish lies from truth, when they’re up against entrepreneurs who are smarter than them.

Similarly, the average politician is smarter than the average voter, so by now most voters are just accustomed to a haze of plausible-sounding arguments. It’s not that you can’t possibly explain a Nash equilibrium. It’s that there are too many people advocating changes in the system for their own reasons, who could also draw diagrams that sounded equally convincing to someone who didn’t already understand Nash equilibria. Any talk of systemic change on this level would just be lost in a haze of equally plausible-sounding-to-the-average-voter blogs, talking about how quantitative easing will cause hyperinflation.

Visitor: Maybe it’s naive of me… but I can’t help but think… that surely there must be some breaking point in this system you describe, of voting for the less bad of two awful people, where the candidates just get worse and worse over time. At some point, shouldn’t this be trumped by the “voters” just getting completely fed up? A spontaneous equilibrium-breaking, where they just didn’t vote for either of the standard lizards no matter what?

Cecie: Perhaps so! But my own cynicism can’t help but suspect that this “trumping” phenomenon of which you speak would be even worse.

Simplicio: I have a technical objection to your ascribing all these sins to first-past-the-post voting rather than, say, the personal vices of the voters. There are numerous parliamentary democracies outside the United States that practice proportional representation, where a party getting 30% of the votes gets 30% of the seats in parliament. And they don’t seem to have solved these problems.

Cecie: Omegaven does happen to be approved in Europe, however. Like, they are not in fact killing those particular babies—

Simplicio: Oh, come on! Yes, the European equivalent of the US’s FDA happens to be a bit less stupid. Lots of other things in European countries happen to be more stupid. Indeed, I’d say that in Europe you have much crazier people getting seats in parliaments, compared to the United States. The problem isn’t the voting system. The problem is the voters.

Cecie: There are indeed some voters who want stupid things, and under the European system, their voice can be heard. There are also voters who want smart things and whose voices can be heard, like in the Pirate Party in Finland. But European parliamentary systems have different problems stemming from different systemic flaws.

Proportional representation would be a good system for a legislature that needed to repeatedly vote on laws, where different legislators could form different coalitions for each vote. If instead you demand that a majority coalition “form a government” to appoint an executive, then you need to give concessions to some factions, while other factions get frozen out. I’m not necessarily saying that it would be easy to fix all the problems simultaneously. Still, I imagine that a proportionally represented legislature, combined with an executive elected at-large by Condorcet voting, might possibly be less stupid—

Simplicio: Or maybe it would just give stupid voters a louder voice. I don’t like the evil conspiracy of the press and political elites that governs my country from the shadows, but I am willing to consider the proposition that the alternative is Donald Trump. I mean, I intend to go on fighting the Conspiracy about many specific issues. But if you’re proposing a reform that puts more power into the hands of sheep not yet awakened, the results could be even worse.

Cecie: Well, I agree that the design of well-functioning political systems is hard. Singapore might be the best-governed country in the world, and their history is approximately, “Lee Kuan Yew gained very strong individual power over a small country, and unlike the hundreds of times in the history of Earth when that went horribly wrong, Lee Kuan Yew happened to know some economics.” But the Visitor asked me why we were killing babies, and I tried to answer in terms of the system that obtained in the part of the world that was actually killing those babies. You asked why Europe wasn’t a paradise since it used proportional representation, and my answer is that parliamentary systems have their own design flaws that induce a different kind of dysfunction.

Simplicio: Then if both systems are bad, how does your hypothesis have any observable consequences?

Cecie: Because different systems are bad in different ways. When you have a “crazy” new idea, whether it’s good or bad, the European parliaments will be allowed to talk about it first. Whether that’s Omegaven, basic income, gay marriage, legalized prostitution, ending the war on drugs, land value taxes, or fascist nationalism, you are more likely to find it talked about in systems of proportional representation. It also happens to be true that those governments bloat up faster because of the repeated bribes required to hold the “governing coalition” together, but that’s a different problem.

ix. The Overton window

Simplicio: I’m beginning to experience the same sort of confusion as the Visitor about your view of the world, Conventional Cynical Economist. If voters weren’t stupid, the world would look very different than it does.

If the ultimate source of stupidity were poorly designed governmental structures, then average voters would sound smarter than average politicians. I don’t think that’s actually true.

Cecie: There are deeper forms of psychological molasses that generalize beyond first-past-the-post political candidates. The still greater force locking bad political systems into place is an equilibrium of silence about policies that aren’t “serious.”

A journalist thinks that a candidate who talks about ending the War on Drugs isn’t a “serious candidate.” And the newspaper won’t cover that candidate because the newspaper itself wants to look serious… or they think voters won’t be interested because everyone knows that candidate can’t win, or something? Maybe in a US-style system, only contrarians and other people who lack the social skill of getting along with the System are voting for Carol, so Carol is uncool the same way Velcro is uncool and so are all her policies and ideas? I’m not sure exactly what the journalists are thinking subjectively, since I’m not a journalist. But if an existing politician talks about a policy outside of what journalists think is appealing to voters, the journalists think the politician has committed a gaffe, and they write about this sports blunder by the politician, and the actual voters take their cues from that. So no politician talks about things that a journalist believes it would be a blunder for a politician to talk about. The space of what it isn’t a “blunder” for a politician to talk about is conventionally termed the “Overton window.”

Simplicio: It’s all well and good to talk about complicated clever things, Cynical Economist, but what explanatory power does all this added complexity have? Why postulate politicians who believe that journalists believe that voters won’t take something seriously? Why not just say that people are sheep?

Cecie: To name a recent example from the United States, it explains how, one year, gay marriage is this taboo topic, and then all of a sudden there’s a huge upswing in everyone being allowed to talk about it for the first time and shortly afterwards it’s a done deal. If you suppose that a huge number of people really did hate gay marriage deep down, or that all the politicians mouthing off about the sanctity of marriage were engaged in a dark conspiracy, then why the sudden change?

With my more complicated model, we can say, “An increasing number of people over time thought that gay marriage was pretty much okay. But while that group didn’t have a majority, journalists modeled a gay marriage endorsement as a ‘gaffe’ or ‘unelectable’, something they’d write about in the sports-coverage overtone of a blunder by the other team—”

Simplicio: Ah, so you say it was a conspiracy by evil journalists?

Cecie: No! Those journalists weren’t consciously deciding the equilibrium. The journalists were writing “serious” articles, i.e., articles about Alice and Bob rather than Carol. The equilibrium consisted of the journalists writing sports coverage of elections, where everything is viewed through the lens of a zero-sum competition for votes between Alice’s team and Bob’s team. Viewed through that lens, the journalists thought a gay marriage endorsement would be a blunder. And if you do something that enough journalists think is a political blunder, it is a political blunder. The journalists’ sports coverage will describe you as an incompetent politician, and primates instinctively want to ally with likely winners. Which meant the equilibrium could have a sharp tipover point, without most of the actual population changing their minds sharply about gay marriage in that particular year. The support level went over a threshold where somebody tested the waters and got away with it, and journalists began to suspect it wasn’t a political blunder to support gay marriage, which let more politicians speak and get away with it, and then the change of belief about what was inside the Overton window snowballed. I think that’s what we saw.

Simplicio: Forgive me for resorting to Occam’s Razor, but is it not simpler just to say that people’s beliefs changed slowly until it reached some level where the military-industrial complex realized they couldn’t win the battle to suppress gay marriage outright, and so stopped fighting?

Cecie: In a sense, that’s not far off from what happened, except without the evil conspiracy part. We might or might not be approaching a similar tipover point about ending the War on Drugs—a long, slow, secular shift in opinion, followed by a sudden tipover point where journalists model politicians as being allowed to talk about it, which means that politicians can talk about it, and then a few years later everyone is acting like they always thought that way. At least, I hope that’s where the current trend is leading.

Simplicio: Several states have already passed laws legalizing marijuana. Why hasn’t that already broken the Overton window?

Cecie: Because voter initiatives don’t break the common belief about what it would be a “gaffe” for a serious, national-level politician to do.

Eliezer: (aside) What broke the silence about artificial general intelligence (AGI) in 2014 wasn’t Stephen Hawking writing a careful, well-considered essay about how this was a real issue. The silence only broke when Elon Musk tweeted about Nick Bostrom’s Superintelligence, and then made an off-the-cuff remark about how AGI was “summoning the demon.”

Why did that heave a rock through the Overton window, when Stephen Hawking couldn’t? Because Stephen Hawking sounded like he was trying hard to appear sober and serious, which signals that this is a subject you have to be careful not to gaffe about. And then Elon Musk was like, “Whoa, look at that apocalypse over there!!” After which there was the equivalent of journalists trying to pile on, shouting, “A gaffe! A gaffe! A… gaffe?” and finding out that, in light of recent news stories about AI and in light of Elon Musk’s good reputation, people weren’t backing them up on that gaffe thing.

Similarly, to heave a rock through the Overton window on the War on Drugs, what you need is not state propositions (although those do help) or articles in The Economist. What you need is for some “serious” politician to say, “This is dumb,” and for the journalists to pile on shouting, “A gaffe! A gaffe… a gaffe?” But it’s a grave personal risk for a politician to test whether the public atmosphere has changed enough, and even if it worked, they’d capture very little of the human benefit for themselves.

Visitor: So… if this is the key meta-level problem… then why can’t your civilization just consider and solve this entire problem on the meta level?

Cecie: Oh, I’m afraid that this entire meta-problem isn’t the sort of thing the “leading candidates” Alice and Bob talk about, so the problem itself isn’t viewed as serious. That is, journalists won’t think it’s serious. Meta-problems in general—even problems as simple as first-past-the-post versus instant runoff for particular electoral districts—are issues outside the Overton window. So the leading candidates Alice and Bob won’t talk about organizational design reform, because it would be very damaging to their careers if they visibly focused their attention on issues that journalists don’t think of as “serious.”

Visitor: Then perhaps the deeper question is, “Why does anyone listen to these ‘journalists’?” You keep attributing power to them, but you haven’t yet explained why they have that power under your equilibrium.

Cecie: People believe that other people believe what’s in the newspapers.

Well, no, that’s too optimistic. A lot of people do believe what’s in the newspapers, so long as it isn’t about a topic regarding which they have any personal knowledge or expertise. The Gell-Mann Amnesia Effect is the term for how we read the paper about subjects we know about, and it’s talking about how wet streets cause rain; and then we turn to the story about international affairs or dieting, and for some reason assume it’s more accurate.

There’s some level on which most people prefer to talk and believe within the same mental world as other people. Nowadays a lot of people believe what they read on, say, Tumblr, and hardly look at The New York Times at all. But even then they still believe that other people believe what’s in The New York Times. That’s what gives The New York Times its special power over the collective consciousness, far out of proportion to their dwindling readership or the vanishing real trust that individuals from various walks of life have in them—what’s printed in The New York Times determines what people believe other people believe.

Simplicio: Do you truly lay all the sins of humanity at the feet of all this weird recursion? Or is this just a sufficiently weird hypothesis that you find it more fun to think about than the alternatives?

Cecie: I’m not sure I’m pointing in exactly the right direction, but I feel that I’m pointing in the general direction of something that’s truly important to the Visitor’s most basic question. The Visitor keeps asking why, in some sense, on some sufficiently general level, we can’t just snap out of it. And to put it in the sort of terms you yourself might want to use, Simplicio, if we’re looking for an explanation of why we can’t just snap out of it, then it might make sense to point to a bad Nash equilibrium covering our collective consciousness and discussion. I suspect that the recursion, the dependency on what people believe other people believe, has a lot to do with making that a sticky equilibrium a la venture capital.

Eliezer: (aside) Returning to my day job: As of 2017, I pretty commonly hear from AI researchers who are worried about AGI safety, but who say that they don’t dare say anything like that aloud. You could see this as either a good sign or a very bad sign, depending on how pessimistic or optimistic you previously were about the adequacy of academic discussion.

Simplicio: But then what, on your view, is the better way?

Cecie: Again, I could pontificate about various ideas, but that’s a different and harder question than looking at the actual equilibrium that currently obtains and forces doctors to poison babies. There doesn’t have to be a better way.

x. Lower-hanging altruistic fruit and bigger problems

(The Visitor takes a deep breath. When the Visitor speaks again, it is louder.)

Visitor: Then what about your <untranslatable 17>?

Cecie: Sorry? That word didn’t come through.

Visitor: What about everyone on your entire planet who could possibly care about babies dying?

So your medical specialists are borked. From the magic-tower analogy, I assume your systems of learning are borked, and that means most of the parents whose responsibility it is to protect the child are borked. Your politicians are borked. Your voters are borked. Your planet has no Serious People who could be trusted to try alternative shoe designs, let alone lead the way on any more complex coordination problem. Your prediction markets, I suppose, are somehow borked in a way that prevents anyone from making a profit by correcting inaccurate policy forecasts… maybe they forecast wrongly bad consequences to unpopular policies, which therefore never get implemented in a way that shows up the inaccurate prediction, since you don’t have any way to test things on a smaller scale? Your economists must somehow be borked—

Cecie: It’s more that nobody ever listens to us. They pay us and then they don’t listen to us.

Visitor: —and your financial system is borked so that nobody can make a profit on saving those babies or doing anything else useful. I’m not stupid. I’ve picked up on the pattern at this point.

But what about everyone else? There are seven billion people on your planet. How is it that none of them step up to save these babies from death and brain damage? How is your entire planet failing to solve this problem?

Cecie: That… sounds like a weird question, to an Earth person.

Visitor: Whatever your problems are, surely out of seven billion human beings there have to be some who could see the problems as you’ve laid them out, who could try to rally others to the cause of saving those babies, who could do whatever it took to save them!

Even if your system declares that saving babies is only the responsibility of “doctors” or “politicians” or whoever is the Someone Else whose Problem it is, there’s no law of physics that stops someone else from walking up to the problem and accepting responsibility for it. Out of seven billion people in your world, I can’t believe that literally all of them are incapable of gathering together some friends and starting things down the path to getting a little fish oil into a baby’s nutritional mixture!

Eliezer: I think I’ll step in myself at this point. There’s one other very general conclusion we can draw from seeing this ever-growing heap of dead babies. We might say, “the inadequacy of the part implies the inadequacy of the whole”—as we’ve defined our terms, if a part of the system is inadequate in X lives saved for Y dollars, then the whole system is inadequate in X lives saved for Y dollars. Someone who is motivated and maximizing will first go after the biggest inadequacy anywhere that they think they can solve, and if they succeed, it pushes forward the adequacy frontier for the whole system. Thus, we can draw one other general conclusion from the observation that babies are still being fed soybean oil. We can conclude that everyone on the planet who is smart enough to understand this problem, and who cares about strangers’ lives, and who maximizes over their opportunities, must have something more important to do than getting started on solving it.

Visitor: (aghast) More important than saving hundreds of babies per year from dying or suffering permanent brain damage?

Eliezer: The observation stands: there must be, in fact, literally nobody on Earth who can read Wikipedia entries and understand that omega-6 and omega-3 fats are different micronutrients, who also cares and maximizes and can head up new projects, who thinks that saving a few hundred babies per year from death and permanent brain damage is the most important thing they could do with their lives.

Visitor: So you’re implying…

Eliezer: Well, mostly I’m implying that maximizing altruism is incredibly rare, especially when you also require sufficiently precise reasoning that you aren’t limited to cases where the large-scale, convincing study has already been done; and then we’re demanding the executive ability to start a new project on top of that. But yes, I’m also saying that here on Earth we have much more horrible problems to worry about.

Cecie: We’ve just been walking through a handful of lay economic concepts here, the kind whose structure I can explain in a few thousand words. If you truly perceived the world through the eyes of a conventional cynical economist, then the horrors, the abominations, the low-hanging fruits you saw unpicked would annihilate your very soul.

Visitor: …

Eliezer: And then some of us have much, much more horrible problems to worry about. Problems that take more than reading Wikipedia entries to understand, so that the pool of potential solvers is even smaller. But even just considering this particular heap of dead babies, we know from observation that this part must be true: If you imagine everyone on Earth who fits the qualifications for the dead-baby problem—enough scientific literacy to understand relevant facts about metabolic pathways, and the caring, and the maximization, and enough scrappiness to be the first one who gets started on it, meeting in a conference room to divide up Earth’s most important problems, with the first subgroup taking on the most neglected problems demanding the most specialized background knowledge, and the second taking on the second-most-incomprehensible set of problems, until the crowdedness of the previously most urgent problem decreases the marginal impact of further contributions to the point where the next-worst problem at that level of background knowledge and insight becomes attractive… and so on down the ladders of urgency inside the levels of discernment… then there must be such a long and terrible list of tasks left undone, and so few people to understand and care, that saving a few hundred babies per year from dying or suffering permanent brain damage didn’t make the list. So it has been observed, and so it must be.

Wandering Bystander: (interjecting) But I just can’t believe our planet would be that dysfunctional. Therefore, by backward chaining, I question the original observation on which you founded your inference. In particular, I’m starting to wonder whether omega-3 and omega-6 could really be such significantly different micronutrients. Maybe that’s just a crackpot diet theory that somehow made it into Wikipedia, and actually all fats are pretty much the same, so there’s nothing especially terrifying about the prospect of feeding babies exclusively fat from soybean oil instead of something more closely resembling the lipid profile of breast milk?

Eliezer: Ah, yes. I’m glad you spoke up. I’ll get to your modest proposal next.

1. Carl Shulman notes that the Affordable Care Act linked federal payments to hospitals with reducing central-line infections (source), which was probably a factor in the change. ↩
2. Around a thousand infants are born with short bowel syndrome per year in the United States, of whom two-thirds develop parenteral nutrition-associated liver disease (source). See Park, Nespor, and Kerner Jr for a 2011 review of the academic literature, and Koch, Cohen, and Carroll and Madrzyk for news coverage. ↩
3. See Tabarrok’s “Assessing the FDA via the Anomaly of Off-Label Drug Prescribing,” which cites the widespread practice of off-label prescription as evidence that the FDA’s efficacy trial requirements are unnecessary. ↩
4. See the “Report Likelihoods, Not p-Values” FAQ, or, in dialogue form: “Likelihood Functions, p-Values, and the Replication Crisis.” ↩
5. From Schmidt and Hunter’s “Select on Intelligence”: “Intelligence is the major determinant of job performance, and therefore hiring people based on intelligence leads to marked improvements in job performance.” See also psychologist Stuart Ritchie’s discussion of IQ in Vox.Software engineer Alyssa Vance adds:
   

   I’ll note that, as far as I can tell, the informal consensus at least among the best-informed people in software is that hiring has tons of obvious irrationality even when there’s definitely no external cause; see [1] and [2]. In terms of Moloch’s toolbox, the obvious reason for that is that interviewers are rarely judged on the quality of the people they accept, and when they are, certainly aren’t paid more or less based on it. (Never mind the people they reject. “Nobody ever got fired because of the later performance of someone they turned down.”) Their incentive, insofar as they have one, is to hire people who they’d most prefer to be on the same floor with all day long. ↩

6. Compare psychiatrist Scott Alexander’s account, in “Against Tulip Subsidies”:
   

   In America, aspiring doctors do four years of undergrad in whatever area they want (I did Philosophy), then four more years of medical school, for a total of eight years post-high school education. In Ireland, aspiring doctors go straight from high school to medical school and finish after five years. I’ve done medicine in both America and Ireland. The doctors in both countries are about equally good. When Irish doctors take the American standardized tests, they usually do pretty well. Ireland is one of the approximately 100% of First World countries that gets better health outcomes than the United States. There’s no evidence whatsoever that American doctors gain anything from those three extra years of undergrad. And why would they? Why is having a philosophy degree under my belt supposed to make me any better at medicine? […]

   I’ll make another confession. Ireland’s medical school is five years as opposed to America’s four because the Irish spend their first year teaching the basic sciences—biology, organic chemistry, physics, calculus. When I applied to medical school in Ireland, they offered me an accelerated four year program on the grounds that I had surely gotten all of those in my American undergraduate work. I hadn’t. I read some books about them over the summer and did just fine.

   Americans take eight years to become doctors. Irishmen can do it in four, and achieve the same result. Each year of higher education at a good school—let’s say an Ivy, doctors don’t study at Podunk Community College—costs about $50,000. So American medical students are paying an extra $200,000 for…what?

   Remember, a modest amount of the current health care crisis is caused by doctors’ crippling level of debt. Socially responsible doctors often consider less lucrative careers helping the needy, right up until the bill comes due from their education and they realize they have to make a lot of money right now. We took one look at that problem and said “You know, let’s make doctors pay an extra $200,000 for no reason.”

   For a more general discussion of the evidence that college is chiefly a costly signal of pre-existing ability, rather than a mechanism for building skills and improving productivity, see Bryan Caplan’s argument in “Is College Worth It?”, also summarized by Roger Barris. ↩

7. See, e.g., Scott Alexander’s “My IRB Nightmare.” ↩
8. From Hyman and Silver, “You Get What You Pay For”:
   

   By the 1950s, death rates ranged between 1 and 10 per 10,000 encounters. Anesthesia mortality stabilized at this rate for more than two decades. Mortality and morbidity rates fell again after a 1978 article reframed the issue of anesthesia safety as one of human factor analysis. In the mid-1980s, the American Society of Anesthesiologists (ASA) promulgated standards of optimal anesthesia practice that relied heavily on systems-based approaches for preventing errors. Because patients frequently sued anesthetists when bad outcomes occurred and because deviations from the ASA guidelines made the imposition of liability much more likely, anesthetists had substantial incentives to comply.

   [… W]e should consider why anesthesia mortality stabilized at a rate more than one hundred times higher than its current level for more than two decades. The problem was not lack of information. To the contrary, anesthesia safety was studied extensively during the period. A better hypothesis is that anesthetists grew accustomed to a mortality rate that was exemplary by health care standards, but that was still higher than it should have been. From a psychological perspective, this low frequency encouraged anesthetists to treat each bad outcome as a tragic but unforeseen and unpreventable event. Indeed, anesthetists likely viewed each individual bad outcome as the manifestation of an irreducible baseline rate of medical mishap.

   Hyman and Silver note other possible factors behind the large change, e.g., the fact that the person responsible for mishaps was often easy to identify since there tended to be only one anesthetist per procedure, and that “because surgical patients had no on-going relationships with their anesthetist, victims were particularly likely to sue.” ↩

9. See Jena, Prasad, Goldman, and Romley, “Mortality and Treatment Patterns Among Patients Hospitalized With Acute Cardiovascular Conditions During Dates of National Cardiology Meetings.” ↩
10. See Glenn Loury’s The Anatomy of Racial Inequality for an early discussion of this issue. Note that some venture capitalists I’ve spoken to endorse this as an account of VC dysfunction, while others have different hypotheses. ↩
11. Carl Shulman argues that the FDA’s clinical trial requirements probably aren’t the reason for recent decades’ slowdown in the development of cool new drugs, given that increased regulation seems to have coincided with but not substantially accelerated the declining efficiency of pharmaceutical research and development (source). Shulman suggests that Baumol’s cost disease and diminishing returns play a larger role in the R&D slowdown.The FDA’s clinical trial requirements are much more likely to play a central role in limiting access to non-patented substances, though it’s worth noting here that the FDA has gotten faster than it used to be (source). ↩
12. Abramowitz and Webster, “All Politics is National.” ↩
13. See Scott Alexander’s “Meditations on Moloch.” ↩