Hasty Generalization Fallacy
Kid-friendly name: Bad Sample
A hasty generalization, or Bad Sample, is a conclusion about a whole group drawn from a sample that is too small or too unrepresentative to support it.
Logicians call this kind of reasoning an inductive argument. It starts with what you saw in some members of a group and draws a conclusion about the whole group: the ones I saw were like this, so they're all like this. An argument based on a sample can never be 100% airtight, because the next sample could always be different. But a good sample can make the conclusion very likely, and a bad sample can't.
For example, let's say an invisible alien researcher secretly visited a middle school classroom during the second hour of a standardized test. Its field report goes something like this:
Earth adolescents are silent, motionless creatures. They do not speak. They do not laugh. They communicate by writing on pieces of paper. Recommend no further study of the species; there is nothing here.
The alien accurately described its observations. The room was silent, the students were mostly motionless, and nobody seemed to be enjoying themselves. The problem is the leap from those observations to a conclusion about every adolescent on the planet.
The sample was too small: one room, for one hour, on one morning. It was also unrepresentative. The alien happened to arrive during the quietest hour of the school year. Had it landed in the same school at lunchtime on a Friday, the report would have read very differently, and would probably have included the word "deafening."
Our brains are extremely good at this mistake. There is a plausible case that the habit once served our ancestors well: a berry that made you sick once was worth avoiding forever. Psychologists call one version of this conditioned taste aversion, and it can put a person off a food for years after a single bad experience. That instinct is still running, and it still fires after just a few data points. It serves us less well in a world of polls, reviews and social media feeds.
To Introduce the Concept
One Rainy Afternoon in SeabrightCan one quick rain shower tell you what a town's weather is like?
Read more →
Can one quick rain shower tell you what a town's weather is like?
The Okonkwo family arrives in the seaside town of Seabright at 2:40 in the afternoon, and at 2:44 it begins to rain. It rains for eleven minutes. From the back seat, Femi concludes that Seabright is the rainiest place on the planet, it rains here constantly, the sun has plainly never been seen in this town, and the entire week is ruined.
Femi is judging the town's entire climate after about fifteen minutes in town, eleven of them rainy. That's far too small a sample to say that Seabright is "the rainiest place on the planet" or that it "rains constantly." To know what a town's weather is really like, you'd need years of records, in every season. Meteorologists don't trust one afternoon either.
In the Wild
United States, 1943Abraham Wald and the Planes That Came BackThe military studied where returning bombers had been hit. A mathematician pointed out which planes were missing from the study.
Read more →
The military studied where returning bombers had been hit. A mathematician pointed out which planes were missing from the study.
During World War II, American planes were being shot down at a terrible rate. Armor could help protect them, but armor is heavy, and a plane can only carry so much weight before it can't fly. That raised a practical question: if armor was the answer, where should it go?
The obvious place to look was the planes themselves. Aircraft came back from missions covered in bullet holes, and those holes could be counted and mapped. The tempting conclusion was to reinforce the areas with the most holes.
Abraham Wald saw the problem with that reasoning. Wald was a Hungarian-born Jewish mathematician who had fled Nazi-occupied Austria and joined the Statistical Research Group at Columbia University. Every plane in the analysis had made it home. A plane hit in a truly vital spot, such as the engine or the cockpit, was missing from the data, not because those spots were rarely hit, but because planes hit there didn't come back. Assuming enemy fire lands more or less evenly across an aircraft, the places where surviving planes show the fewest holes are exactly where a plane can least afford to be hit. Wald worked out methods for estimating how vulnerable each part of a plane was, using only the damage on the survivors. In short, the armor belonged where the holes weren't.
In other words, the sample was unrepresentative. It contained only the successes and none of the failures.
This error is called survivorship bias, and it is everywhere. When we only look at the winners (the planes that returned, the startups that succeeded, the buildings that are still standing), we can draw exactly the wrong conclusions about what works. If you only look at billionaires like Bill Gates and Mark Zuckerberg, you might conclude that dropping out of college is a great strategy for getting rich. You would only think that because the many dropouts who didn't get rich aren't writing books or giving TED Talks.
A final note on the story itself, since this is a lesson about evidence. Wald's wartime memos still exist, and his colleague W. Allen Wallis later wrote about the group's work. But those memos are technical reports full of equations, not a dramatic scene. The popular version, with stubborn generals pointing at the wings and a lone genius pointing at the engines, is a later embellishment. Even the famous picture of a plane covered in red dots traces back to a diagram that designer Cameron Moll made around 2005, with dots he admits were made up. It's a fine teaching picture, as long as we remember it's a modern invention. Wald's real contribution was careful math, delivered in memos.
Sources: Abraham Wald, A Method of Estimating Plane Vulnerability Based on Damage of Survivors (Statistical Research Group, 1943; reprinted by the Center for Naval Analyses, 1980); W. Allen Wallis, "The Statistical Research Group, 1942–1945," Journal of the American Statistical Association (1980); Marc Mangel and Francisco Samaniego, "Abraham Wald's Work on Aircraft Survivability," Journal of the American Statistical Association (1984); Cameron Moll on the origin of the red-dot diagram
What's something you decided you didn't like after trying it only once? How many tries would it take before your conclusion is a fair one?
This is 1 fallacy. Mind Merit covers 25.
Mind Merit's Bad Sample Family Guide
- •More examples: Fun, simple examples to illustrate the flaw.
- •More real-world cases: Real-world examples to dig into the flaw in more detail.
- •Family guidance: How to teach recognizing the fallacy to your family.
Inside Mind Merit
Learn to separate the main conclusion from the evidence backing it up.
Learn to spot the flaw in the reasoning, even when the conclusion seems convincing.
- •Adaptive practice: Questions adjust automatically as reasoning improves, keeping practice challenging without being overwhelming.
- •Parent dashboard: Track which logic skills are clicking, and where your student needs more practice.