Thanks for this enlightening post, Annie. I, too, initially believed Danziger’s ‘hungry judges’ study. It’s troubling that Kahnemann et al. uncritically endorse this in ‘Noise’, ignoring significant criticisms. On page 17, they assert: ‘For example, judges have been found more likely to grant parole at the beginning of the day or after a food break than immediately before such a break. If judges are hungry, they are tougher.’
Science self-corrects but slowly. This is true not just for social sciences but also for hard sciences. The replication crisis continues to snowball with easier publication via preprints, for example. And with autosummarization and AI it's only become easier for popular science media to communicate questionable findings. But as you say beliefs are incredibly sticky and long after a theory has been corrected, it continues to linger in its older erroneous form. Are there any examples where collective intuition has been put on ice until the science has been corrected? Or is this simply the cost of doing science?
There is a trivial way to establish whether the effect is real: randomize the people applying for parole without the judges knowing that that's what's happening. It's easy to see what's in front of you; it takes a real pro to notice that something's just a little bit off--like that unrepresented prisoners are being mixed in with represented ones. It would honk off attorneys representing prisoners because they'd have longer waits, but it would tell the tale.
What the study did was essentially collect a series of anecdotes. Anecdote is not the singular of data. If you want to find out the truth, you have to find a way to consider all possible cases. Only then, and when you have enough data points so that the probability that you have specious results becomes minuscule, do you have facts.
Before I retired, I practiced neurointerventional surgery [don't bother trying to pronounce it]. Sometimes cases were quite long. One of my trainees once simply accepted that angiographers are chronically hypoglycemic. My brain worked just fine with little fuel since logic doesn't require much besides oxygen. Didn't mean I wasn't hungry, tho. And patients did as well before lunch as after it. And in the middle of the night.
I am not sure what you mean by it being a series of anecdotes. Both were very large sample sizes with rigorous data analysis. I think with the first study, the issue was there was a strong confounding factor: unrepresented defendants came in front of the judges before meal breaks. There are replicable studies that show an after lunch dip, so this study conflicted with that previous data. The new work shows that in the case of judges who are fasting, they get more lenient the hungrier they are. This conflicts with the previous explanation. Obviously, more work needs to be done, but the data itself is not anecdotal.
It’s straightforward. The “study” was an attempt at a meta-analysis. But it was a meta-analysis of non-scientific information. I outlined what would be necessary to do a scientific study, but I’ll review it.
First, remove confounding factors, like time of day individuals are seen in court, whether they have legal representation or appear pro se—those have to be randomized in time. A non-represented person has to be seen at any given time with the same probability as a represented one. Then you also have to make sure the judge does not know whether the person has a lawyer representing him—a not simple thing to do. And then you have to do the same in each individual venue.
Doing it that way will yield data that you can partially evaluate. I can imagine 2 different outcomes. One is that regardless of representation and venue, everyone gets a lower likelihood of parole the later it gets. That removes the consideration of diet across the board. Second is that some venues get less and some get more lenient as the judges get hungrier.
But still, that just says that the later it gets, the less—or more—lenient judges get. Maybe they’re hungry, maybe they’re just tired of sitting and want to get up. Maybe they don’t even have lunch; maybe they just need a rest and food is unrelated.
You could control for that variable by having snacks that the judges could nosh on any time they want. Or that every hour they have a mandatory 10 minute break to stretch their legs. Were any of these explanations considered? I don’t recall seeing them mentioned.
My point is that until you control for all possible variables, all you have is supposition, which is no more valid than a bunch of anecdotes. I review papers for a medical journal and I see things like this all the time. It’s critical to see how a study is designed, even whether something is an actual study at all. The ones in the blog aren’t scientific studies. They lack randomization and they lack controls for cultural norms, for prisoner representation, for judge fatigue vs hunger, etc. they’re little more than stories, i.e. anecdotes.
Might be a function of 1) training--if you're used to working at a disadvantage, it stops being a disadvantage; or 2) selection--some people can't work on an empty stomach.
Can't tell without doing a controlled experiment. Which is very much ~not~ what they did. It's a credible thought, though, in that deciding who gets paroled isn't exactly a hard science. And what's not a hard science isn't really a science.
Thanks for this enlightening post, Annie. I, too, initially believed Danziger’s ‘hungry judges’ study. It’s troubling that Kahnemann et al. uncritically endorse this in ‘Noise’, ignoring significant criticisms. On page 17, they assert: ‘For example, judges have been found more likely to grant parole at the beginning of the day or after a food break than immediately before such a break. If judges are hungry, they are tougher.’
Wonderfully written, Annie.
Science self-corrects but slowly. This is true not just for social sciences but also for hard sciences. The replication crisis continues to snowball with easier publication via preprints, for example. And with autosummarization and AI it's only become easier for popular science media to communicate questionable findings. But as you say beliefs are incredibly sticky and long after a theory has been corrected, it continues to linger in its older erroneous form. Are there any examples where collective intuition has been put on ice until the science has been corrected? Or is this simply the cost of doing science?
There is a trivial way to establish whether the effect is real: randomize the people applying for parole without the judges knowing that that's what's happening. It's easy to see what's in front of you; it takes a real pro to notice that something's just a little bit off--like that unrepresented prisoners are being mixed in with represented ones. It would honk off attorneys representing prisoners because they'd have longer waits, but it would tell the tale.
What the study did was essentially collect a series of anecdotes. Anecdote is not the singular of data. If you want to find out the truth, you have to find a way to consider all possible cases. Only then, and when you have enough data points so that the probability that you have specious results becomes minuscule, do you have facts.
Before I retired, I practiced neurointerventional surgery [don't bother trying to pronounce it]. Sometimes cases were quite long. One of my trainees once simply accepted that angiographers are chronically hypoglycemic. My brain worked just fine with little fuel since logic doesn't require much besides oxygen. Didn't mean I wasn't hungry, tho. And patients did as well before lunch as after it. And in the middle of the night.
I am not sure what you mean by it being a series of anecdotes. Both were very large sample sizes with rigorous data analysis. I think with the first study, the issue was there was a strong confounding factor: unrepresented defendants came in front of the judges before meal breaks. There are replicable studies that show an after lunch dip, so this study conflicted with that previous data. The new work shows that in the case of judges who are fasting, they get more lenient the hungrier they are. This conflicts with the previous explanation. Obviously, more work needs to be done, but the data itself is not anecdotal.
It’s straightforward. The “study” was an attempt at a meta-analysis. But it was a meta-analysis of non-scientific information. I outlined what would be necessary to do a scientific study, but I’ll review it.
First, remove confounding factors, like time of day individuals are seen in court, whether they have legal representation or appear pro se—those have to be randomized in time. A non-represented person has to be seen at any given time with the same probability as a represented one. Then you also have to make sure the judge does not know whether the person has a lawyer representing him—a not simple thing to do. And then you have to do the same in each individual venue.
Doing it that way will yield data that you can partially evaluate. I can imagine 2 different outcomes. One is that regardless of representation and venue, everyone gets a lower likelihood of parole the later it gets. That removes the consideration of diet across the board. Second is that some venues get less and some get more lenient as the judges get hungrier.
But still, that just says that the later it gets, the less—or more—lenient judges get. Maybe they’re hungry, maybe they’re just tired of sitting and want to get up. Maybe they don’t even have lunch; maybe they just need a rest and food is unrelated.
You could control for that variable by having snacks that the judges could nosh on any time they want. Or that every hour they have a mandatory 10 minute break to stretch their legs. Were any of these explanations considered? I don’t recall seeing them mentioned.
My point is that until you control for all possible variables, all you have is supposition, which is no more valid than a bunch of anecdotes. I review papers for a medical journal and I see things like this all the time. It’s critical to see how a study is designed, even whether something is an actual study at all. The ones in the blog aren’t scientific studies. They lack randomization and they lack controls for cultural norms, for prisoner representation, for judge fatigue vs hunger, etc. they’re little more than stories, i.e. anecdotes.
Good point, Joe. Narrowly framing questions means we run the risk of leaving out salient factors in exchange for a simplistic narrative.
I wonder with intermittent fasting picking up, what current research has to say about the correlation between an empty stomach and a dodgy mind.
Might be a function of 1) training--if you're used to working at a disadvantage, it stops being a disadvantage; or 2) selection--some people can't work on an empty stomach.
Can't tell without doing a controlled experiment. Which is very much ~not~ what they did. It's a credible thought, though, in that deciding who gets paroled isn't exactly a hard science. And what's not a hard science isn't really a science.