Source: Peter Mende-Siedlecki & Jay Van Bavel—New York University
When we are considering a tough choice between two or more attractive options, we often end up actively weighing the pros and cons of each alternative. By reflecting on their advantages and disadvantages, we attempt to fit a complex, subjective decision into an orderly set of criteria. However, research in psychology suggests that this sort of introspective approach might not always yield the most optimal outcomes.1
In other words, sometimes thinking hard about a problem or a choice may not produce desired results. Similar results have been demonstrated in the domains of emotion (participants who ruminated about a bad mood showed less mood improvement than participants who were merely distracted from their mood;2 and memory (verbalizing the details of a criminal’s face led to poorer recognition in a photo array of possible suspects.3 Furthermore, Wilson and colleagues observed that reflecting on the reasons behind one’s attitudes (i.e., considering “why” one feels a certain way) can disrupt the consistency between attitudes and behavior, and can even change attitudes.4
Why might this be the case? Wilson and colleagues speculate that often we don’t typically have a very good understanding of why we actually feel the way we do.5 Upon introspection of our feelings, we may hone in on irrelevant but salient details that may offer plausible explanations, but may also have little direct influence on our actual attitudes. Wilson and Schooler devised an experiment designed to test this possibility in the domain of subjective preferences. Specifically, they compared participants’ evaluations of a series of jams with experts’ evaluations, and tested whether asking participants to analyze the reasons for their choices would have a negative impact on their evaluations.
This investigation capitalizes on what might be termed the unreliability of introspection, or the introspection illusion. Evidence in social psychology suggests that we have very little direct access into the mental processes giving rise to our perceptions and behavior.5 When we attempt to introspect on these processes, we often miss the mark—providing plausible but inaccurate post-hoc explanations based on implicit causal theories of what is most likely to have affected our mental states.
1. Participant Recruitment
- Conduct a power analysis and recruit a sufficient number of participants and obtain informed consent from the participants.
- Gather two experimenters: One to administer the jam taste test, and another (blind to condition) to obtain participants’ ratings of each jam.
- Recruit volunteers for a study entitled "Jam Taste Test". Instruct volunteer subjects not to eat anything for 3 hrs prior to the study.
2. Data Collection
- Purchase five brands of strawberry jams or preserves that vary in their overall quality.
- To provide participants with a large range of jam quality, the original researchers purchased the 1st-, 11th-, 24th-, 32nd-, and 44th-ranking jams in a report produced by Consumer Reports magazine of 45 different jam brands (“Strawberry Jams," 1985). This report contained expert rankings based on the ratings of seven consultants who were trained sensory panelists. Each expert consultant rated 45 jams across 16 specific sensory characteristics, such as sweetness, aroma, etc. Averages were calculated across all seven expert consultants’ ratings to yield rankings of all 45 jams.
- Prior to running each participant, set up the testing room. You’ll need a table, a chair, plates, spoons, jams, labels, and a trash can.
- Arrange the five jams in a random on the table, labeled A through E.
- Keep track of which jam receives which label for each participant, and in which order the jams are arranged.
- Take one teaspoon of Jam A and place that spoonful on a plate in front of the Jam A jar.
- Repeat for Jams B through E.
- Run participants individually.
- Have Experimenter 1 inform each participant that the purpose of the study is to evaluate different kinds of jams under different conditions, as part of a consumer psychology experiment.
- Explain that some participants will taste jams on crackers, while others will taste jams on plastic spoons.
- Critically, inform the participant that they have been randomly assigned to the condition in which they’ll be tasting jams on spoons. Specifically, after tasting the jams, they will rate their liking for each one.
- Have each participant sign a consent form, giving their informed consent to participate.
- Randomly assign participants to either the “reasons-analysis” condition, or a control condition.
- Provide “reasons-analysis” participants with written instructions asking them to "analyze why you feel the way you do about each" jam," in order to prepare yourself for your evaluations."
- Inform these participants that they will be asked to list their reasons for liking or disliking each jam after tasting. Describe this task as a means of organizing participants’ thoughts prior to rating, and further, inform participants that they will not be asked to hand in their list of reasons.
- Do not give the rest of participants (e.g., control participants) any additional instructions.
- Ask participants to be seated at the table where the five jams have been arranged spoons with approximately 1 tsp of strawberry jam.
- For “reasons-analysis” participants, have Experimenter 1 ask the participants to taste each of the five jams.
- During the tasting, leave the room.
- Upon returning, administer the reasons questionnaire. This questionnaire should ask participants to list their reasons for liking or disliking each of the Jams A through E.
- Reiterate that the purpose of the questionnaire is to organize the subjects' thoughts and that they would not be asked to hand it in.
- Leave the room once again.
- Upon returning, explain that the completed questionnaire will no longer be needed, and should put the questionnaire in the trash can.
- For control participants, have Experimenter 1 also ask the participants to taste each of the five jams.
- During tasting, leave the room.
- Upon returning, administer a filler questionnaire instructing participants to list reasons why they chose their major.
- Leave the room once again.
- Upon returning, collect the completed filler questionnaire.
- Have Experimenter 1 now introduce the participant to Experimenter 2, who must be unaware of the condition that the participant was assigned to.
- Have Experimenter 2 administer a questionnaire to the participant asking them to evaluate each of the jams on a 9-point scale ranging from disliked (1) to liked (9).
- Instruct participants to complete the questionnaire and to place it through a slot in a covered box on the table, to maintain anonymity.
- Leave the room while participants are completing this questionnaire.
- Upon returning, fully debrief the participant regarding the purpose and procedures of the study.
3. Data Analysis
- After running all participants, test whether evaluations of the jams varied between participants in the “reasons-analysis” condition (e.g., those who analyzed how they felt about the jams) and those in the control condition (e.g., those who did not) via a 2 (condition: reasons analysis vs. control) x 5 (jams 1, 2, 3, 4, and 5) multivariate ANOVA.
- To compare the degree of agreement with expert ratings between participants in the “reasons-analysis” condition and those in the control condition, simply compute the Spearman’s rank-order correlation between each participant’s subjective ratings of each jam and the expert ratings of each jam, on a subject-by-subject basis.
- Convert these values to z-scores via Fisher’s r-to-z transformation for purposes of analyses.
- Perform an independent-samples t-test examining how the average participant/expert correlation varies as a function of group.
- Use two, one-sample t-tests comparing the average participant/expert correlation in the “reasons-analysis” condition and the control condition against zero separately.
Decision-making occurs on a continuum of thoughts with varying levels of consciousness. In most cases, people have no idea why they feel the way they do.
At one extreme, individuals may think very little—making decisions quickly and habitually—like pulling out already prepared foods for dinner.
In contrast, choosing where to dine out results in thinking too much—reading through reviews to weigh all of the pros and cons because their friend also asks for an explanation regarding the final selection.
Upon introspection—without knowing why—the individual honed in on irrelevant yet salient details that made sense, but in fact, did not lead to the best decision, at least by experts’ standards.
This example capitalizes on the unreliability of thinking consciously—also called the introspection illusion—where an attempt to understand one’s own perceptions and behaviors results in erroneous post-hoc explanations.
Based on the classic experiment by Wilson and Schooler, this video demonstrates whether asking participants to think about reasons for rating different jams interfered with their preferences compared to those who were simply asked to provide rankings.
In this experiment, two groups of participants—reasons-analysis and controls—are asked to taste test different jams that have been previously analyzed by experts, and then evaluate each one based on their preference.
During the tasting phase, those in the reasons-analysis group are asked to analyze the way they feel about each one by listing five reasons for liking or disliking it after tasting. This step allows participants to organize their thoughts and encourage better preparation for providing final ratings.
Each participant in the control group, on the other hand, is asked to complete a filler questionnaire and list reasons why they chose their college major. In either case, these first forms are discarded.
Subsequently, participants are given a second preference questionnaire by a different experimenter blinded to their condition. They are asked to anonymously evaluate each jam on a 9-point scale ranging from 1-disliked to 9-liked.
In this experiment, the dependent variable is the average preference rating of each jam as a function of group, reasons-analysis versus controls.
Participants who spend time deliberating about the reasons supporting their preferences are expected to show much less correspondence with an objective criterion of quality—the expert ratings—compared to controls. Such findings would support the illusion of introspection, where individuals change decisions because they aren’t truly aware of their attitudes.
Prior to the experiment, conduct a power analysis to recruit a sufficient number of individuals for a study entitled "Jam Taste Test". Call those enlisted and instruct them not to eat anything for 3 hrs prior to the study.
To continue preparation, acquire five brands of jam that vary in overall quality based on expert opinions, and randomly label them A through E. Now, set up the testing room.
Arrange the five brands on the table, and for each participant, record the order and what label each specific jam received.
Then, place a spoonful of each jar’s content directly in front.
To begin, escort a participant into the testing room, and have them sign the consent forms.
Explain that the purpose of the study is to evaluate different kinds of jams under different conditions and they have been randomly chosen to taste on spoons. Remind them to rate their liking preference for each one.
For participants randomly assigned to the reasons-analysis group, provide additional written instructions stating that they need to list their reasons behind liking or disliking each one to organize their thoughts prior to the final evaluations. Note, for control participants, do not provide any further instructions.
Have each participant sit at the table where the five jams were arranged, ask them to start their evaluation, and leave the room.
Following the tasting period, administer a questionnaire and remind them that the purpose of it is to simply organize their thoughts and that they do not need to hand it in. For the control participants, administer a filler questionnaire. Exit the room while the forms are being filled out.
Upon return, explain that this information will no longer be needed and throw it in the trash can. Now, introduce a second experimenter who is unaware of the participants’ assigned conditions.
To capture participants’ ratings, have this new researcher administer preference forms, in which they should place into a covered box on the table when they finish. Leave the room while they record their evaluations.
To end the experiment, fully debrief all participants and discuss the true purpose and procedures of the study.
To visualize the data, first plot the mean ratings across the five jams, in order of the experts’ ratings, and split by condition—controls versus reasons. Notice that control participants were more accurate compared to those in the reasons group, as their scores patterned the professional rankings.
To quantify the amount of agreement, compute correlation values between each participant’s and the expert’s ratings and plot the averages by condition. The average correlation in the control group was near 60%, whereas the mean value in the reasons condition was less than 20%, not significantly different from zero.
Thus, participants who spent time deliberating about reasons supporting their own evaluations showed less correspondence to an objective criterion, which supports the introspection illusion: Thinking too much does impair decision-making!
Now that you are familiar with how introspective thoughts affect decision making in an experimental setting, let’s look at other cases where deliberating between choices leads to suboptimal results.
When it comes to large purchases, like buying a car, many people take their time to weigh the options and consider the pros and cons at every angle, before committing to such an investment.
Although there may be economic advantages, psychologists have found that consumers who exhaustively deliberate these tough choices end up feeling less satisfied with their purchase—essentially showing buyer’s remorse.
Interestingly, older consumers tend to vacillate much less in the car buying process by sticking to brands that they have remained loyal to over the years.
In addition, relationships call for moments of intensive introspection, but all of this deep thought can cause people to make decisions that they also later regret. Researchers postulate that this happens because the introspective process causes them to lose touch with their feelings, leading them to act based on a false impression of their true emotions.
You’ve just watched JoVE’s video on how thinking too much impairs decision-making. Now you should have a good understanding of how to design, conduct, and analyze an experiment to study how our inner thoughts influence behavior, and how introspecting on one’s preferences can be applied to consumer behavior and interpersonal relationships.
Thanks for watching!
In the original Wilson and Schooler investigation, the authors observed that asking participants to think about their reasons for their evaluations did indeed change their ratings of the set of jams, as compared to the ratings given by control participants.
Critically, when comparing against the objective criterion (e.g., experts ratings), the average correlation between “reasons-analysis” participants’ ratings and experts’ ratings was significantly lower than the average correlation between control participants’ ratings and experts’ ratings. Moreover, while this average correlation between participants’ and experts’ ratings was significantly greater than zero in the control condition, in the “reasons-analysis condition,” it was not.
Figure 1: Mean liking ratings for the five jams as a function condition. Average liking ratings provided by participants in the control (left) and reasons conditions (right) are displayed for all five jams. Jam 1 was the top ranked jam, based on experts’ ratings. Jams 2, 3, 4, and 5 were the 11th-, 24th-, 32nd-, and 45th-ranked jams, respectively. Participants were more accurate in rating the jam in the control condition (left).
Figure 2: Average correlation between participants’ ratings and experts’ ratings as a function of condition. Bars represent the average correlation between participants’ ratings of jam liking and experts’ rankings, as a function of whether they were in the control (left) or reasons condition (right). Within-participant correlations were Fisher-transformed and then compared against zero and against each other. The average correlation in the control condition was different from zero, and significantly stronger than the average correlation in the reasons condition, which was not different from zero.
Applications and Summary
Based on these results, the authors concluded that while the control participants formed jam preferences that were very similar to an objective criterion of quality (e.g., experts’ ratings), participants who spent time deliberating about the reasons supporting their evaluations showed much less correspondence with this criterion. The authors suggest that these participants’ preferences were influenced by the process of introspection, which likely caused them to focus on salient, but ultimately irrelevant attributes of the jams.
The results of this study have clear implications in the marketing domain, and for consumers in general. Introspecting on one’s preferences—specifically on the reasons supporting those preferences—may lead to less-than-optimal decision-making. In other words, a consumer who exhaustively deliberates on the pros and cons of a choice may end up feeling less satisfied with the ultimate results of that choice.7
These principles may also be extrapolated to other complex decision domains (e.g., legal, economic, and even interpersonal decision-making), though testing the boundary conditions of these results is critical. One could examine how manipulating various aspects of the choice architecture (e.g., the number of alternatives the participant is considering, the objective range of alternatives, the average quality of alternatives, etc.) or the instructions (e.g., asking participants to focus on abstract versus concrete reasons, pitting a “reasons analysis” against a “feelings analysis”, etc.) might exaggerate or attenuate this pattern of results. One could also test the degree to which these results generalize to other decision domains or attitude objects, or consider individual differences in these effects. For example, one might ask for whom will introspection not have a deleterious effect? One possibility is that people who enjoy thinking will prefer to engage in introspective and it might even lead to better outcomes. Of course, this is a question for future research.
- Wilson, T. D., & Schooler, J. W. (1991). Thinking too much: introspection can reduce the quality of preferences and decisions. Journal of personality and social psychology, 60(2), 181.
- Morrow, J., & Nolen-Hoeksema, S. (1990). Effects of responses to depression on the remediation of depressive affect. Journal of personality and social psychology, 58(3), 519.
- Schooler, J. W., & Engstler-Schooler, T. Y. (1990). Verbal overshadowing of visual memories: Some things are better left unsaid. Cognitive psychology, 22, 36-71.
- Wilson, T. D., Dunn, D. S., Kraft, D., & Lisle, D. J. (1989). Introspection, attitude change, and attitude-behavior consistency: The disruptive effects of explaining why we feel the way we do. Advances in experimental social psychology, 22, 287-343.
- Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological review, 84, 231-259.
- Lisle, D., Schooler, J., Hodges, S. D., Klaaren, K. J., & LaFleur, S. J. (1993). ‘Introspecting about reasons can reduce post-choice satisfaction. Personality and Social Psychology Bulletin, 19, 331-39.
- Wilson, T. D., & Gilbert, D. T. (2003). Affective forecasting. Advances in Experimental Social Psychology, 35, 345-411.