Name: testing measurement quality in research
Uploaded: 2020-02-12T11:07:40.000Z
Duration: 1 min 29 s
Channel: JoVE

K12 Standards

Virginia  Standards of Learning

HIgh School

Earth Science

ES.1

ES.1.d

Reliability and Validity

In a scientific setting, suppose a researcher wants to create a test to measure the compatibility of potential partners on an online dating website. They must consider two important factors to generate successful outcomes.

One is reliability, which refers to the ability of a test&mdash;or other research instrument&mdash;to provide consistent, and thus, reproducible, results under similar circumstances.

In this context, the compatibility test would be considered reliable if the same people take the test twice and perform similarly each time. This situation is also known as test-retest reliability.

However, getting consistent results does not ensure that a test is accurate. Now, the second factor, validity&mdash;the extent to which a test accurately measures or predicts what it set out to measure&mdash;must be reflected.

Will they actually enjoy spending time together on a date?

If the pair scored low in dating compatibility and still went out to dinner together, perhaps they had a miserable time. In this case, the test did have high predictive validity&mdash;it forecasted the behavior.

In the end, researchers strive for reproducibility and accuracy: here, the test is successful if other incompatible couples cont

reliability and validity

Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.

Unfortunately, being consistent in measurement does not necessarily mean that you have measured something correctly. To illustrate this concept, consider a kitchen scale that would be used to measure the weight of cereal that you eat in the morning. If the scale is not properly calibrated, it may consistently under- or overestimate the amount of cereal that&rsquo;s being measured. While the scale is highly reliable in producing consistent results (e.g., the same amount of cereal poured onto the scale produces the same reading each time), those results are incorrect. This is where validity comes into play. Validity refers to the extent to which a given instrument or tool accurately measures what it&rsquo;s supposed to measure. While any valid measure is by necessity reliable, the reverse is not necessarily true. Researchers strive to use instruments that are both highly reliable and valid.

How Valid Is the SAT? 

Standardized tests like the SAT are supposed to measure an individual&rsquo;s aptitude for a college education, but how reliable and valid are such tests? Research conducted by the College Board suggests that scores on the SAT have high predictive validity for first-year college students&rsquo; GPA (Kobrin, Patterson, Shaw, Mattern, &amp; Barbuti, 2008). In this context, predictive validity refers to the test&rsquo;s ability to effectively predict the GPA of college freshmen. Given that many institutions of higher education require the SAT for admission, this high degree of predictive validity might be comforting.

However, the emphasis placed on SAT scores in college admissions has generated some controversy on a number of fronts. For one, some researchers assert that the SAT is a biased test that places minority students at a disadvantage and unfairly reduces the likelihood of being admitted into a college (Santelices &amp; Wilson, 2010). Additionally, some research has suggested that the predictive validity of the SAT is grossly exaggerated in how well it is able to predict the GPA of first-year college students. In fact, it has been suggested that the SAT&rsquo;s predictive validity may be overestimated by as much as 150% (Rothstein, 2004). Many institutions of higher education are beginning to consider de-emphasizing the significance of SAT scores in making admission decisions (Rimer, 2008).

In 2014, College Board president David Coleman expressed his awareness of these problems, recognizing that college success is more accurately predicted by high school grades than by SAT scores. To address these concerns, he has called for significant changes to the SAT exam (Lewin, 2014).

&nbsp;

This text is adapted from <a href="https://cnx.org/contents/Sr8Ev5Og@10.24:mfArybye@16/2-3-Analyzing-Findings" target="_blank">OpenStax, Psychology. OpenStax CNX.</a>

Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to ...

Education

JoVE Core

Social Psychology

Foundations of Social Psychology

Research Methods

KEY TERMS AND CONCEPTS

testing measurement quality in research

Reliable and valid measurements are essential in data collection and research. Reliability means a tool or test gives the same result in a consistent way. In psychology, that means the instruments used to collect data should produce reproducible results.

Consistency alone is not enough. A kitchen scale can show the same reading every time, but it can still be wrong if it is not calibrated correctly. In that case, the scale is reliable because it is consistent, but it does not measure the cereal’s weight accurately.

Validity is the next step. Validity means a tool measures what it is supposed to measure. A valid measure must also be reliable, but a reliable measure is not always valid. Researchers aim to use tools that are both consistent and accurate.

Standardized tests like the SAT show how this issue matters in real life. The SAT is meant to measure aptitude for college, and research from the College Board suggests that it has high predictive validity for first-year college GPA. Predictive validity means a test can help predict a future outcome, such as a freshman’s grades.

Even so, the SAT has been debated. Some researchers argue that it is biased against minority students and may lower their chances of college admission. Other research suggests that its predictive validity has been overstated, possibly by as much as 150%.

Because of these concerns, some colleges are placing less weight on SAT scores. In 2014, College Board president David Coleman also noted that high school grades predict college success more accurately than SAT scores. He called for major changes to the exam.

Testing Measurement Quality in Research

the scientific method

Research Questions and Falsifiable Hypotheses

Research questions and falsifiable hypotheses help scientists separate facts from opinions. Facts are observable realities. Opinions are personal judgments, conclusions, or attitudes that may or may not be accurate. In science, facts are established with evidence from empirical research, which means research based on observation and experience. Scientific knowledge grows through the scientific method, a process that connects ideas with real-world evidence. Researchers test theories and...

case studies

Studying Rare Cases in Psychology

Case studies help psychologists study rare people or rare situations in detail. This research method is used in observational research when scientists focus on one person or just a few individuals. Because the group is so small, researchers can collect very rich information and gain a deep understanding of the case. A well-known example is the New York Times story about Krista and Tatiana Hogan, Canadian twin girls who are conjoined at the head. Scientists have evidence that the twins may be...

naturalistic observations

Watching Behavior in Real Settings

Naturalistic observation is a research method for watching behavior in its real setting. Researchers try to record what people or animals do without changing the situation. The goal is to see behavior as naturally as possible. People often act differently when they know they are being watched. That is why observers must be inconspicuous, or hard to notice. In a classroom hand-washing example, a student observer might stand at a sink and pretend to put in contact lenses while secretly recording...

surveys

Survey Data in Psychology

Survey data in psychology comes from sets of questions answered by research participants. Surveys can be given on paper, online, or by speaking with people directly. They are usually quick to complete, which makes them useful when researchers want information from many people. A survey helps researchers study a sample, which is a smaller group chosen from a population, the larger group they want to understand. Because surveys can reach larger and more diverse samples, the results may reflect...

archival research

Using Records to Study Behavior

Archival research uses existing records and data sets to study behavior and answer research questions. Researchers do not collect new data from participants in this method. Instead, they look back at records that already exist to find patterns and relationships. A researcher might, for example, examine the academic records of students who enrolled in college over the past ten years. Those records could show how long students took to finish their degrees, along with course loads, grades, and...

longitudinal research

Tracking the Same People Over Time

Longitudinal research tracks the same group of people over time. Researchers use this design when they want to study change in human development or across the lifespan. Data are collected from the same participants repeatedly over an extended period. A simple example is a survey on dietary habits. The same people might answer at age 20, again at age 30, and again at age 40. This repeated testing helps researchers see how responses change over the years. Longitudinal research can also help...

cross-sectional research

Studying Age Groups at One Point in Time

Cross-sectional research compares different groups of people at the same time. Researchers use it when they want to study how a trait or behavior varies across age groups or other segments of the population. For example, a researcher interested in dietary habits could compare 20-year-old, 30-year-old, and 40-year-old individuals in one study. This approach does not require waiting 20 years to track how one group changes from decade to decade. It is a shorter-term way to collect data. <br...

group design

Experimental and Control Groups

Experimental and control groups are the two parts of a basic experiment. The groups should be the same except for one difference: the experimental manipulation. The experimental group gets the treatment or variable being tested, and the control group does not. Because that is the only planned difference, researchers can link any change between the groups to the manipulation rather than to chance. A clear example is a study on violent television and children’s behavior. In that setup, the...

factorial design

Factorial Analysis in ANOVA

Factorial analysis is a research design that uses ANOVA, or analysis of variance, to test how more than one independent variable affects a dependent variable. The independent variables are also called factors. A worker productivity study can show how salary and skill level may each influence the result. In this example, salary can be grouped into three levels: low, moderate, and high. Skill level can be grouped into two levels: entry level and experienced. The study would be called a two-way...

the placebo effect

How Expectations Shape Placebo Results

The placebo effect shows how expectations and beliefs can shape what people experience. In some situations, simply expecting something to happen can make it feel real. This effect is often discussed when scientists test a new medicine. A common example is a study on depression treatment. A pharmaceutical company may compare two groups. The experimental group receives the medication, and the control group does not. To keep participants from knowing which treatment they got, researchers may give...

blind procedures

Single-Blind and Double-Blind Studies

Single-blind and double-blind studies help researchers reduce bias in an experiment. Bias can happen when expectations affect how a study is carried out or how results are read. In this kind of research, blinding keeps those expectations from shaping the outcome. In a single-blind study, the participants do not know whether they are in the experimental group or the control group. The researcher who designed the study still knows who is in each group. This setup helps limit the effect of...

ethics in research

Research Oversight for Human and Animal Studies

Research oversight protects human dignity and safety in modern science. Good research must be ethical, and researchers must show that their studies meet that standard. This idea is especially important when studies involve people or animals. Studies with human participants follow strict rules to prevent harm. Research institutions that receive federal support for human studies must have access to an institutional review board, or IRB. An IRB is a committee made up of administrators,...

correlations

Reading Correlation Coefficients

Correlation describes a relationship between two or more variables. It does not automatically mean that one variable causes the other. When variables are correlated, one changes as the other changes, and that relationship can be measured with a correlation coefficient. A correlation coefficient is a number from -1 to +1. It is usually written as r. The number shows the strength of the relationship. Values closer to 1, whether positive or negative, show a stronger relationship and make changes...

cause and effect

Why Correlation Can Mislead

Correlation can be misleading when two variables move together for a different reason. A confounding variable, or hidden factor, may be causing the pattern instead of one variable directly affecting the other. The ice cream and crime example shows this clearly. When temperatures rise, more people go outside, buy ice cream, and may also have more chances for conflict. A correlation does not prove causation. Even when a relationship looks strong, we should not assume that one variable causes the...

Reliable and valid measurements are essential in data collection and research. Reliability means a tool or test gives the same result in a consistent way. In psychology, that means the instruments used to collect data should produce reproducible results. Consistency alone is not enough. A kitchen scale can show the same reading every time, but it can still be wrong if it is not calibrated correctly. In that case, the scale is reliable because it is consistent, but it does not measure the...

regression toward the mean

Regression Toward the Mean in Data

Regression toward the mean is a statistical pattern in which very high or very low values move closer to the average when they are measured again. A blood pressure reading taken at one moment is one example. The change is usually caused by random error and chance, not by a real underlying shift. This idea matters because it can affect many kinds of research. If scientists do not account for regression toward the mean, they may think a treatment worked when the difference was really just a...

measures of central tendency

Mean, Median, and Mode in Data

Mean, median, and mode are the main ways to describe the center of a data set. The center is also a way to describe location. The mean and the median are the two most widely used measures of center. The mean is the arithmetic mean, which most people call the average. In everyday use, average and mean are often used to mean the same thing. The mode is another measure of center. It is the value that appears most often in a data set. If two values occur the same number of times, the data set is...

variation: normal distribution, range, and standard deviation

variation-normal-distribution-range-and-standard-deviation

Measuring Data Spread with Range and Variance

Variation describes how far scores are from each other in a data set. In psychology, researchers organize measurements of traits, features, or characteristics, called variables, in different ways. Qualitative data, such as ethnicity, can be counted to show the proportion and variety of groups in a sample or population. Quantitative data allow for more calculations, including measures of central tendency and variability. The mean, mode, and median are central tendency measures. They identify a...

statistical significance

Testing Whether Group Differences Matter

Statistical analysis helps determine whether differences between an experimental group and a control group are meaningful. After data are collected, researchers check whether the results are likely to be real or just due to chance. If a difference is caused by chance, it is not considered meaningful. In psychology, a group difference is usually called significant when the chance of it happening by accident is 5 percent or less. Another way to say this is that if the experiment were repeated...

key terms and concepts

Learn how reliability and validity shape research measurements, from calibrated scales to SAT scores, and why both accuracy and consistency matter.

Testing Measurement Quality in Research | JoVE K12

JoVE Core Social Psychology offers a comprehensive learning experience on the principles of social behavior through concise, animated video lessons. The series explores key topics such as self-concept, social cognition, group dynamics, stereotypes, and relationships. It provides a structured foundation for understanding human behavior in social situations. Students gain insights into how individuals think, feel, and behave in social contexts while examining the psychological processes that influence interactions and judgments. By integrating theory with real-world applications, these lessons offer a clear and engaging approach to mastering essential concepts in social psychology.

social psychology

Life Science

LS.1

LS.1.d

Texas Essential Knowledge and Skills

High School

Biology

BIO.2

BIO.2.B

Scientific Method

The scientific method is used to solve problems and explain phenomena. The development of the scientific method coincided with changes in philosophy ...

Scientific Laws and Theories

Scientific Laws

In science, a law is defined as a concise, verbal or mathematical, statement that summarizes a vast number of experimental observations. ...

The Scientific Method

The scientific method is a detailed, empirical problem-solving process used by biologists and other scientists. This iterative approach involves ...

Chemistry is an empirical science. Scientists often pose questions to understand the chemistry in everyday life and seek answers to these questions. To ...

What is a Hypothesis?

A hypothesis can be a simple sentence or statement about a property or any phenomenon observed or predicted for a population. It is usually a claim about ...

Types of Hypothesis Testing

There are three types of hypothesis tests: right-tailed, left-tailed, and two-tailed.
When the null and alternative hypotheses are stated, it is observed ...

Hypothesis: Accept or Fail to Reject?

The outcome of any hypothesis testing leads to rejecting or not rejecting the null hypothesis. This decision is taken based on the analysis of the data, ...

The student will demonstrate an understanding of scientific and engineering practices by asking questions and defining problems
• ask questions that arise from careful observation of phenomena, examination of a model or theory, or unexpected results, and/or to seek additional information
• determine which questions can be investigated within the scope of the school laboratory or field experience
• generate hypotheses based on research and scientific principles
• make hypotheses that specify what happens to a dependent variable when an independent variable is manipulated
• define design problems that involve the development of a process or system with multiple components and criteria

ES.1.a

Inductive Reasoning

Inductive reasoning is a form of logical thinking that uses related observations to arrive at a general conclusion. It is uncertain and operates in ...

Deductive Reasoning

Deductive reasoning, or deduction, is the type of logic used in hypothesis-based science. In deductive reasoning, the pattern of thinking moves in the ...

Correlation and Causation

Correlation and CausationStatistical tests can calculate whether there is a relationship, or correlation, between independent and dependent variables. A ...

Demonstrating the Scientific Method with Tree Branches
ExpandObtain branches containing a few dozen typical leaves from five different local tree species. ...

Observation and Background Research
ExpandNOTE: The first two steps in the scientific process involve studying and observing natural phenomena of ...

The student will demonstrate an understanding of scientific and engineering practices by planning and carrying out investigations
• individually and collaboratively plan and conduct observational and experimental investigations
• plan and conduct investigations to test design solutions in a safe and ethical manner including considerations of environmental, social and personal effects
• select and use appropriate tools and technology to collect, record, analyze, and evaluate data

ES.1.b

Introduction to Test of Independence

In statistics, the term independence means that one can directly obtain the probability of any event involving both variables by multiplying their ...

Statistical Analysis: Overview

An analysis is usually conducted by replicated sampling or repeated measurements on the same sample. This leads to scattered results rather than a single ...

Statistical Significance

Once data is collected from both the experimental and the control groups, a statistical analysis is conducted to find out if there are meaningful ...

Review and Preview

Data are individual items of information obtained from a population or sample. Data may be classified as qualitative (categorical), quantitative ...

Data: Types and Distribution

In biostatistics, data refers to the collected observations that are subject to analysis. Data can be parametric or nonparametric.
Nonparametric data ...

Histogram

The histogram is a graphical representation in the x-y form of data distribution in a data set. The horizontal x-axis is labeled with what the data ...

Scatter Plot

The most common and easiest way to display the relationship between two variables, x and y, is a scatter plot. A scatter plot shows the direction of a ...

Bar Graph

A bar graph is also called a bar chart and consists of bars that are separated from each other. It either uses horizontal or vertical bars to show ...

Pie Chart

A pie chart (or a pie graph) is a circular graphical chart or a pictorial representation of categorical data. It is divided into slices of pie each ...

Numerical Calculations

In engineering applications, the representation of the numerical value is critical. Presenting or reporting the answer is one of the essential parts of ...

Accuracy and Precision

Scientists typically make repeated measurements of a quantity to ensure the quality of their findings and to evaluate both the precision and the accuracy ...

Significant Figures in Calculations

Uncertainty in measurements can be avoided by reporting the results of a calculation with the correct number of significant figures. This can be ...

Realism in Experimentation

Source: Laboratories of Gary Lewandowski, Dave Strohmetz, and Natalie Ciarocco&#8212;Monmouth University

In an ideal world researchers would conduct ...

The student will demonstrate an understanding of scientific and engineering practices by pnterpreting, analyzing, and evaluating data
• construct and interpret data tables showing independent and dependent variables, repeated trials, and means
• construct, analyze, and interpret graphical displays of data and consider limitations of data analysis
• apply mathematical concepts and processes to scientific questions
• use data in building and revising models, supporting explanations of phenomena, or testing solutions to problems
• analyze data using tools, technologies, and/or models in order to make valid and reliable scientific claims or determine an optimal design solution

ES.1.c

Data Collection by Observations

Data collection refers to a systematic way of obtaining, observing, measuring, and analyzing accurate information. Observational studies are one of the ...

Data Collection by Experiments

Data collection is a systematic method of obtaining, observing, measuring, and analyzing accurate information. An experimental study is a standard method ...

Data Collection by Survey

The systematic method of obtaining and analyzing accurate information of a population is called data collection. A survey is a standard method of data ...

Reasoning

Reasoning involves transforming information into conclusions, which is essential for problem-solving, decision-making, and critical thinking.
Reasoning ...

The student will demonstrate an understanding of scientific and engineering practices by constructing and critiquing conclusions and explanations
• make quantitative and/or qualitative claims based on data
• construct and revise explanations based on valid and reliable evidence obtained from a variety of sources, including students' own investigations, models, theories, simulations, and peer review
• apply scientific ideas, principles, and/or evidence to provide an explanation of phenomena or design solutions
• construct arguments or counterarguments based on data and evidence
• differentiate between a scientific hypothesis, theory, and law

using-topographic-maps-to-generate-topographic-profiles

Using Topographic Maps to Generate Topographic Profiles

Source: Laboratory of Alan Lester - University of Colorado Boulder
Topographic maps are &#34;plan-view&#34; representations of Earth&#39;s ...

The student will demonstrate an understanding of scientific and engineering practices by developing and using models
• evaluate the merits and limitations of models
• develop, revise, and/or use models based on evidence to illustrate or predict relationships
• construct and interpret scales, diagrams, classification charts, graphs, tables, imagery, models, including geologic cross sections and topographic profiles
• read and interpret topographic and basic geologic maps and globes, including location by latitude and longitude

ES.1.e

Data Source

A data source is the origin of collected data, whether physical or digital, and includes all channels, platforms, and repositories for gathering and ...

Communication

Sharing information, concepts, and emotions to foster mutual understanding is communication. The sender, recipient, and transaction must be considered in ...

levels-of-communication-i-intrapersonal-interpersonal-and-small-group

Levels of Communication I: Intrapersonal, Interpersonal, and Small Group

Interpersonal communication focuses on the exchange of messages between two people.
We can participate in these relationships through verbal, nonverbal, ...

levels-of-communication-ii-organizational-public-and-group-dynamics

Levels of Communication II: Organizational, Public, and Group Dynamics

Effective communication is the foundation of a good organization. Communication is the lifeblood of an organization that connects the group with messages. ...

The student will demonstrate an understanding of scientific and engineering practices by obtaining, evaluating, and communicating information
• compare, integrate, and evaluate sources of information presented in different media or formats to address a scientific question or solve a problem
• gather, read, and evaluate scientific and/or technical information from multiple sources, assessing the evidence and credibility of each source
• communicate scientific and/or technical information about phenomena and/or a design process in multiple formats

ES.1.f

Testing Measurement Quality in Research

Description

Transcript

Tags

More videos by Standards