Name: Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning (Video) | JoVE Journal
Uploaded: 2025-08-29T14:10:00.000+00:00
Duration: 10 min 39 s
Description: Learn how to validate assessment tools for university service-learning using qualitative and quantitative methods. This protocol demonstrates rigorous instrument development through Delphi and factor analysis techniques.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

The authors would like to express their gratitude to the National Commission for Scientific and Technological Research of Chile (CONICYT Fondecyt/Initiation), as well as to all the experts who participated in the qualitative validation through the Delphi method and to all the participants in the quantitative validation and scoring.

The authors have nothing to disclose.

The methods used for instrument validation offer several advantages. The Delphi method facilitated the collection of insights from a panel of experts, enabling the systematic organization of this information for analysis. It allowed for the selection and refinement of responses based on expert estimates and suggestions32,47,48,49. Furthermore, this approach yielded significant results within a ...

Service-Learning (SL), an educational approach that integrates academic learning with community service, has increasingly gained recognition in contemporary education. This pedagogical approach enables students to apply classroom knowledge to real-world challenges, while also fostering civic responsibility and community engagement1. The significance of SL in current education cannot be overstated, as it equips students with practical professional competencies, promotes civic engagement, and provides a deeper understanding of social issues2.
Recent research on SL has grown significantly, fueled by increasing acknowledgment of its potential to enhance both academic learning and community involvement. This growth is driven by the rising demand for educational models that not only promote academic development but also address societal challenges3. Students have explored the diverse impacts of SL, from promoting social justice to cultivating civic-mindedness among students4. Additionally, service-learning research has increasingly focused on its application in specific fields such as STEM, healthcare, and environmental science, where real-world problem-solving is critical5,6,7. As the body of literature expands, researchers are also beginning to explore how service-learning can effectively bridge the gap between theory and practice in ways that are meaningful to both students and the communities they serve8.
However, despite its growing role, significant challenges remain in measuring the effectiveness of SL programs9. There are various proposals in the form of a rubric, such as those of Furco10, Mart&#237;n et al.11, Lorenzo and Belando-Montoro12, Shek et al.13, or Lu and Lambright14, which, despite their proven usefulness in field studies, have not yet been validated. On the other hand, there are other proposals, such as those of Escofet-Roig et al.15, Rodr&#237;guez-Izquierdo16, Le&#243;n-Carrascosa et al.17, Ruiz-Ordo&#241;ez et al.18, Santos-Pastor et al.19, or Gul et al.20 that lack qualitative validation or present very limited quantitative validations. Other shortcomings observed in previous assessment tools are the lack of optimization of factors and items to comply with the principle of parsimony and the complete lack of rating scales. This situation is common in new scientific fields and is the natural process that all areas of study go through to achieve optimal instruments for widespread use.
At the time of the creation of the Questionnaire for the Self-Assessment of University Service-Learning Experiences (QaSLu) a tool to evaluate the quality of the SL experiences at university level from the point of view of university professors, the most recognized tools that defined the standards for ensuring the quality of SL experiences were the following: Campo (2015)21, Europe Engage (2017)22, GREM (2014)23, RMC Research Corporation (2008)24, Puig-Rovira, Mart&#237;n, and Rubio-Serrano (2017)25, and Rubio-Serrano, Puig-Rovira, Mart&#237;n, and Palos (2015)26. While there are now additional tools, current measurement instruments often fail to adequately capture the broad and nuanced learning outcomes associated with these programs, especially in terms of long-term personal growth, community impact, with some found to be overly general or not based on rigorous scientific frameworks9,27. This gap in measurement tools hinders a comprehensive understanding and optimization of the impact of SL on both students and communities.
The next step in advancing SL research involves developing scientifically rigorous measurement instruments. These tools should be designed not only to capture academic outcomes but also to assess the personal and social impacts that SL can have on students and the communities they serve. Our research aims to address this gap by creating an instrument grounded in scientific rigor, offering a more comprehensive evaluation of SL projects and guiding their future implementation.
In addition to creating the instrument, it will be validated, first qualitatively and then quantitatively. For the initial validation, the modified online Delphi method will be used28. The term &#34;modified&#34; refers to Cabero&#39;s proposal to limit the Delphi method to two rounds of consultation, thereby avoiding dropout among the expert group members. The term &#34;online&#34; was introduced by Cruz-Ram&#237;rez and R&#250;a-V&#225;squez29 to highlight that consultations were conducted via email, as the Internet helps overcome certain limitations inherent in traditional Delphi, such as groupthink or the influence of dominant expert opinions, which could affect item selection30,31.
Using the modified online Delphi method, two types of validity will be assessed: construct validity, following the process outlined in Bakieva et al.&#39;s study32, where experts are consulted on the relevance of each item, and content validity, based on the work of Gil and Pascual-Ezama33, which evaluates the clarity of each item through expert consultation.
Experts will be provided with a Likert scale containing four response options (none, little, quite a lot, a lot). By not introducing any intermediate response alternatives, experts are required to take a definitive position in favor of or against each item34, thus facilitating the decision-making process on whether to retain or remove items in the questionnaire.
Regarding quantitative validation, it is based on the law of parsimony, a fundamental principle in science that states that, given a scientific problem, under equal conditions, the shortest solution is preferable35,36. Therefore, the search for the shortest and most optimized valid instrument offers the same scientific guarantees, being more practical, exportable and general for everyday use in real contexts without sacrificing reliability or validity. Confirmatory Factor Analysis (CFA)37 and Robust Unweighted Least Squares (RULS) Exploratory Factor Analysis38,39 will be used as analytical techniques to reduce unnecessary items and validate the final structure of the instrument. Fit and reliability indices are used as indicators of the validity of the new structure with respect to the previous one, taking as a reference to maintain or improve the reliability and validity of the optimized instrument within the levels that guarantee the adequacy and robustness of the new structure of the reduced and optimized questionnaire.

<table><tbody><tr><td>Encuestafacil.com Saas</td><td>Encuestafacil.com</td><td>2025</td><td>Software as a Service in the Encuestafacil.com private data server</td></tr><tr><td>Factor Analysis</td><td>Universitat Rovira i Virgili</td><td>12.06.07</td><td>Software package used in factorial analysis of data</td></tr><tr><td>Statistical Package for the Social Sciences (SPSS)</td><td>IBM</td><td>24</td><td>Software package used in statistical analysis of data</td></tr></tbody></table>

qualitative and quantitative validation of tools with rating scales aimed at assessing the quality of university service-learning

Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning

Service-Learning (SL) has become a highly effective learning methodology in recent years, bridging the gap between theory and practice while contextualizing learning in real-world scenarios. It has been successfully implemented across all levels of higher education, resulting in notable improvements in university students' skills. However, as a transformative methodology, SL is still in the early stages of development in terms of assessment methods. Consequently, there is a pressing need to design and validate reliable and, above all, valid measurement instruments, as the application of SL to real-world contexts is its primary value. This paper demonstrates how to optimally design and validate university SL instruments to evaluate the quality of the SL experiences at the university level from the point of view of university professors, guided by the principle of parsimony. The online modified Delphi method, Robust Unweighted Least Squares Exploratory Factor Analysis, and Confirmatory Factor Analysis have been applied to create a new instrument, the QaSLu-27, validated through a sample of 16 experts using the online modified Delphi method and 118 university teachers who implement SL projects. The results show optimal psychometric conditions for the new instrument (W = 0.11; &#945; = 0.92; CMIN/DF=1.027; CFI= 0.979; WRMR= 0.0985; RMSEA= 0.046), leading to the development of new rating scales that allow comparisons of future users of this methodology, considering their attributional and demographic characteristics. The design of new validated instruments, capable of being used in the real world and allowing the creation of scales, will be very useful to measure the implementation of new experiences in SL, to evaluate their progress, their participants, their managers and their monitors, contributing to the establishment of this field of study in the current academic framework and in the generation of new scientific knowledge.

Following this rigorous qualitative-quantitative validation protocol, we developed a robust, optimized, and parsimonious instrument for the self-assessment of university service-learning experiences, named QaSLu (Questionnaire for the Self-Assessment of Service-Learning Experiences in Higher Education). This final instrument has received qualitative approval from 16 SL experts under the Delphi method, with two rounds of online consultations and adaptations, as well as robust quantitative validation through Confirmatory F...

This paper outlines the protocol for qualitatively and quantitatively validating a university service-learning instrument, following the parsimony criterion to create the most robust and optimal version possible. The Delphi qualitative validation method and Robust Unweighted Least Squares Exploratory Factor Analysis, as a quantitative item optimization method, are utilized for this purpose.

Service-Learning (SL) has become a highly effective learning methodology in recent years, bridging the gap between theory and practice while ...

In the emerging field of university service-learning, we lack standardized instruments that combine reliability and validity, both qualitative and quantitative, and brief enough to be easily used. The incorporation of rating and scales allow for comparison between professionals to evaluate the quality of university service-learning. Currently, there is no other instrument that achieve this.QaSLu is the first instrument created through a rigorous qualitative and quantitative validation process guided by the principle of parsimony. It is brief, but extremely scientifically robust. This protocol sets the path forward for new scientific instruments that are valid, robust and practical.Not only for service-learning, but also in the field of education in general. We plan to create new rating and scales for other different groups and expand the participating sample to improve the valuation of QaSLu, and make it useful in different context. To begin, create the first draft of a QaSLu on a desktop.Conduct a content analysis of items from primary tools recognized for evaluating service-learning experiences. Review items from international tools that define standards for quality and from the primary rubrics recognized by the service-learning expert community. Categorize all items by the phases involved in designing service-learning experience, like design, implementation and evaluation.Then categorize all items by the application of service-learning, like initial phase, planning, execution, closure, and replication. Select a large sample of experts in university service-learning to evaluate the draft instrument. Configure the final expert group to evaluate the definitive instrument for assessing USL, aiming for a list of experts with academic and professional diversity, geographic diversity and gender diversity.Once the data from the experts in round 1 has been received, calculate Kendall's coefficient of concordance for relevance by sequentially clicking on Analyze, Nonparametric Tests, Legacy Dialogues, and K Related Samples. Choose Kendall's W under Test Type. Then select all responses from the experts regarding relevance and clarity.Move items to Test Variables list and click OK to run the analysis. After receiving the round 2 expert feedback, reassess Kendall's coefficient of concordance is done earlier and remove items rated as highly irrelevant. Reformulate items considered highly relevant but unclear to improve wording and precision to yield the final version of the Questionnaire for the Self-Assessment of University Service-Learning Experiences, QaSLu-45.Once sufficient completed questionnaires have been obtained, assess the internal consistency of the questionnaire using statistical software package one. Click on Analyze, followed by Scale, then Reliability Analysis. Choose Alpha in the dialogue box, then transfer all relevant variables into the Reliability Analysis dialogue box and click OK to generate the output.Perform exploratory factor analysis in statistical software package one by opening Analyze, then click on Dimension Reduction, followed by Factor. In the Factor Analysis dialogue, transfer all questionnaire items into the Variables box, then press Descriptives and KMO and Bartlett's test of sphericity and select Univariate Descriptives. Click Continue.Next, press Extraction and select Principal Components as the extraction method. Check the options for Unrotated factor solution and Scree plot, then click Continue, and press OK to generate the output. Identify the items for removal that show an inverse relationship with the principal component.Using the component matrix obtained in the results of the exploratory factor analysis, flag items with negative loadings on the principal factor and exclude them from the reduced, optimized single factor version of QaSLu. Next, perform confirmatory factor analysis with statistical software package two to assess the goodness of fit of the one factor model. Select Read Data, click Browse in the Sample dialogue and select the previously created ASCII data file.Verify that the number of participants and variables is displayed in green in Size of data matrices. Now press Open single group dataset and return to the main menu. Click Configure Analysis in the main menu, then check Confirmatory Factor Analysis in the dialogue menu and click Confirm, an the CFA:Factor analysis configuration window.accept the default settings and click Confirm to return to the main menu. When the message, Confirmatory Factor Analysis ready! appears, click on the Compute button and wait for the results.Next, open the statistical software package two to perform a robust unweighted least squares exploratory factor analysis. Select Read Data, then click Browse in the Sample dialogue box and choose the previously created ASCII file. The number of participants and variables will be displayed green in the Size of data matrices section.Now, click on the Open single group dataset icon to return to the main menu. In the main menu, click on Configure Analysis, then select Exploratory Factor Analysis in the dialogue menu and click the Confirm button. In the Exploratory Factor Analysis Configuration menu, choose the items identified as having an inverse relationship to the principal component and move them to the Excluded variables box.Now, check the Pearson correlation matrix and Parallel Analysis options, set Number of factors or components to 1 and select Robust Factor Analysis. Click Confirm to return to the main menu and then click Compute to generate the results. To identify items for removal that have a measure of sampling adequacy or MSA score below 0.49, review the results obtained from the RULS exploratory factor analysis with statistical software package two, specifically the item location and item adequacy indices, and exclude them from the reduced optimized single factor version of QaSLu.Create a new variable in statistical software package one without the eliminated items by selecting the Transform menu and pressing Compute Variable. Name the new variable as SUM 27, then select the SUM function and complete the numeric expression with the optimized item list. Click OK to generate the new variable, SUM 27.Calculate percentiles for SUM 27 by selecting Analyze, followed by Descriptive Statistics and Frequencies. Move SUM 27 into Variables box then click on the Statistics icon. Select Percentiles and add 1, 5, 15, 25, 35, 45, 55, 65, 75, 85, 95, and 99.Click on Continue then OK to generate percentiles. Repeat the percentile procedure for each rating scale by clicking on Split File icon and choosing each relevant variable, then select Organize output by groups. Click the arrow button and press OK to generate grouped percentiles for SUM 27.To create a table for each rating scale, first add a title row and rows for the levels Low, Medium/Low, Medium, Medium/High, and High with the specified percentile subrows. In the title row, add a Level column, a Percentile column, and additional columns for the conditions of the selected variable. Complete each table with score ranges mapped to the calculated percentiles.The results obtained from the participating sample are presented to established comparisons based on the rating scales used in the study. The different ranks for each category were made up of the intermediate points between them. For example, the 50th percentile were obtained from the scores of the 45th percentile and the 54th percentile.All ranks except the 1st and 99th percentile show perfectly bounded ranges corresponded to the 4th and 95th percentiles plus any possible score adjustments. Each of these scales allows comparing the scores of each perspective user by selecting the appropriate rating scale according to their characteristics.

qualitative-quantitative-validation-tools-with-rating-scales-aimed-at

Research

JoVE Journal

Behavior

The structure and process for measuring student learning outcomes through customizable assessment rubrics is discussed and applied to a doctoral nursing education program.

Using Learning Outcome Measures to assess Doctoral Nursing Education

The current work proposes a multimodal evaluation protocol focused on metacognitive, self-regulation of learning, and emotional processes, which make up the basis of the difficulties in adults with LDs.

Multimodal Protocol for Assessing Metacognition and Self-Regulation in Adults with Learning Difficulties

Here, we present a protocol for the behavioral analysis of a project-based learning methodology for health sciences students (20-56 years old). The protocol facilitates the comparison of the participants' performance in e-Learning versus blended-Learning (b-Learning) through a monitoring tool. The results are analyzed using Educational Data Mining and qualitative techniques.

Project-Based Learning Guidelines for Health Sciences Students: An Analysis with Data Mining and Qualitative Techniques

Three new behavioral tests (forelimb step-alternation, postural instability test, pasta handling test) for evaluating forelimb function after cervical spinal cord injury in rodents are described.

Assessing Forelimb Function after Unilateral Cervical SCI using Novel Tasks: Limb Step-alternation, Postural Instability and Pasta Handling

Objective and easy measurement of sensory processing is extremely difficult in nonverbal or vulnerable pediatric patients. We developed a new methodology to quantitatively assess infants and children's cortical processing of light touch, speech sounds, and the multisensory processing of the 2 stimuli, without requiring active subject participation or causing discomfort in vulnerable patients.

Quantitative Assessment of Cortical Auditory-tactile Processing in Children with Disabilities

We present a method to compare functional brain activity recorded during a naturalistic task using fNIRS with activity recorded during fMRI.

fMRI Validation of fNIRS Measurements During a Naturalistic Task

This manuscript describes a semi-automated task that quantifies supination in rats. Rats reach, grasp, and supinate a spherical manipulandum. The rat is rewarded with a pellet if the turn angle exceeds a criterion set by the user. This task increases throughput, sensitivity to injury, and objectivity compared to traditional tasks.

The Knob Supination Task: A Semi-automated Method for Assessing Forelimb Function in Rats

This article describes the procedures for conducting a verbal operant analysis (VOA), calculating the stimulus control ratio equation (SCoRE), and developing individualized verbal behavior treatment plans, specifically significant to address the deficits characteristic of autism.

Modeling Verbal Behavior Deficits with the Stimulus Control Ratio Equation, SCoRE

This article presents a new conceptualization of the in-store search process, the 3S Model, which captures customers’ visual attention at three distinct levels of analysis: Stock, Shelf, and Store. We illustrate the usefulness of our conceptualization through three eye-tracking studies, one from each level of analysis in the 3S Model.

Spotlighting Customers' Visual Attention at the Stock, Shelf and Store Levels with the 3S Model

The subconcussive soccer heading model is a safe and concise methodological approach to isolate and measure the effects of subconcussive head impacts.

In Vivo Protocol of Controlled Subconcussive Head Impacts for the Validation of Field Study Data

Learn how to validate assessment tools for university service-learning using qualitative and quantitative methods. This protocol demonstrates rigorous instrument development through Delphi and factor analysis techniques.

Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning (Video) | JoVE Journal

Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning

Explore More Videos