Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning

In the emerging field of university service-learning, we lack standardized instruments that combine reliability and validity, both qualitative and quantitative, and brief enough to be easily used. The incorporation of rating and scales allow for comparison between professionals to evaluate the quality of university service-learning. Currently, there is no other instrument that achieve this.

QaSLu is the first instrument created through a rigorous qualitative and quantitative validation process guided by the principle of parsimony. It is brief, but extremely scientifically robust. This protocol sets the path forward for new scientific instruments that are valid, robust and practical.

Not only for service-learning, but also in the field of education in general. We plan to create new rating and scales for other different groups and expand the participating sample to improve the valuation of QaSLu, and make it useful in different context. To begin, create the first draft of a QaSLu on a desktop.

Conduct a content analysis of items from primary tools recognized for evaluating service-learning experiences. Review items from international tools that define standards for quality and from the primary rubrics recognized by the service-learning expert community. Categorize all items by the phases involved in designing service-learning experience, like design, implementation and evaluation.

Then categorize all items by the application of service-learning, like initial phase, planning, execution, closure, and replication. Select a large sample of experts in university service-learning to evaluate the draft instrument. Configure the final expert group to evaluate the definitive instrument for assessing USL, aiming for a list of experts with academic and professional diversity, geographic diversity and gender diversity.

Once the data from the experts in round 1 has been received, calculate Kendall's coefficient of concordance for relevance by sequentially clicking on Analyze, Nonparametric Tests, Legacy Dialogues, and K Related Samples. Choose Kendall's W under Test Type. Then select all responses from the experts regarding relevance and clarity.

Move items to Test Variables list and click OK to run the analysis. After receiving the round 2 expert feedback, reassess Kendall's coefficient of concordance is done earlier and remove items rated as highly irrelevant. Reformulate items considered highly relevant but unclear to improve wording and precision to yield the final version of the Questionnaire for the Self-Assessment of University Service-Learning Experiences, QaSLu-45.

Once sufficient completed questionnaires have been obtained, assess the internal consistency of the questionnaire using statistical software package one. Click on Analyze, followed by Scale, then Reliability Analysis. Choose Alpha in the dialogue box, then transfer all relevant variables into the Reliability Analysis dialogue box and click OK to generate the output.

Perform exploratory factor analysis in statistical software package one by opening Analyze, then click on Dimension Reduction, followed by Factor. In the Factor Analysis dialogue, transfer all questionnaire items into the Variables box, then press Descriptives and KMO and Bartlett's test of sphericity and select Univariate Descriptives. Click Continue.

Next, press Extraction and select Principal Components as the extraction method. Check the options for Unrotated factor solution and Scree plot, then click Continue, and press OK to generate the output. Identify the items for removal that show an inverse relationship with the principal component.

Using the component matrix obtained in the results of the exploratory factor analysis, flag items with negative loadings on the principal factor and exclude them from the reduced, optimized single factor version of QaSLu. Next, perform confirmatory factor analysis with statistical software package two to assess the goodness of fit of the one factor model. Select Read Data, click Browse in the Sample dialogue and select the previously created ASCII data file.

Verify that the number of participants and variables is displayed in green in Size of data matrices. Now press Open single group dataset and return to the main menu. Click Configure Analysis in the main menu, then check Confirmatory Factor Analysis in the dialogue menu and click Confirm, an the CFA:Factor analysis configuration window.

accept the default settings and click Confirm to return to the main menu. When the message, Confirmatory Factor Analysis ready! appears, click on the Compute button and wait for the results.

Next, open the statistical software package two to perform a robust unweighted least squares exploratory factor analysis. Select Read Data, then click Browse in the Sample dialogue box and choose the previously created ASCII file. The number of participants and variables will be displayed green in the Size of data matrices section.

Now, click on the Open single group dataset icon to return to the main menu. In the main menu, click on Configure Analysis, then select Exploratory Factor Analysis in the dialogue menu and click the Confirm button. In the Exploratory Factor Analysis Configuration menu, choose the items identified as having an inverse relationship to the principal component and move them to the Excluded variables box.

Now, check the Pearson correlation matrix and Parallel Analysis options, set Number of factors or components to 1 and select Robust Factor Analysis. Click Confirm to return to the main menu and then click Compute to generate the results. To identify items for removal that have a measure of sampling adequacy or MSA score below 0.49, review the results obtained from the RULS exploratory factor analysis with statistical software package two, specifically the item location and item adequacy indices, and exclude them from the reduced optimized single factor version of QaSLu.

Create a new variable in statistical software package one without the eliminated items by selecting the Transform menu and pressing Compute Variable. Name the new variable as SUM 27, then select the SUM function and complete the numeric expression with the optimized item list. Click OK to generate the new variable, SUM 27.

Calculate percentiles for SUM 27 by selecting Analyze, followed by Descriptive Statistics and Frequencies. Move SUM 27 into Variables box then click on the Statistics icon. Select Percentiles and add 1, 5, 15, 25, 35, 45, 55, 65, 75, 85, 95, and 99.

Click on Continue then OK to generate percentiles. Repeat the percentile procedure for each rating scale by clicking on Split File icon and choosing each relevant variable, then select Organize output by groups. Click the arrow button and press OK to generate grouped percentiles for SUM 27.

To create a table for each rating scale, first add a title row and rows for the levels Low, Medium/Low, Medium, Medium/High, and High with the specified percentile subrows. In the title row, add a Level column, a Percentile column, and additional columns for the conditions of the selected variable. Complete each table with score ranges mapped to the calculated percentiles.

The results obtained from the participating sample are presented to established comparisons based on the rating scales used in the study. The different ranks for each category were made up of the intermediate points between them. For example, the 50th percentile were obtained from the scores of the 45th percentile and the 54th percentile.

All ranks except the 1st and 99th percentile show perfectly bounded ranges corresponded to the 4th and 95th percentiles plus any possible score adjustments. Each of these scales allows comparing the scores of each perspective user by selecting the appropriate rating scale according to their characteristics.