Sustainability of Digital International Chinese Literature Teaching via Fuzzy Analytic Hierarchy Process and Fuzzy Comprehensive Evaluation

Summary

This protocol details a reproducible fuzzy analytic hierarchy process and fuzzy comprehensive evaluation (FAHP-FCE) workflow to quantify the sustainability of digitally assisted international Chinese literature teaching. It operationalizes sustainability across technical, pedagogical, cultural, and policy dimensions, and outputs indicator weights, dimension scores, and an interpretable sustainability index for cross-context comparison.

Abstract

This protocol describes a replicable method to evaluate the sustainability of digitally assisted international Chinese literature teaching using a hybrid FAHP–FCE model. The overall goal is to convert multidimensional sustainability judgments into a transparent, quantitative sustainability index (SI). The workflow constructs a four-dimensional indicator system comprising 19 indicators. Expert judgments derive indicator weights via FAHP, while learner and teacher questionnaire responses are mapped to membership degrees using the FCE procedure. Weighted fuzzy synthesis and defuzzification then yield the SI. Representative implementation across three universities in China (237 learners and 9 teachers) produced an overall SI of 78.2 (“Good”), with meaningful regional variation consistent with differences in cultural adaptation and institutional resources. The protocol is highly dependent on convening a credible expert panel. Furthermore, indicators must be operationalized consistently, as differing local policy definitions or curricular goals may complicate direct cross-site comparisons. Future applications should include longitudinal implementation across multiple semesters to test the stability of weights and membership distributions, alongside comparative studies that benchmark this fuzzy approach against alternative multi-criteria models.

Introduction

In recent years, the accelerating convergence of globalization and digitalization has reshaped international Chinese education, shifting from technology-assisted classrooms towards digitally native learning ecosystems^1,2. Within this shift, Chinese literature courses occupy a distinctive position because they do more than support language acquisition; they also cultivate cultural identity, aesthetic sensibility, and interpretive competence^3,4. These outcomes can be especially fragile in digital environments, where prevailing evaluation approaches often prioritize immediate, easily quantifiable cognitive gains while under-representing longer-horizon goals such as emotional resonance, sustained cultural engagement, and interpretive depth^5,6. Crucially, this protocol evaluates educational sustainability through the lens of human behavior. By quantifying the cognitive responses, emotional engagement, and instructional behaviors of learners and teachers interacting with digital literature platforms, the framework translates subjective human experiences into measurable behavioral indicators.

Existing assessments remain limited in both scope and duration. Many studies still rely on simplified measures of knowledge mastery⁷, leaving complex learner experiences such as aesthetic pleasure, cultural empathy, and dialogic interpretation underspecified or treated as peripheral outcomes^8,9. Moreover, short observation windows-often shorter than a single semester-may miss cumulative effects, including how digital tools gradually shape learners’ interpretive routines and how teachers’ digital-pedagogical expertise develops through iterative practice^10,11. Evaluation frameworks are also frequently fragmented: technical or pedagogical indicators are reported in isolation, while cultural and policy conditions that can constrain or enable sustainable implementation are rarely integrated into one coherent model¹². This fragmentation reduces transferability across sites, particularly in cross-cultural literature classrooms, where meaning-making is inherently open-ended and context-sensitive^13,14.

While previous research has extensively explored digital tool adoption and short-term cognitive gains in language acquisition, evaluation frameworks remain frequently fragmented. Existing literature lacks comprehensive models that integrate technical, pedagogical, cultural, and policy dimensions into a single measurable standard. This study needs to be conducted to address this critical gap by introducing a standardized FAHP-FCE framework, transitioning from subjective, short-term evaluations to a transparent, reproducible sustainability index that supports cross-site comparison.

To address these limitations, this protocol introduces a reproducible framework for evaluating the sustainability of digitally assisted international Chinese literature teaching. Building on fuzzy multi-criteria decision-making logic, the method integrates fuzzy analytic hierarchy process (FAHP) with fuzzy comprehensive evaluation (FCE)¹⁵ to operationalize sustainability across four linked dimensions: technical functionality, pedagogical effectiveness, cultural adaptability, and policy support. The approach is designed to offer three practical advantages over “crisp” scoring models: it transparently derives indicator weights from expert judgments while accommodating linguistic uncertainty; it converts questionnaire responses into membership degrees that better reflect gradated perceptions; and it yields an interpretable Sustainability Index that supports cross-site comparison and sensitivity analysis^14,16,17. By embedding interpretive transparency, cross-cultural adaptability, and long-term reliability into the evaluation workflow, the protocol moves beyond short-term outcome checks and provides a transferable assessment tool for other digitally mediated and cross-cultural teaching contexts¹⁸.

Protocol

Ensured the study complies with institutional human research ethics guidelines and applicable data-protection regulations^19,20. Written informed consent was obtained from each participant before any study procedure. Assigned each participant a study code, anonymized all datasets, and stored the linkage key (identifiers <> codes) in a separate, access-restricted file. Restricted raw data access to authorized personnel only, and document the retention period and deletion plan in the ethics application.

1. Preparation of evaluation context and materials

1. Define the instructional context (i.e., the specific educational environment, learner demographic profile, and pedagogical framework in which the digitally assisted teaching occurs).
   1. Specify the course type (international Chinese literature), target population, and delivery mode (in-person, blended, or online).
   2. Define the minimum learner proficiency threshold (e.g., Chinese Proficiency Test [HSK] level 5 or equivalent) and state the operational eligibility criteria (test certificate, placement result, or instructor confirmation)^21,22.
   3. Specify the term length (weeks), contact hours, platform/tooling constraints (without brand names), and the single evaluation objective: quantify teaching sustainability using a FAHP–FCE workflow^23,24,25.
2. Compile the materials list
   1. Record all supplies (forms, instruments, scripts, software, and storage) in the Table of Materials, including: (i) the 19-item learner questionnaire (Supplementary File S1), (ii) the expert pairwise-comparison form (Supplementary File S2), (iii) informed-consent forms, (iv) data-capture templates, and (v) analysis scripts and their versions.
   2. Assign file names and version identifiers for reproducibility (e.g., Supplementary File S1_questionnaire_items.pdf, Supplementary File S2_expert_form.pdf, fahp_weighting.py, fce_membership.py, si_compute.py, and analysis_readme.txt)^26,27.
3. Create the workflow diagram (Figure 1)
   1. Draw a one-page flowchart summarizing sections 1–10, and label inputs/outputs at each handoff (expert judgments > fuzzy matrices > weights; learner responses > membership matrices; synthesis > Sustainability Index).
   2. Mark PAUSE POINTS (Section 3.2, 4.2, 8.3), which indicate critical stages where the procedure can be safely halted and where specific data artifacts must be archived to ensure reproducibility, and indicate what artifacts must be saved at each point.

2. Construct the indicator framework

1. Build a three-level hierarchy.
   1. Map the evaluation objective to four dimensions (technical functionality, pedagogical effectiveness, cultural adaptability, policy support).
   2. Define the 19 indicators under the four dimensions (Objective > Dimension > Indicator), and ensure each indicator is measurable via questionnaire response or documented evidence.
2. Define indicators and the response format.
   1. Write an operational definition for each indicator, specify whether it is positive (higher is better) or negative (lower is better), and record the indicator–dimension mapping in a data dictionary (Table 1).
   2. Specify the learner response scale as a 7-point Likert format with anchors: 1 = strongly disagree; 7 = strongly agree.
      NOTE: Finalize and freeze the indicator set for reuse across cohorts and archive the data dictionary and Table 1 version.

3. Assemble the expert panel and collect pairwise judgments (FAHP)

1. Recruit an expert panel.
   1. Recruit a cross-role panel (e.g., literature instructors, educational-technology specialists, assessment researchers, and administrators).
   2. Define eligibility criteria (years of experience, familiarity with digital instruction, and cross-cultural teaching exposure), and document the panel composition for transparency.
2. Collect fuzzy pairwise comparisons.
   1. Train experts with a short briefing on the dimensions/indicators and the meaning of triangular fuzzy numbers (TFNs)^28,29,30.
   2. Provide explicit anchors and examples to reduce variability (e.g., TFN [2, 3, 4] = “important”; TFN [4, 5, 6] = “very important”).
   3. Collect pairwise judgments for (i) the four dimensions and (ii) indicators within each dimension and verify completeness before closing the session.

4. Build fuzzy judgment matrices and test consistency

1. Construct fuzzy judgment matrices.
   1. Convert each expert’s pairwise responses into a fuzzy reciprocal matrix at each hierarchy level²⁵.
   2. Check for missing entries and enforce reciprocity rules before aggregation.
2. Test consistency and reconcile disagreements.
   1. Defuzzify each expert matrix to a crisp matrix using a prespecified rule, such as the centroid of a triangular fuzzy number (TFN)^31,32,33. The crisp value is calculated using the following equation:
      
      where C is the defuzzified crisp value, and , , and represent the lower bound, modal (most likely) value, and upper bound of the TFN, respectively.
   2. Compute the consistency ratio (CR) on the crisp matrix using the Analytic Hierarchy Process (AHP) procedure^34,35. This involves calculating the principal eigenvalue (λ_max) of the matrix, determining the consistency index (, where n is the matrix size), and dividing it by the appropriate random index (RI) to obtain the CR (). Set the acceptance threshold as CR < 0.10.
   3. If CR ≥ 0.10, return the inconsistent comparisons to the expert, reconcile disagreements using facilitated discussion, and recollect only the disputed judgments.
      PAUSE POINT: Archive each matrix, CR report, and reconciliation log as replication artifacts.

5. Derive fuzzy weights (FAHP output)

1. Aggregate individual judgments.
   1. Aggregate expert matrices into a group-level fuzzy matrix using a specified operator (e.g., fuzzy geometric mean)^17,36.
   2. Record the aggregation operator and parameters in Supplementary File S2.
2. Defuzzify and normalize weights.
   1. Defuzzify the group-level fuzzy priorities to obtain crisp weights using the same prespecified defuzzification rule¹⁰.
   2. Normalize weights within each level (dimensions sum to 1; indicators within each dimension sum to 1) and output the final dimension and indicator weight vectors.
   3. Populate Table 1 with final weights and lock the table version used for the subsequent FCE section.

6. Design and administer the learner questionnaire

1. Prepare the instrument.
   1. Provide the full 19 items verbatim in Supplementary File S1, each mapped to one indicator in Table 1.
   2. Use a 7-point Likert scale with clear anchor text printed on every page/screen (1 = strongly disagree; 7 = strongly agree).
   3. Pilot-test the questionnaire with a small group (e.g., 10–15 learners) to confirm comprehension and completion time³⁷.
2. Administer the questionnaire and enforce one unique response per participant.
   1. Invite eligible learners (HSK 5+ or operational equivalent) and establish a standardized data collection protocol. Specify the exact administration window (e.g., a defined two-week period at the end of the term) and the completion mode (e.g., in-class supervised sessions lasting 15–20 min to ensure focused participation, or controlled online distribution via secure, timed survey links). Instructors or researchers must provide a standardized briefing before questionnaire distribution, clearly explaining the study's purpose, assuring data anonymity, and defining the 7-point Likert scale anchors to all respondents.
   2. Enforce one response per participant by issuing a single-use access token per learner (or an equivalent one-time code), disabling repeat submissions, and checking for duplicates using the token log.
   3. If anonymity is required, store tokens separately from responses and hash any login identifiers before analysis.
   4. Document all exclusion rules (e.g., incomplete responses, duplicate token entries) before data cleaning to prevent post hoc bias.
3. Record metadata for subgroup analysis.
   1. Record comprehensive non-identifying metadata to clearly define the respondent profile for subgroup analysis. For learners, this must include demographic and academic characteristics such as native language group, age, gender, years of Mandarin study, current HSK proficiency level, academic major, course type, and study site. For the expert panel, explicitly record professional demographics including years of teaching experience, highest academic degree obtained, and specific domain expertise (e.g., literature instruction, educational technology).
      ​CAUTION: Protect personally identifiable information and follow approved data-handling procedures.

7. Compute membership degrees (FCE)

1. Define evaluation grades and numeric scores.
   1. Define a five-grade evaluation set: Excellent, Good, Moderate, Relatively Poor, Poor.
   2. Assign numeric scores for grade-to-index conversion (e.g., 90, 80, 70, 60, 50), and record the scoring scheme in Supplementary File S3³⁸.
2. Specify membership functions and build membership matrices.
   1. Specify membership functions for mapping Likert responses to grade memberships (e.g., trapezoidal functions)^39,40.
   2. Provide explicit parameters for each grade boundary in Supplementary File S3, and state whether the same parameters apply across all indicators.
   3. For each indicator, convert each learner response to a 5-element membership vector, then aggregate across learners (e.g., by averaging memberships) to form the indicator-level membership vector.
   4. Stack all indicator-level membership vectors to construct the indicator-by-grade membership matrix.

8. Perform fuzzy synthesis and derive the sustainability index (SI)

1. Apply a weighted fuzzy operator.
   1. Combine the indicator weight vector with the indicator membership matrix using a specified weighted fuzzy synthesis operator (e.g., weighted average operator) to obtain the comprehensive membership vector^41,42.
   2. Report the operator’s name and formula in Supplementary File S3 to ensure standalone reproducibility.
2. Defuzzify to compute SI on a 0–100 scale.
   1. Convert the comprehensive membership vector to a single SI value by weighted averaging of grade scores (as defined in section 8.1.2).
   2. Output dimension-level SI values by repeating section 9.1–9.2 within each dimension using dimension-specific indicator weights.
   3. Classify SI values using prespecified cutoffs: 85–90 = Excellent; 75–84 = Good; 65–74 = Moderate; 55–64 = Relatively Poor; ≤ 54 = Poor.
      ​PAUSE POINT: Store intermediate vectors, matrices, and calculation sheets (or script outputs) with file hashes in a repository.

9. Optional qualitative validation (mixed methods)

1. Sample participants for interviews.
   1. Sample instructors and learners using a purposive scheme aligned with SI patterns (e.g., high vs. low cultural adaptability) and obtain additional consent for recording.
   2. Use a semi-structured interview guide focused on interpretability of scores, perceived drivers of sustainability, and contextual constraints (Supplementary File S4)⁴³.
2. Code and triangulate findings.
   1. Transcribe recordings verbatim and de-identify transcripts before coding.
   2. Code transcripts using a prespecified coding scheme, and document coder training, inter-coder agreement (if applicable), and resolution rules^44,45.
   3. Triangulate qualitative themes with dimension-level SI results to contextualize strengths and weaknesses without overriding the quantitative procedure.

10. Report and archive (reproducibility package)

1. Report on core outputs.
   1. Report dimension and indicator weights, membership matrices (or summaries), SI values, grade labels, and subgroup comparisons (e.g., site or proficiency groups).
   2. State all exclusion rules and the final analyzed sample sizes (ensure consistency with Abstract/Results).
2. Archive materials and code.
   1. Archive anonymized datasets, scripts, and spreadsheets used for FAHP weighting, membership computation, synthesis, and defuzzification.
   2. Provide software/runtime versions (e.g., programming language version; package versions) and fixed random seeds (if used)^46,47.
3. Run sensitivity analysis (recommended).
   1. Perturb indicator weights by ±10% (one-at-a-time or within dimension) while renormalizing, recompute SI, and summarize SI fluctuations to identify leverage points (Table 2).
   2. Interpret indicators with low weight but high sensitivity as potential “threshold risks,” and report these explicitly.

Results

Dimension weights and key indicators
Using the fuzzy analytic hierarchy process (FAHP), the dimension weights indicated that the technical dimension (T) contributed the largest share to sustainability (0.32), followed by the cultural dimension (C, 0.26) and the pedagogical dimension (P, 0.23), whereas the policy dimension (G) received the lowest weight (0.19)¹⁷. At the indicator level, the technical dimension prioritized functional iteration efficiency (T3, 0.25)

Discussion

This protocol demonstrates that a hybrid FAHP–FCE workflow can operationalize “teaching sustainability” in digitally assisted international Chinese literature courses as a transparent, reproducible index rather than a loosely defined outcome. The results indicate that technical functionality functions as a prerequisite layer for sustainability, with algorithmic interpretability showing the highest sensitivity to sustainability index (SI) fluctuations³¹. Practically, this pattern ...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Learner Questionnaire (19-item)	Self-designed	N/A	Used to collect learners’ evaluations of technology, pedagogy, culture, and policy dimensions; 7-point Likert scale
Expert Pairwise-Comparison Form	Self-designed	N/A	Used to collect experts’ fuzzy pairwise comparisons for FAHP weight calculation
Data-Capture Template & Scripts	Excel / Python	N/A	Used for data storage, membership matrix computation, and Sustainability Index (SI) generation