Lightweight English Text Classification with Deep Learning Based on Complex System Theory

Summary

This study proposes a lightweight graph neural network with meta-distillation and meta-learning to improve English text classification. Enhances generalization in low-data and cross-domain settings while reducing computational cost and maintaining high performance.

Abstract

Globalization and the development of information technology have driven a surge in English text data, and there is an urgent need for efficient classification. To improve the generalization ability of English text classification in small-sample and cross-domain scenarios and to achieve lightweight models, this study combines complex systems theory with deep learning to construct a graph neural network model that fully accounts for the distributional characteristics of text samples. A two-level meta-distillation method, combined with meta-learning strategies, dynamically adjusts the teacher model's parameters and optimizes the entire knowledge-transfer process. The experimental results show that the classification performance of the proposed model in few-shot and cross-domain text classification tasks is significantly superior to that of traditional graph neural networks and other mainstream comparative models. In terms of lightweighting, the SM generated by the two-stage meta-distillation mechanism maintains an extremely high level of performance retention on multi-topic text classification datasets, while significantly reducing computational time. In addition, this method can maintain high efficiency and stable classification performance in long text processing scenarios and offers clear advantages in computational efficiency compared with traditional distillation methods and other methods reported in the literature. The proposed method effectively enhances the classification performance of English text in low-sample and cross-domain application scenarios while significantly reducing the computational cost, thus providing a feasible and efficient technical solution for English text classification in resource-constrained environments.

Introduction

The acceleration of globalization and advances in information technology have led to the explosive growth of English-language text data across fields such as social media, academic research, and business communication. Efficiently classifying these texts for information retrieval, sentiment analysis, content management, and other functions has become an important task in natural language processing¹. The goal of English Text Classification (ETC) is to classify texts into predefined categories based on their semantic content. However, the complexity of natural language, including its ambiguity, context dependency, and diversity of expression, poses many challenges to text classification tasks². Currently, the growth of Deep Learning (DL) technology has created novel opportunities for text classification. Pre-trained language models represented by Bidirectional Encoder Representations from Transformers (BERT) and its variants can learn rich contextual semantic information through pre-training on large-scale corpora, significantly improving the performance of text classification³. However, current ETC methods still face two key limitations: first, they often emphasize optimization of a single model or feature, neglecting the intrinsic properties of language as a complex system, such as its dynamicity and emergent behavior, which leads to inadequate generalization under low-data or cross-domain scenarios; second, despite the strong performance of pre-trained models, their high computational cost and large parameter counts severely impede practical deployment in resource-constrained environments^4,5. To address these issues, this study proposes a lightweight text classification framework integrating Complex Systems Theory (CST) and Two-Stage Meta-Distillation Mechanism (TSMDM).

In the context of this study, CST refers to the theoretical framework that regards natural language as a typical complex system with dynamic evolution, semantic emergence, and heterogeneous distribution of lexical node influence. It is applied to simulate the dominant role of core vocabulary in semantic propagation and the emergent law of overall semantics from local lexical features, thereby enhancing the model's deep understanding of text semantics and adaptability to few-shot and cross-domain data scenarios. Its core contributions are as follows: theoretically, to our knowledge, CST is systematically introduced into text classification tasks, simulating the dynamic evolution and semantic emergence of language systems to enhance the model’s deep understanding and adaptability to few-shot and cross-domain data. Methodologically, a Graph Neural Network (GNN) containing sample distribution characteristics is designed. The combined innovative TSMDM is essentially different from standard knowledge distillation. Standard knowledge distillation realizes one-way knowledge transfer from a fixed-parameter Teacher Model (TM) to a lightweight Student Model (SM). The meta-distillation in this study integrates meta-learning strategies on the basis of distillation and can dynamically adjust the parameters of the TM based on the learning feedback of the SM. The two-stage design additionally sets a teaching assistant model as an intermediate buffer layer to mitigate information loss caused by excessive complexity gaps between the TM and SM, achieving significant model lightweighting while maintaining high accuracy, thereby providing an efficient classification solution for resource-constrained scenarios.

In the field of ETC, researchers have conducted extensive exploration and proposed various solutions that integrate different features and model architectures to address technical challenges in different scenarios. R. Zhang et al. designed a Multilingual Pre-Trained Multi-Feature Fusion Model (MP-MFFM) to address the issue of insufficient capture of long-range contextual structural information in ETC using current DL methods. This method combined multilingual BERT, BiLSTM, and text CNN to extract deep semantic, global, and local structural information and fuse them. This method improved accuracy, sensitivity, and precision on three datasets, verifying its effectiveness⁶. C. Madhu et al. adopted a Dialect Feature-Based Fuzzy Graph Learning Framework (DF-FGLF) to address the text classification needs of unstructured and low-annotated data in social networks, as well as the impact of dialect differences on international English classification. They combined semantic and dialect difference learning features to construct an optimized fuzzy graph. When there were only 10 annotated samples, the F1 value for dialect recognition and text classification exceeded 93% and 80%, which was superior to similar methods and suitable for practical classification⁷. B. Shannaq et al. conducted experiments using English and Arabic datasets to investigate the impact of n-gram length on text classification in different writing systems and the effect of language features on the optimal n-gram length. The performance of English 2-gram was the best (accuracy 0.482, recall 0.489, F1 value =0.472), while Arabic 6-gram was the best (F1 value about 0.85). Language morphology and syntactic features significantly affected the performance of n-gram models⁸. N. S. Lagutina et al. used a text vector constructed based on character, vocabulary, and sentence structure bibliometric features to achieve automatic classification of English short texts for evaluating student learning, combined with standard machine learning classifiers such as SVM. The F1 value of SVM in the Common European Framework of Reference for Languages corpus was 67%, and the F1 value of the best-BERT model (bert-base-cased) was 69%. The errors were mostly in adjacent levels, and the classification quality depended on the corpus⁹.

Knowledge distillation, as an effective means to achieve model lightweighting, improve model performance in small sample scenarios, and reduce computational costs, has also made significant progress in related research. Prior research has explored the application of knowledge distillation to address key natural language processing challenges, including: boosting the performance of small-parameter models under low-label conditions, optimizing multi-lingual text classification tasks, compressing large-scale pre-trained models via hybrid compression strategies, and distilling effective sentence representations to narrow the performance gap between large and small language models while adapting to hardware constraints^10,11,12,13. Despite these advancements, existing knowledge distillation approaches still have critical limitations for ETC: they lack dynamic optimization mechanisms for the knowledge transfer process, often suffer from severe information loss when distilling between highly complex TMs and lightweight SMs, and fail to effectively integrate with the inherent distribution characteristics of text data. These drawbacks result in suboptimal generalization performance in few-shot and cross-domain scenarios. The lightweight models obtained via such methods often struggle to balance classification effectiveness and computational efficiency in resource-constrained environments. This is the key gap that the research in this paper aims to address.

This study tests the following research hypothesis: First, integrating CST into ETC can effectively simulate the dynamic evolution and semantic emergence characteristics of natural language systems. This theoretical integration can significantly enhance the model’s generalization ability in few-shot and cross-domain scenarios compared with traditional text classification models that ignore the complex system properties of language. Second, a GNN model designed with explicit consideration of text sample distribution characteristics can make up for the limitation of traditional GNNs that only focus on instance-level relational modeling, and realize deeper mining of global and local semantic information in English texts. Third, the TSMDM combined with meta-learning strategies and a teaching assistant model can achieve efficient and low-loss knowledge transfer from complex TMs to lightweight SMs. This can significantly reduce the model’s computational cost and parameter scale while maintaining high classification performance. In addition, the lightweight model derived from this mechanism can still retain stable and efficient classification performance in long text processing scenarios with complex semantic structures.

In summary, relevant research has improved text classification performance through multi-feature fusion, dialect feature learning, n-gram optimization, and stylistic metric features, and also achieved model lightweighting through knowledge distillation. However, existing methods still have shortcomings in long-distance information capture, small sample generalization, and adaptation of complex and simple models. To reduce the computational cost of the ETC model while ensuring its accuracy, this study constructs a lightweight model by combining CST and TSMDM to enhance the model's generalization capacity and computational efficiency.

Protocol

To improve the generalization ability of ETC in scenarios of small samples and cross-domains and realize a lightweight model, this study proposes a text classification framework integrating CST and TSMDM. The overall process of this framework is shown in Figure 1.

Figure 1: Visualization of the four-stage lightweight English text classification framework integrating CST and TSMDM. The workflow consists of four stages. Stage 1 performs text representation using BERT-based dynamic embeddings from the input English text. Stage 2 applies graph neural network modeling by constructing a text graph and computing instance-level and distribution-level relationships. Stage 3 models semantic emergence through node centrality reconstruction and a multi-level prototype generation framework to obtain the final category prototype. Stage 4 conducts two-stage elemental distillation, where a teacher model is trained and knowledge is transferred through meta-distillation to produce the final student model. Please click here to view a larger version of this figure.

Firstly, in the text representation and dynamic embedding stage, the BERT model is utilized to dynamically embed the input text and capture the contextual semantic information. The second stage is distribution-aware GNN modeling, which considers the distribution characteristics of samples, constructs a GNN model, and enhances feature representation through dual information interaction at the instance level and distribution level. The third stage is semantic emergence modeling guided by the hybrid system theory, introducing node centrality reconstruction and a three-level prototype generation mechanism to simulate the dynamics and emergence of the language system. Finally, in the fourth stage, a two-level meta-distillation lightweight operation is carried out. The knowledge distillation process is optimized through the assistant model and meta-learning strategy to achieve efficient model compression and acceleration.

ETC model based on distribution characteristics and CST
Model Architecture Overview
The proposed text classification model first employs a pre-trained BERT to perform dynamic contextual encoding of the input text, generating semantically rich word embeddings and a global representation. Next, an instance graph and a distribution graph are constructed from these features: the instance graph captures pairwise sample similarities, while the distribution graph models the overall data distribution; iterative information exchange between the two graphs enables synergistic enhancement of individual features and global structure. Subsequently, CST is integrated to deeply augment the graph network. Node centrality reconstruction uses PageRank to quantify lexical influence, simulating the dominant role of core elements in language networks. This is a widely validated method of measuring the global influence of nodes in complex network topologies and is well-suited to capture the asymmetric semantic spread of core lexical nodes in language systems. A three-level “micro–meso–macro” prototype generation framework emulates the emergent process from local semantics to holistic category representations. Finally, the enhanced feature representations are fed into a classifier to produce the final class probabilities.

Feature Interaction with Distribution-aware GNN
To optimize the generalization ability of ETC, effectively capture complex semantic relationships between texts and Sample Distribution Features (SDFs), this study constructs an ETC model based on distribution features and CST. It achieves deep mining of global and local information in text by integrating mechanisms such as GNN and dynamic text embedding. GNN is a DL model specifically designed for processing graph-structured data. The core principle is to use a message passing mechanism (where each node in the graph aggregates the feature information of its neighboring nodes) and combine it with its own state. Through multiple rounds of iterative updates, it gradually learns a high-dimensional representation that includes the topology structure. This process enables nodes to capture the structure and feature dependencies of the local neighborhood and even the global graph. To overcome the limitations of traditional sequence models in modeling long-range dependencies and global relationships, this study utilizes GNN to capture complex semantic associations and structural relationships between samples. Due to the focus of GNN on the direct correlation information between individual samples, it ignores the overall distribution characteristics of the sample set^14,15,16. Therefore, this study proposes a GNN model that considers SDF. This model introduces BERT for dynamic text embedding and combines it with a prototype network to calculate the feature differences of the samples. BERT is a pre-trained language model based on the Transformer architecture. The core principle is to simultaneously capture the semantic information of each word in the context through a bidirectional encoder, and pre-train it on a large-scale corpus using two tasks: "masked language model" and "next sentence prediction". In masked language models, some words in the input are randomly masked, and the model needs to predict the original word based on the context. The next prediction determines whether the two sentences are consecutive. After pre-training, BERT can quickly adapt to various natural language processing tasks through fine-tuning, significantly improving performance and providing high-quality feature representations for subsequent graph structure construction. The specific structure of the ETC model constructed in this study is shown in Figure 2.

Figure 2: Architectural design of the CST-integrated English text classification model considering sample distribution features. The figure exhibits the architectural design of the English text classification model integrated with complex system theory (CST) and accounting for text sample distribution characteristics. The architecture fuses BERT-based dynamic contextual embedding, distribution-aware graph neural network (GNN) and CST enhancement mechanisms, and realizes in-depth mining of global and local semantic information in English texts through multi-module collaborative modeling. Please click here to view a larger version of this figure.

In this model, for a general dataset, the English text input into the BERT model generates a token sequence through a tokenizer. The sequence construction method is shown in equation (1).

[CLS], a₁, a₂,...a_n, [SEP]   (1)

In equation (1), [CLS] is the classification marker located at the beginning of the sequence. BERT will generate a fixed position hidden state for [CLS] during the training process. This state is directly used for the global semantic representation of the entire text. a₁, a₂,...a_n represents a word or subword in the text. [SEP] is a delimiter used to mark the end of a sentence. After Bert’s processing of the above sequence, the features of each token are obtained, providing structured contextual information for the model. For special datasets containing entities, if conventional sequence construction methods are used, the positional information contained by the entities will be lost. Therefore, this study constructs the sequence using the method shown in equation (2).

 (2)

In equation (2), [E1], [/E1], [E2], and [/E2] are the tokens set to determine the starting and ending positions of two entities. Similarly, after Bert processing, the output features of these tokens will carry semantic information of the corresponding entities, thus better utilizing the important information of entity location. Subsequently, this study uses a GNN model considering SDF to enhance the distribution and instance features of the samples. The structure of the model is shown in Figure 3.

Figure 3: Dual-graph feature interaction architecture of the distribution-aware graph neural network model. The figure presents the dual-graph feature interaction architecture of the distribution-aware GNN model for English text classification. The architecture builds a point graph for instance-level similarity encoding and a distribution graph for distribution-level similarity encoding, and achieves feature enhancement via iterative information interaction between the two graphs, completing the cross-level information cycle of instance features and distribution features. Please click here to view a larger version of this figure.

The model achieves feature enhancement through a two-stage interaction mechanism. Firstly, it parses distribution features from instance-level associations of data samples, and then dynamically optimizes instance features through information exchange at the distribution level. By iteratively strengthening the features of instances and distribution levels, it ultimately completes the classification task. Specifically, the model uses point graphs and distribution graphs to achieve information exchange. The point map uses the text sample feature vector (updated by the last two layers of BERT) as a node, quantifies the sample feature difference to calculate the edge weight through a dedicated encoding network (one sigmoid layer + two convolutional batch normalization layers), and uses min-max normalization to normalize to the [0,1 interval. An empirical similarity threshold of 0.2 is set for edge thresholding, where edges with weights below this threshold are pruned to eliminate weak instance-level correlations. The distribution graph takes the fused text distribution features as nodes, with edge weights computed based on cosine similarity of distribution feature vectors and normalized via L2 normalization to ensure metric consistency. An edge threshold of 0.15 is adopted, and edges with weights below this value are removed, while the remaining valid edge weights are further mapped and standardized by a multi-layer perceptron to enhance the discriminability of distribution-level associations. This study defines a point graph for the l-layer iteration, where node represents sample features and edge encodes instance level similarity. Distribution map , node represents sample features, and edge encodes distribution level similarity. Taking the l-th to l+1-th wheel as an example, instance level relationship calculation calculates point similarity through feature differences, as shown in equation (3).

(3)

In equation (3),  and   are the edge weights between nodes i and j during the l-th and l-1-th iterations of the point graph, reflecting the similarity of sample features. is an encoding network used to convert node feature differences into edge weight scales, consisting of one layer of sigmoid and two layers of convolution batch normalization activation functions. When   and are the l-1-th iteration, the eigenvectors of nodes i and j are updated by Bert in the last two layers. Afterwards, the distribution characteristics are calculated based on the instance relationships, as shown in equation (4).

 (4)

In equation (4), is the eigenvector of node i in the l-th layer distribution graph. MLP₁ is a multi-layer perceptron composed of activation functions and fully connected layers, which maps the concatenated composite features into new distribution features. Afterwards, the distribution relationship is calculated based on the distribution characteristics, and the instance features are calculated from the distribution relationship to complete the cross-level information loop.

Complex Systems Enhancement Mechanisms
To further improve the generalization capability, CST is applied to the above model in this study. The dynamic evolution of complex systems is manifested by the heterogeneous distribution of node influence. To simulate the dominant role of core vocabulary in semantic propagation in language systems, this study introduces the node centrality index to reconstruct the information propagation mechanism. The point graph H_p constructed for each scenario task uses the PageRank algorithm to quantify the centrality of node C(h_i), as shown in equation (5)¹⁷.

    (5)

In equation (5), α is the damping coefficient, α = 0.85. N(h_i) represents the set of neighboring nodes, and L(h_j) is the node degree. Equation (5) replaces the cascading propagation process of vocabulary influence in simulated language networks, allowing core vocabulary (such as verbs and topic words) to gain higher centrality. Afterwards, the node centrality is coupled with edge weights to dynamically adjust the propagation strength of information between nodes. The coupling function λ_ij is shown in equation (6).

    (6)

In equation (6), σ is the Sigmoid activation function, W^T is the learnable weight vector, and b means the bias term. The coupling of centrality and edge weights dynamically balances local relationships with global importance, thereby enhancing the model’s adaptability to dynamic changes in tasks. The emergence of CST is manifested in the English language as the overall semantics surpassing the sum of local vocabulary¹⁸. To utilize this feature in ETC, this study designs a three-level prototype generation framework to simulate the emergence process from micro features to macro categories, as shown in Figure 4.

Figure 4: Stepwise construction of the micro-meso-macro three-level prototype generation framework for linguistic semantic emergence. The figure illustrates the step-by-step construction of the micro-meso-macro three-level prototype generation framework for simulating linguistic semantic emergence in English text classification. The framework builds hierarchical text category representations through micro subclass clustering with K-means, meso subclass prototype fusion based on distribution similarity weighting, and macro nonlinear integration via attention mechanism, simulating the emergent characteristic of "the whole is greater than the sum of its parts" in language systems. Please click here to view a larger version of this figure.

The above process constructs text category representations step by step from micro to macro to lift the model's generalization ability. At the micro level, K subclasses are generated by clustering the sample embedding representations of each text category. K-means is utilized to divide the samples within the category into subgroups, and the centroid position of each subgroup is calculated as the subclass prototype¹⁹. In the mesoscopic level, the distribution similarity of each subclass prototype is calculated, a similarity matrix is generated, and then the subclass prototypes are reassembled based on similarity weights to quantify the correlation between subclasses. The calculation of subclass similarity weight w_ij is shown in equation (7).

(7)

In equation (7), is the Jensen Shannon divergence, used to measure the difference in distribution between two subclasses and ²⁰. Finally, the subclass prototypes are weighted and fused to obtain a set of class distribution representations . At the macro level, the importance weights of each subclass are learned through attention mechanisms, and nonlinear transformations are introduced to simulate the emergence process, resulting in the final class prototype c_final as shown in equation (8).

  (8)

In equation (8), α_k means the attention weight of the k-class. U denotes the non-linear transformation weight matrix. tanh(·) is used to enhance nonlinear representation and simulate the overall greater than the sum of parts characteristic of complex systems. Each level captures the diversity within categories through a subclass structure, weights distribution similarity to reduce the influence of noisy subclasses, and dynamically focuses on discriminative features through attention mechanisms to enhance the model's generalization ability. CST is mainly integrated with GNN through two mechanisms. The first is to introduce node centrality to reconstruct the information dissemination process and simulate the heterogeneous distribution of lexical influence in the language system. The node centrality is quantified through the PageRank algorithm and dynamically coupled with the edge weights, enabling the core vocabulary to dominate in semantic propagation and enhancing the model's adaptability to the dynamic changes of tasks. The second is to design a three-level prototype generation framework, from micro-subclass clustering to macro-category prototype fusion, to simulate the emergent characteristic in the language system that "the whole is greater than the sum of its parts". This framework achieves hierarchical integration from local features to global semantics through distribution similarity weighting and attention mechanisms.

In summary, the core innovation of the constructed ETC model lies in deeply integrating CST's ideas into the framework of GNN. By reconstructing the information dissemination mechanism through node centrality, the model simulates the leading role of the core elements in the language system and enhances its adaptability to task dynamics. Through the three-level prototype generation framework, the model realizes the emergence process from local features to global semantics, thereby enhancing the model's ability to capture the diversity and discriminative features within the category. This method avoids the limitation of traditional GNNs that only rely on instance-level relationships. Through distribution-level interaction and complex system characteristics, it provides a new solution for improving the generalization ability of the model in few-shot and cross-domain scenarios.

The emergent properties of CST correspond to a three-level prototype generation framework. In language systems, the principle that “the whole is greater than the sum of its parts” manifests as a semantic leap from local word meanings to overall text categories. This study simulates this process through a “micro-meso-macro” three-level prototype generation mechanism: at the micro level, sample embeddings are clustered to form sub-class prototypes; at the meso level, these prototypes are weighted and fused based on distributional similarity; and at the macro level, final category prototypes are synthesized non-linearly via an attention mechanism. For instance, in sentiment classification, multiple negative words like “disappointing,” “slow,” and “expensive” blend and transform non-linearly to emerge as a strong overall sentiment of “negative evaluation.” Node centrality and heterogeneity in CST align with a node centrality reconstruction-based information propagation mechanism. Within language networks, the influence of words is unevenly distributed, with core words (e.g., verbs, thematic terms) playing dominant roles in semantic propagation. This study quantifies node centrality using the PageRank algorithm and dynamically couples it with edge weights to adjust the intensity of information spread between nodes. For example, in news categorization, the term “election” has high centrality in political texts, leading adjacent nodes such as “vote” and “campaign” to gain higher weights during information aggregation, thereby enhancing the semantic representation of that topic. The dynamic and adaptive nature of CST corresponds to distribution-aware GNNs and a teaching experiment mechanism within meta-distillation. Language systems evolve dynamically according to context and task requirements. Models capture overall distribution changes in datasets through distribution-aware GNNs. Simultaneously, an instructional experiment mechanism driven by meta-learning is introduced in TSMDM, allowing TMs to dynamically adjust parameters based on feedback from SMs. For example, in cross-domain sentiment analysis, the model can quickly adapt feature weights based on the target domain’s data distribution and refine the TM’s parameters during meta-distillation to improve adaptation efficiency for new domains.

LTC model based on TSMDM
Two-Stage Meta-Distillation Pipeline
To address the challenges of high computational cost and deployment difficulties in resource-constrained environments, a lightweight text classification model based on a TSMDM is proposed. This meta-distillation method executes the distillation process in a strict sequential order with two distinct stages. The second stage is initiated only after the completion and convergence of the first stage’s training: 1) the Teacher-to-Teaching Assistant (TA) knowledge compression stage, and 2) the TA-to-Student lightweight distillation stage integrated with meta-learning parameter adaptation. This approach achieves efficient knowledge transfer from a complex TM to a lightweight SM through two-stage distillation, while incorporating a meta-learning mechanism to dynamically optimize the knowledge transfer strategy during the process^21,22. The overall pipeline is illustrated in Figure 5.

Figure 5: Pipeline design of the lightweight text classification model with two-stage meta-distillation mechanism. The figure designs the pipeline of the lightweight text classification model with the two-stage meta-distillation mechanism (TSMDM). The pipeline comprises a teacher model (high complexity knowledge source), a teaching assistant model (intermediate complexity buffer layer) and a student model (lightweight target model), and implements efficient knowledge transfer through two sequential stages: teacher-to-teaching assistant knowledge compression and teaching assistant-to-student lightweight distillation integrated with meta-learning. Please click here to view a larger version of this figure.

The method involves three roles: a TM, a TA model, and a SM, with the distillation process divided into two stages: teacher-TA distillation and TA-student distillation. In the first Teacher-to-TA knowledge compression stage, knowledge from the TM is first compressed into an intermediate-complexity TA model to mitigate information loss caused by excessive capacity gaps in direct distillation. The TM, serving as the knowledge source, transfers both its output classification probabilities and intermediate-layer features to the TA model. The TA model is trained by aligning its outputs and feature representations with those of the teacher, using a combined loss function that integrates classification loss and feature alignment loss²¹. In the second TA-to-Student lightweight distillation stage, which is launched upon the convergence of the TA model, knowledge is further distilled from the TA model to the final lightweight SM. This stage also employs dual supervision (classification logits and intermediate features) and incorporates a meta-learning mechanism: the TM conducts “teaching experiments” on auxiliary tasks to evaluate the student’s learning state and dynamically adjusts its own parameters to deliver more adaptive and targeted guidance. The SM completes training by minimizing both the output discrepancy and feature distance relative to the TA model, thereby achieving significant parameter reduction while preserving classification performance to the greatest extent possible²².

Meta-Learning with Teaching Experiments
In traditional knowledge distillation methods, the trained TM guides the SM by participating in loss calculation. However, fixed parameter TMs are difficult to evaluate the actual learning status of SMs, nor can they quantify the specific contribution of their guidance to updating SM parameters^23,24. Therefore, this study introduces meta learning ideas into the distillation process, enabling the TM to adjust its own parameters based on the learning feedback of the SM. The distillation process is displayed in Figure 6.

Figure 6: Iterative parameter optimization process of meta-distillation combined with teaching experiment mechanism. The figure depicts the iterative parameter optimization process of meta-distillation combined with the teaching experiment mechanism for English text classification model lightweighting. The process generates a temporary inner learner from the student model, quantifies the teaching effect loss through the inner learner’s simulated training performance, and enables the teacher model to adjust its own parameters via the backpropagation of the loss, thus optimizing the subsequent knowledge transfer strategy. Please click here to view a larger version of this figure.

The TM optimizes parameters through conventional classification tasks during its own training phase, resulting in a basic classification loss L_T that reflects its original performance. After completing the training, the SM S is replicated to generate a temporary copy of the internal learner S₁. The TM performs teaching experiments on S₁, and the comprehensive performance error exhibited by S₁ in this simulated training is quantified as loss L_Q. This signal is used as feedback to the TM for evaluating teaching effectiveness. Through the backpropagation of L_Q, the TM actively adjusts its own parameters and optimizes its knowledge transmission method. After completing the meta learning optimization, the TM performs formal knowledge distillation on the original SM S, and also uses the model loss L_S as feedback for teaching effectiveness evaluation to the TM. To achieve lightweighting of the ETC model, this study will consider incorporating the SDF GNN model as a TM into the teaching experiment process, as shown in Figure 7.

Figure 7: Parameter update flow of the teaching experiment mechanism for teacher model optimization in meta-distillation. The figure shows the parameter update flow of the teaching experiment mechanism for teacher model optimization in the meta-distillation process. The flow first calculates the soft loss between the inner learner’s prediction and the teacher model’s prediction with the mean square error loss function, combines the soft loss with hard label error to form the total training loss, and updates the teacher model’s parameters through the backpropagation of the weighted total error to improve its teaching effectiveness. Please click here to view a larger version of this figure.

After completing knowledge distillation training, the SM first calculates the difference between the predicted results of the TM and the true labels. Subsequently, the MSE loss function is used to measure the distance between the SM prediction (inner learner S₁) and the TM prediction. This distance value is used as the soft loss²⁵. The final training objective consists of a weighted combination of hard label error and soft loss. By backpropagating the weighted total error, the parameters of the TM are updated, thereby improving the teaching effectiveness of the TM. The parameter calculation of the internal learner S₁ is shown in equation (9).

(9)

In equation (9), and are the internal learner parameters and their initial parameters influenced by the TM parameter γ_T. λ (the learning rate, λ = 0.005) controls the parameter update step size for inner learners (i.e., copies of SMs) during meta-distillation²⁶. Equation (9) calculates loss gradients through backpropagation, while λ updates the parameters of inner learners. This mechanism reflects the learning characteristics of SMs, enabling TMs to receive feedback and adjust their teaching strategies, thereby enhancing subsequent knowledge distillation effectiveness. The calculation of loss L_S in meta learning is shown in equation (10).

(10)

In equation (10), B is the meta-learning sample sequence.

Optimizing Knowledge Transfer with Loss Design and Assistant Model
To further enhance the effectiveness of knowledge distillation, this study obtains the overall loss by calculating the classification loss and feature loss. Among them, the feature loss adopts a retrospective mechanism, using the shallow and current feature outputs of the TM to guide the SM, as expressed in equation (11).

(11)

In equation (11), I is the set of feature layer indices. D is a distance function taken to calculate the difference between two feature representations. and are objective representation functions in the TM and SM, which convert the feature outputs and of the i-th and j-th layers into attention matrices. The classification loss combines the cross-entropy loss between the SM output and the true label, as well as the MSE between the two models output, as the soft loss^27,28. Ultimately, the overall loss is obtained by weighting the two, helping the SM better receive guidance from the TM. To minimize the performance loss during the knowledge distillation process, this study places the teaching assistant model between two models, as exhibited in Figure 8.

Figure 8: Two-stage knowledge distillation processing flow with teaching assistant model as intermediate buffer layer. The figure demonstrates the two-stage knowledge distillation processing flow with the teaching assistant model as the intermediate buffer layer. The first stage fuses feature loss and classification loss to construct the total distillation loss, and trains the teaching assistant model with the teacher model’s knowledge; the second stage adopts the same loss fusion strategy to distill the teaching assistant model’s knowledge to the student model, reducing information loss caused by excessive complexity gaps between the teacher and student models. Please click here to view a larger version of this figure.

After completing the meta distillation stage, the TA model and the TM undergo conventional knowledge distillation. During this process, the features of both are synchronously extracted and the corresponding loss L_a is calculated. Subsequently, this feature loss is fused with the classification loss L_M1 of the model to form the distillation loss function in the first stage, as shown in equation (12).

(12)

In equation (12), β is the weight coefficient used to control the proportion of feature loss and classification loss in the total loss. and are objective representation functions in the teaching assistant model and the TM, which convert the feature outputs and of the i-th and j-th layers into attention matrices. z_T and z_M are the outputs of the TM and the assistant model. The distillation process from the teaching assistant model to the SM is the same as the above process, which minimizes the loss of the SM through multiple iterations to complete knowledge transfer.

In conclusion, the core contribution of the proposed TSMDM lies in optimizing the process of knowledge transfer through the meta-learning mechanism. Compared with traditional distillation, the core advantage of this method lies in its dynamic teaching ability: the TM can adjust its own parameters through the teaching experiment mechanism based on the real-time feedback of the SM, thereby providing more targeted guidance. Meanwhile, the teaching assistant model is introduced as a buffer layer, effectively bridging the complexity gap between the TMs and SMs and reducing information loss in knowledge transfer. The collaborative design of this "meta-learning optimization teaching strategy" and "two-level buffering to reduce transfer difficulty" enables the final SM to achieve significant lightweighting while retaining the core knowledge distilled from the complex TM to the greatest extent.

Data Availability
The datasets generated during and/or analyzed during the current study are provided in Supplementary File 1.

Results

Experimental setup and dataset:
To comprehensively evaluate the proposed model's performance and lightweight efficiency in small-sample cross-domain classification tasks, experiments are conducted on four datasets: FewRel (small-sample relational classification), ARSC (cross-domain sentiment analysis), Reuters-21578 (multi-topic news categorization), and ArXiv (long-form academic abstracts). FewRel, a widely-used small-sample relational classification dataset sourced from Wikipedia, contains 100 sema...

Discussion

To enhance the generalization ability of ETC under few-shot and cross-domain scenarios while reducing computational costs, this study proposes a lightweight text classification framework integrating CST with TSMDM. Integrating CST with a TSMDM framework addresses the core limitations of existing ETC methods (poor few-shot/cross-domain generalization and high computational costs) by infusing linguistic system dynamics and emergent semantic properties into model design and enabling efficient knowledge transfer for lightwei...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Central Processing Unit (CPU)	Commercial hardware vendors (Intel, Dell, Lenovo)	Intel i5-7300HQ	Hardware Specifications
Graphics Processing Unit (GPU)	Commercial hardware vendors (NVIDIA, ASUS)	NVIDIA GeForce RTX 3060, 12 GB VRAM	Hardware Specifications
Random Access Memory (RAM)	Commercial hardware vendors	16 GB DDR4	Hardware Specifications
Solid State Drive (SSD)	Commercial hardware vendors	1 TB NVMe SSD	Hardware Specifications
Operating System	Microsoft Official Website	Windows 10 (64-bit)	System Software
Python	Python Official Website (python.org)	Python 3.8	Programming Language
PyTorch	PyTorch Official Website (pytorch.org)	PyTorch 1.11.0	Deep Learning Frameworks & Core Libraries
CUDA	NVIDIA Official Website	CUDA 11.3	Deep Learning Frameworks & Core Libraries
Hugging Face Transformers	Hugging Face Hub (huggingface.co)	Stable version (adapted to Python 3.8/PyTorch 1.11.0)	Deep Learning Frameworks & Core Libraries
NumPy	PyPI (pypi.org)	Stable version (adapted to Python 3.8)	Data Processing & Statistical Libraries
Pandas	PyPI (pypi.org)	Stable version (adapted to Python 3.8)	Data Processing & Statistical Libraries
Scikit-learn	PyPI (pypi.org)	Stable version (adapted to Python 3.8)	Data Processing & Statistical Libraries
SciPy	PyPI (pypi.org)	Stable version (adapted to Python 3.8)	Data Processing & Statistical Libraries
Matplotlib	PyPI (pypi.org)	Stable version (adapted to Python 3.8)	Visualization Library
AdamW	PyTorch built-in	Integrated in PyTorch 1.11.0	Optimizer
BERT	Hugging Face Hub (huggingface.co)	BERT-base (bert-base-cased)	Pre-trained Models
FewRel	FewRel Official Repository (GitHub) / Wikipedia	100 semantic relationship categories, 700 sentences per category; train/validation/test split (5:2:3)	Datasets
ARSC	Public cross-domain sentiment analysis datasets (GitHub/ACL Anthology)	23 Amazon product categories, 69 binary sentiment analysis tasks; train/validation/test split (4:2:3)	Datasets
Reuters-21578	UCI Machine Learning Repository / Reuters Official Archive	21,578 news articles, 90 thematic categories; ModApte partitioning method	Datasets
ArXiv	ArXiv Public API (arxiv.org)	Computer science paper abstracts; train/validation/test split (7:2:1)	Datasets
Tokenizer	Hugging Face Hub (huggingface.co)	BERT-base-cased Tokenizer	Other Essential Tools
Git	Git Official Website (git-scm.com)	Latest stable version	Other Essential Tools