Two-Stage Recruitment Text Labeling via Lexicon-Guided Routing and Retrieval-Augmented Large Language Models

Summary

A two-stage workflow is described for classifying recruitment texts into Artificial Intelligence, Environmental Protection, or Other. Lexicon-guided triage assigns routine cases, while retrieval-augmented, prompt-optimized large language model inference resolves ambiguous cases, enabling accurate large-scale labeling and downstream descriptive labor-market summarization.

Abstract

Recruitment texts are heterogeneous, and domain terminology evolves over time, making large-scale labeling difficult with limited manual annotation capacity. This protocol provides a reproducible two-stage workflow for classifying Chinese recruitment texts into Artificial Intelligence, Environmental Protection, or Other. Stage one performs lexicon-guided triage using two curated domain keyword lists. Postings that match only one domain lexicon are directly labeled. Postings that match neither lexicon are labeled as Other, while dual-match postings are routed to stage two. Stage two applies retrieval-augmented large language model inference. A knowledge base compiled from 12 authoritative domain documents is converted to text, split into about 1,000-character chunks with 20-character overlap, embedded using all-MiniLM-L6-v2, and indexed in LanceDB. For each uncertain posting, cosine-similarity k-nearest-neighbor retrieval returns candidate chunks filtered by a minimum similarity threshold (τ), and up to four chunks are injected into a fixed prompt template that defines label boundaries and constrains the output to a single label. A benchmark set of 3,000 postings is manually verified, with 1,000 postings per class, and achieves an inter-annotator agreement of 95.0 % and a Cohen's kappa (κ) of about 0.93. Evaluation compares multiple open-source large language models across a prompt-only setting and a retrieval-augmented setting using Qwen2.5-7B-Instruct. The retrieval-augmented configuration achieves 98.47 %accuracy and supports corpus-scale labeling of about 6.93 million postings for downstream descriptive aggregation.

Introduction

In recent years, with the rapid development of AI and environmental technologies, a large number of emerging careers have emerged. On the one hand, the global job market is flooded with many brand-new positions based on AI and sustainability. Vacancy-based evidence documents a rapid expansion in AI-related skill demand, which increases the heterogeneity of recruitment texts and motivates a reproducible, scalable domain-tagging protocol¹. On the other hand, China's latest revision of the Occupational Classification Dictionary (OCD) has seen a similarly robust growth in new occupations in the fields of employment informatization and green and low-carbon, with a net increase of 158 new occupations in the 2022 revision compared to the 2015 edition of the Dictionary, with a total of 97 digital occupations, or 6% of the total number of occupations, labeled, along with 134 green occupations, or 8% of the total number of occupations².

As these new occupations continue to emerge, the content and form of job postings by companies are becoming more complex, as job descriptions often involve subject knowledge and diverse skill requirements, containing both structured fields such as job titles and skill lists, as well as a large number of unstructured free-text descriptions, resulting in increasingly complex job posting structures. Such complex job postings typically contain clearly defined structured elements (e.g., job titles, lists of required skills) alongside extensive unstructured free-text descriptions. For example, an AI engineer job ad might list technical jargon like ‘proficiency in a deep learning framework’ and ‘experience optimizing backpropagation algorithms’ as part of a skill list, while also including a detailed narrative of responsibilities; similarly, an environmental engineer posting might reference specific regulatory standards and certifications. This combination of structured and unstructured content results in highly specialized and convoluted job descriptions.

Artificial Intelligence and Environmental Protection are selected to evaluate the protocol under realistic cross-domain ambiguity in recruitment classification. In practice, mainstream occupational resources such as O*NET and job boards assign coarse tags, making cross-industry roles difficult to distinguish using keywords alone. Environmental Protection represents a broad domain overlapping with multiple scientific fields, while Artificial Intelligence reflects a finer-grained slice within computing often conflated with general software roles. This pairing captures both broad and narrow cross-domain settings with distinct terminology and suitable authoritative documents for knowledge-base construction, while allowing adaptation to other industries by replacing the domain corpus without altering the core workflow.

Recent years have seen a surge in research on job posting classification. Researchers are increasingly leveraging large pre-trained models to improve the classification of occupations. BERT’s introduction in 2019 marked a breakthrough that enabled deeply contextual language understanding and significantly improved text classification performance³. For instance, Clavié et al. (2023) explored using an instruction-tuned large language model(LLM) for job classification, finding that proper prompt engineering enabled the LLM to outperform state-of-the-art supervised models on a graduate job classification task⁴. Similarly, Li et al. (2023) proposed an LLM4Jobs approach that combines large-language-model summarization with occupation-code matching, significantly improving classification accuracy on lengthy job descriptions by first extracting summaries and then aligning them with standard job codes⁵. In addition, a 2024 survey by Senger et al. catalogues recent advances in deep learning for HR, noting that skill extraction and job classification have become core tasks in computational job market analysis⁶. Kavas et al. (2024) enhanced job posting classification by combining multilingual text embeddings with LLM-based categorization, achieving notable accuracy gains⁷. On the traditional front, earlier systems ranged from keyword-based heuristics with relatively coarse category sets to taxonomy-driven frameworks such as Carotene, which used a hierarchy over more than 4,000 job titles^8,9. Javed et al. (2016) presented an early job title classification framework, marking one of the first steps toward systematic occupation categorization¹⁰. These supervised approaches, however, require extensive labeled data and struggle to adapt to the rapid emergence of new job roles, as classification taxonomies often evolve more slowly than the labor market⁴.

To address such limitations, recent work has turned to hybrid strategies that incorporate external knowledge; for example, Lewis et al. (2020) introduced the Retrieval-Augmented Generation (RAG) framework, which combines pre-trained language models with a retriever to inject relevant domain knowledge, achieving superior accuracy and more factual outputs on knowledge-intensive tasks¹¹. Lester et al. demonstrated that prompt tuning yields larger performance gains as model size increases¹², reinforcing the use of a strong base LLM. For example, Han et al. showed that using rule-guided prompts can boost text classification accuracy by focusing the model on key features¹³. Furthermore, Brown et al. (2020) famously showed that a large language model can perform new tasks from only a few examples or instructions, highlighting the power of prompt-driven few-shot learning¹⁴. Notably, a recent survey of prompt-based NLP techniques underscores that prompt wording can significantly steer model outputs¹⁵. At the same time, macro-level shifts and uncertainty in the labor market reinforce the need for scalable and update-friendly labeling protocols that can be refreshed as new roles emerge¹⁶. In the job-market NLP domain, taxonomy-driven multilingual pre-training has also been explored to better align representations with occupational structures and cross-lingual variability¹⁷. Collectively, these studies inform the approach and demonstrate the potential of using large language models with retrieval and prompt optimization to more effectively recognize and adapt to emerging occupation contexts.

In this study, domain knowledge is curated only for Artificial Intelligence and Environmental Protection, as corpus construction, validation, and maintenance represent the primary operational costs of cross-domain recruitment classification. Accordingly, Other serves as an out-of-scope rejection category rather than a substantive industry class. The reported high accuracy should be interpreted cautiously, as the three-label setting simplifies labeling and may inflate performance compared with finer-grained taxonomies. The protocol is intended as a lightweight, knowledge-grounded labeling layer for targeted domains, not as a replacement for comprehensive multi-category classification systems. Expanding the label set may reduce accuracy due to increased overlap, ambiguity, and stricter corpus-coverage requirements. In practice, the rejection set can be progressively refined by extending the domain corpus and incorporating internal knowledge bases. The three-label configuration, therefore, demonstrates a reusable paradigm rather than an exhaustive occupational taxonomy.

The dataset combines structured and unstructured sources. The structured corpus was collected and curated by the authors from job advertisements publicly posted by publicly listed companies on major online recruitment platforms in China between 2014 and 2023. After deduplication and basic cleaning, the structured corpus contains approximately 6.93 million postings. Unstructured domain documents released by relevant authorities were used to define domain knowledge and labeling criteria. From these sources, three benchmark subsets were manually constructed, each containing 1,000 postings for Artificial Intelligence, Environmental Protection, and unrelated fields, respectively, and verified for evaluation.

Model backbones are evaluated using accuracy, precision, recall, and F1 (the harmonic mean of precision and recall, F1 = 2PR/(P+R)) to select the optimal foundation for the retrieval-augmented workflow. Authoritative domain documents are compiled into a knowledge base for retrieval-augmented generation (RAG). Relevant passages are retrieved and injected into a fixed prompt template to support classification. Prompt variants are systematically tested, and a cue-word optimization strategy is applied to determine the final configuration. Retrieved evidence is filtered for relevance to minimize noise and support accurate, efficient inference.

To enable large-scale classification, the workflow adopts a staged pipeline: lexicon-guided triage efficiently resolves routine cases, while retrieval-augmented, prompt-optimized LLM inference handles ambiguous postings, balancing scalability and accuracy (Figure 1).

Protocol

This study did not involve human participants or animal subjects. Therefore, ethical approval and informed consent were not required. The workflow was executed in a Python 3.10 environment on a 64-bit Windows operating system using the hardware, tools, and software listed in the Table of Materials.

1. Modeling

1. Constructing the data and knowledge base
   1. Collect and curate an in-house structured corpus of job advertisements for publicly listed companies from major online recruitment platforms in China (2014–2023), ensuring compliance with platform policies and applicable regulations. For each posting, record the job title, job description, company name, posting date, and a unique identifier (e.g., posting ID or URL, if available). Remove duplicates and invalid entries, and verify that the final corpus contains approximately 6.93 million records.
   2. Collect authoritative domain documents related to Artificial Intelligence and Environmental Protection, including the documents listed in Supplementary File 1. Ensure that the documents define competencies, qualification standards, task scopes, or regulated procedures.
2. Constructing the benchmark dataset
   1. Screen the structured dataset manually. Select postings that clearly represent Artificial Intelligence, Environmental Protection, and unrelated fields. Include borderline cases to improve coverage.
   2. Label each posting using job title, job description, and employer information.
   3. Construct three benchmark subsets containing 1,000 postings each for Artificial Intelligence, Environmental Protection, and Other.
   4. Perform dual annotation. Assign one annotator to label each posting and a second annotator to verify the label using a written guideline.
   5. Resolve disagreements through discussion and adjudication by a third annotator.
   6. Compute inter-annotator agreement. Record the percentage agreement and κ.
   7. Preserve the verified benchmark dataset for model evaluation.
3. Evaluating model backbones
   1. Evaluate the instruction-tuned LLM backbones using the same prompt template and benchmark dataset.
   2. Disable retrieval augmentation during backbone comparison.
   3. Keep prompt wording, decoding parameters, and label mapping identical across models.
   4. Compute overall accuracy, per-class precision, recall, and F1.
   5. Select the best-performing backbone and record its configuration in Table 1.
   6. Report the complete evaluation results in Table 2.
4. Performing domain-adaptive encoder training
   1. Build an unlabeled corpus using only the authoritative documents listed in Supplementary File 1.
   2. Extract plain text from all documents.
   3. Segment text into sequences with a maximum length of 512 and a stride of 492.
   4. Discard segments shorter than 50 characters.
   5. Ensure that benchmark postings are excluded from this corpus.
   6. Initialize the Chinese BERT base checkpoints.
   7. Perform masked language model pretraining with masking probability 0.15.
   8. Train for 3 epochs using AdamW with learning rate 5e-5 and batch size 2.
   9. Save the pretrained checkpoint.
   10. Fine-tune the encoder for three-class classification using a learning rate of 2e-5, a batch size of 2, and a maximum sequence length of 512.
   11. Fix the random seed to 42 and apply stratified train-validation splitting.
   12. Report accuracy, macro-averaged precision, macro-averaged recall, macro-averaged F1, per-class metrics, and the confusion matrix.
   13. Save evaluation outputs for verification.
5. Constructing the Retrieval-Augmented Classification pipeline
   1. Build the knowledge base
      1. Compile 12 authoritative domains.
      2. Remove duplicate or outdated versions.
      3. Convert documents to plain text.
      4. Record document metadata, including issuer and publication year.
      5. Split text into chunks of approximately 1,000 characters with 20-character overlap.
      6. Record the total number of chunks and the chunk-length distribution.
         NOTE: Revalidate retrieval performance if the knowledge base is expanded.
   2. Encode and store embeddings
      1. Encode all chunks using the embedding model and store embeddings in the vector database.
      2. Archive chunk identifiers and provenance metadata.
      3. Verify that the number of stored vectors matches the chunk count.
   3. Perform retrieval.
      1. Embed each job posting using the same embedding model.
      2. Perform k-nearest neighbor search using cosine similarity.
      3. Apply similarity threshold τ to filter low-relevance matches.
      4. Fix τ and top-κ after tuning on a held-out subset.
      5. Record retrieved chunk identifiers and similarity scores.
   4. Perform Retrieval-Augmented Inference
      1. Insert up to four retrieved chunks into the evidence section of the prompt template (Supplementary File 2).
      2. Concatenate the job posting text with retrieved evidence.
      3. Generate the final three-class label using the selected LLM.
      4. Save retrieval logs, prompts, and model outputs.
6. Performing lexicon-guided triage
   1. Construct Keyword Lexicons
      1. Compile two keyword lexicons: one for Artificial Intelligence and one for Environmental Protection (Supplementary File 3).
      2. Extract candidate terms from job postings and authoritative documents.
      3. Tokenize text using the specified tokenization library.
      4. Compute term frequency and domain contrast statistics and remove ambiguous or overly generic terms.
      5. Finalize and archive the lexicons.
   2. Route postings
      1. Apply exact string and token-level matching.
      2. Route postings containing only Artificial Intelligence keywords to the Artificial Intelligence branch.
      3. Route postings containing only Environmental Protection keywords to the Environmental Protection branch.
      4. Label no-match postings as Other.
      5. Route dual-match postings to the retrieval-augmented step.
      6. Record routing statistics and lexicon versions.
   3. Classify ambiguous postings
      1. Retrieve relevant chunks under fixed top-κ and τ settings.
      2. Inject retrieved evidence into the prompt template.
      3. Generate the final label and save per-instance logs.

2. Reclassification of results

1. Thesaurus construction
   1. Construct a thesaurus of Artificial Intelligence terms (Supplementary File 4). Include terms from machine learning, natural language processing, computer vision, robotics, and related subfields. Organize detailed terms under each category.
   2. Construct a separate thesaurus of Environmental Protection terms. Include environmental science fundamentals, environmental monitoring and analysis, pollution control and management, and environmental planning and management.
   3. Add all Artificial Intelligence and Environmental Protection terms to the tokenization dictionary. Ensure that these terms are recognized as complete units during text segmentation.
2. Preprocessing text
   1. Remove punctuation marks and special characters using regular expressions. Retain only text and spaces.
   2. Perform word segmentation to split continuous text into individual tokens.
3. Performing feature matching and categorization
   1. Preprocess the input text to obtain a list of segmented tokens.
   2. Select the appropriate thesaurus according to the preliminary classification.
   3. Traverse the segmented tokens and perform exact matching against thesaurus terms after normalization. Apply lowercasing and unify predefined variants before matching.
   4. Record each matched term and its corresponding thesaurus branch.
      ​NOTE: If multiple branches match, resolve ties using a fixed rule (e.g., highest match count followed by earliest occurrence).
   5. Export the matched-term list and the final within-domain tag for downstream aggregation.

Results

Four open-source, instruction-tuned large language model backbones (Backbone A–D) listed in the Table of Materials are benchmarked on the three-class job-posting classification task. Evaluation is conducted using the same test protocol, preprocessing procedures, and metrics across all backbones, including overall accuracy, precision, recall, and F1. Prompt refinement and retrieval-augmented generation (RAG) are subsequently applied to Backbone B and re-evaluated under identical conditions. The performanc...

Discussion

Prior work has applied classical machine learning and neural models to recruitment text categorization, including resume and job description classification, but such approaches often require substantial labeled data and may degrade when domain terminology shifts. Ali et al. proposed a resume classification system using NLP and machine learning techniques¹⁸. Jalili et al. explored BiLSTM-based resume classification¹⁹. Pal et al. evaluated resume categorization using classica...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
all-MiniLM-L6-v2 (sentence encoder)	Hugging Face (sentence-transformers)	https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2	Sentence embedding model used for vectorization.
bert-base-chinese (baseline encoder)	Hugging Face (google-bert)	https://huggingface.co/google-bert/bert-base-chinese	Baseline encoder (Chinese BERT base).
DDR5 memory, 16 GB	Workstation/laptop configuration	N/A	System memory (16 GB DDR5); vendor/part number not specified.
DeepSeek-R1-Distill-Qwen-7B (Backbone A)	Hugging Face (deepseek-ai)	https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B	LLM backbone A.
GLM-4-9B-Chat (Backbone C)	Hugging Face (zai-org)	https://huggingface.co/zai-org/glm-4-9b-chat-hf	LLM backbone C.
Intel Core i5-13500HX CPU	Intel	https://www.intel.com/content/www/us/en/products/sku/232156/intel-core-i513500hx-processor-24m-cache-up-to-4-70-ghz/specifications.html	Workstation CPU used for experiments.
InternLM2.5-7B-Chat (Backbone D)	Hugging Face (internlm)	https://huggingface.co/internlm/internlm2_5-7b-chat	LLM backbone D.
jieba (Chinese tokenizer)	PyPI (jieba)	https://pypi.org/project/jieba/	Chinese tokenization/segmentation library; version not specified.
Keyword libraries (AI & Environmental) (Supplement File 3)	This study	Supplement File 3	Domain keyword lexicons used for lexicon-guided pre-filtering; archive exact version used per run.
LanceDB (vector database)	LanceDB	N/A	Vector database for storing/searching embeddings; version not specified.
NVIDIA GeForce RTX 4060 Laptop GPU (8 GB VRAM)	NVIDIA	https://www.nvidia.com/en-us/geforce/laptops/40-series/	GPU used for inference/experiments.
Online recruitment platform job postings (2014–2023)	Major Chinese recruitment platforms (public web data)	N/A	Publicly posted job advertisements collected and curated by the authors; ~6.93 million postings after cleaning.
pip (Python package installer)	PyPA	N/A	Package installer used to install dependencies and export an environment snapshot using pip freeze.
Prompt template evolution (Supplement File 2)	This study	Supplement File 2	Successive prompt variants and the final prompt used for evaluation.
Python 3.10	Python Software Foundation	N/A	Runtime interpreter (Python 3.10); patch version not specified.
Qwen2.5-7B-Instruct (Backbone B)	Hugging Face (Qwen)	https://huggingface.co/Qwen/Qwen2.5-7B-Instruct	LLM backbone B.
RAG knowledge base field descriptions/index (Supplement File 1)	This study	Supplement File 1	Schema/field notes and document index for the knowledge base used in retrieval-augmented inference.
Thesaurus (AI & Environmental) (Supplement File 4)	This study	Supplement File 4	Term dictionaries used in reclassification and analysis; archive exact version used per run.
Windows 11 (64-bit)	Microsoft	N/A	Operating system (Windows 11, 64-bit); build/version not specified.