Machine Learning-assisted Raman Spectral Analysis of Serotonin-responsive ssDNA-SWCNT Nanosensor for Improved Selectivity against Dopamine

Summary

The protocol describes Raman spectral analysis of ssDNA-wrapped SWCNT nanosensor, enabling the improved selective measurement of serotonin against dopamine with the assistance of a machine learning model.

Abstract

Serotonin (5-hydroxytryptamine, 5-HT) plays critical roles in neuromodulation, yet current detection methods struggle to provide real-time sensing of 5-HT with high sensitivity and selectivity. We previously developed a near-infrared serotonin nanosensor (nIRHT), which consists of ssDNA-wrapped single-walled carbon nanotube that sensitively detects 5-HT. However, nIRHT’s fluorescence response cannot discriminate between 5HT and dopamine (DA), limiting its practical applications. In this study, Raman spectroscopy combined with machine learning overcomes this selectivity challenge. G-band spectral features revealed distinct signatures for 5HT versus DA binding to nIRHT, with DA causing greater G-band suppression. We employed differential Raman (ΔRaman) to isolate analyte-specific spectral changes and trained three machine learning models for classification. The random forest model with ΔRaman achieved optimal performance with 95.8% accuracy, significantly outperforming models using raw Raman spectra. This approach showed improved specificity, with negligible responses to acetylcholine, GABA, and glutamate, and achieved a detection limit of 0.1 µM suitable for physiological applications. This Raman-based approach transforms the non-selective nIRHT fluorescence sensor into a platform capable of robust neurotransmitter discrimination, overcoming selectivity issues in single-walled carbon nanotube (SWCNT)-based molecular sensing.

Introduction

Serotonin (5-hydroxytryptamine, 5-HT) is a critical neurotransmitter regulating mood, cognition, sleep, and appetite, with dysfunction implicated in depression, anxiety, and other neuropsychiatric disorders¹. Real-time monitoring of serotonin dynamics in biological systems remains challenging due to the millisecond timescale of synaptic transmission and the complex chemical environment of neural tissue². Despite significant advances in neurotransmitter detection technologies, achieving both high sensitivity and molecular selectivity for serotonin continues to pose substantial analytical challenges.

Current serotonin detection methods span from traditional analytical techniques to emerging nanosensor platforms. High-performance liquid chromatography with electrochemical detection (HPLC-ED) remains the gold standard, achieving detection limits of 0.056 nM with excellent reproducibility³. However, HPLC requires extensive sample preparation and 15-30 min analysis times, precluding real-time monitoring⁴. Fast-scan cyclic voltammetry (FSCV) enables subsecond temporal resolution at carbon fiber microelectrodes but suffers from electrode fouling and limited selectivity between structurally similar neurotransmitters^5,6. Recent developments in genetically encoded fluorescent sensors, including iSeroSnFR⁷ and GRAB-5HT⁸, achieve remarkable sensitivity and dynamic range but require genetic modification of target tissues, thereby limiting clinical translation.

Single-walled carbon nanotube (SWCNT) optical sensors have emerged as promising platforms for label-free neurotransmitter detection⁹. DNA-functionalized SWCNTs exhibit near-infrared (nIR) fluorescence modulation upon analyte binding, enabling detection in the tissue-transparent window (900-1400 nm)^10,11. Landry's group developed the nIRHT sensor through systematic evolution of ligands by exponential enrichment (SELEC), screening ~10¹⁰ unique ssDNA sequences to identify serotonin-responsive constructs¹². This sensor achieves micromolar sensitivity with 200% fluorescence enhancement upon serotonin binding, operating independently of dopamine or serotonin oxidation. To rule out oxidative interference, our previous study confirmed the stability of these neurotransmitter solutions; UV-Vis and NMR analyses revealed no oxidative changes or new peak formations compared to pure 5HT¹². Furthermore, the short incubation time and neutral pH conditions used in the system are designed to prevent significant quinone formation. Similarly, the Strano group pioneered corona phase molecular recognition (CoPhMoRe), demonstrating that (GT)₁₅-wrapped SWCNTs detect dopamine at 11 nM concentrations¹³. Our group recently advanced this field through machine learning-guided optimization, achieving 2.5-fold sensitivity improvements using high-throughput screening of systematically mutated sequences¹⁴.

Despite these advances, a fundamental limitation persists in the insufficient selectivity between serotonin and dopamine. Both neurotransmitters contain aromatic rings and ethylamine chains, differing by only a single hydroxyl group¹⁵. Current fluorescence-based SWCNT sensors typically achieve selectivity ratios below 5:1, inadequate for distinguishing co-released neurotransmitters in synaptic environments¹⁶. This selectivity challenge is particularly critical given that serotonin and dopamine often colocalize in brain regions governing reward and motivation¹⁷.

Raman spectroscopy offers complementary chemical information through vibrational mode analysis of the SWCNT-analyte complex¹⁸. The G-band (~1580 cm⁻¹) of SWCNTs splits into G⁺ and G⁻ components, with frequencies sensitive to molecular adsorption and electronic doping¹⁹. A previous study demonstrated that non-covalent surfactants on SWCNT induce distinct Raman spectral changes in the G-band region²⁰.

In this work, we demonstrate machine learning-assisted Raman spectral analysis of the nIRHT sensor to achieve enhanced selectivity between serotonin and dopamine. We systematically characterized concentration-dependent Raman responses from 0.01 to 100 µM, revealing distinct spectral fingerprints for each neurotransmitter. Three machine learning models—convolutional neural network (CNN), support vector machine (SVM), and random forest (RF) were trained on spectral datasets to classify analyte identity. Critically, we introduce differential Raman processing, which removes baseline signals to isolate analyte-specific features. The optimized RF model with DR processing achieves 95.8% classification accuracy, representing a significant advance in molecular discrimination. This multimodal approach combining nIR fluorescence sensitivity with Raman chemical specificity provides a pathway toward real-time, selective serotonin monitoring in complex biological environments.

Protocol

1. ssDNA-functionalized SWCNT nIRHT nanosensor fabrication

1. Combine 1 mg of single-walled carbon nanotubes (SWCNTs), 100 µL of 1 mM E6#9 ssDNA, and 900 µL of 1x phosphate-buffered saline (PBS) in a 1.5 mL tube.
   CAUTION: Dry SWCNT powder is a potential inhalation hazard. Handle the powder inside a certified chemical fume hood and wear appropriate personal protective equipment, including a mask and gloves.
2. Disperse the mixture using a bath sonicator for 5 min. Sonicate the mixture using a tip sonicator equipped with a 3 mm probe at 50% amplitude for 30 min in an ice bath.
3. Centrifuge the sonicated solution at 21,500 x g for 1 h to precipitate non-dispersed SWCNTs. Carefully collect 850 µL of the supernatant and transfer it to a new tube.
4. Measure the absorbance of the collected SWCNT suspension at a wavelength of 632 nm. Calculate the concentration using the SWCNT extinction coefficient of 0.036 (mg/L)⁻¹ cm⁻¹. Store the final nIRHT sensor solution at 4 °C until further use.

2. Raman spectra measurement

1. Load 90 µL of the 10 mg/L nIRHT dispersion into a 96-well plate. Measure the Raman spectrum of the sensor in a PBS buffer using a 532 nm laser excitation source. Set this recorded spectrum as the baseline.
   CAUTION: It is crucial to strictly maintain the laser power at 10 mW and wavelength at 532nm when investigating live biological samples. Higher laser powers can induce severe phototoxicity and compromise cell viability. The 532 nm laser at 10 mW utilized in this protocol has been well-established as safe and non-destructive for live-cell analysis, preserving sensitive organelles²¹ and the electrophysiological activity of primary neurons²² while ensuring robust Raman signal acquisition.
2. Spike each 90 µL sensor solution (10mg/L) with 10 µL of the analyte solution to reach final concentrations of 0.01, 0.1, 1, 5, 10, 50, and 100 µM. Wait for 5 min after the analyte addition to allow for sufficient incubation.
3. Measure the Raman spectra of the incubated samples. Prepare samples in triplicate for each concentration and repeat steps 2.1 to 2.2 to ensure reproducibility.

3. Machine learning-based data analysis

1. Construct the dataset from a total of 208 Raman spectra, comprising 77 dopamine (DA) responses, 76 serotonin (5HT) responses, and 47 control (PBS) group responses.
2. Perform background subtraction on the G-band region of the spectral data. Normalize the background-subtracted data based on the G+ band peak.
3. Apply a median filter to reduce noise in the normalized spectra. Isolate spectral changes (ΔRaman) by subtracting the normalized control spectrum from each analyte-treated spectrum.
4. Develop a Convolutional Neural Network (CNN), a Random Forest (RF), and a Support Vector Machine (SVM) model for the ternary classification of 5HT, DA, or the control.
5. Split the entire dataset into training, validation, and test sets using a 6:2:2 ratio.
6. Configure the CNN architecture with three 1D convolution layers utilizing a ReLU activation function and a max pooling size of 2, followed by two fully connected layers.
7. Configure the SVM using a Radial Basis Function (RBF) kernel for non-linear classification.
8. Configure the RF model using default hyperparameters.
9. Train the constructed models using 5-fold cross-validation. Calculate the average performance metrics across all 5 folds to ensure statistical robustness.
10. Select the model that yields the highest accuracy on the validation data for the final analysis.
11. Evaluate the selected model on the test set to verify its generalization capability.
12. Determine the indirect limit of detection (LOD) by sequentially excluding data from the training set that falls below a concentration threshold of 0.1 µM.
13. Test the trained model on the entire concentration range, including concentrations below the criteria, to establish the final LOD.

Results

The E6#9 ssDNA-functionalized SWCNT nanosensor (nIRHT), previously developed through SELEC methodology, demonstrates reversible nIR fluorescence enhancement upon 5HT binding, enabling real-time 5HT imaging in vitro and ex vivo¹². The E6#9 sequence (5'-CCCCCCAGCACCAGACAGCACACTCCCCCC-3') wraps around SWCNTs to create binding sites for 5HT. However, this sensor lacks specificity: both 5-HT and DA result in comparable intensity increase (Figure 1B), limiting its utility to discriminate between DA and 5-HT.

To overcome this limitation, we employed Raman spectroscopy to investigate molecular-level interactions between neurochemicals and nIRHT nanosensors. Raman analysis of nIRHT nanosensor solution revealed distinct spectral features in the G-band region (~1580 cm^-1 , which originates from in-plane carbon vibrations in the graphene-like structure of SWCNTs (Figure 1C). The G band could be split into two components: a sharp G+ band (1580 cm^-1) corresponds to longitudinal optical phonons along the nanotube axis, and a broader G- band (1540-1570 cm^-1) corresponds to circumferential vibrations^23,24. The G+ mode is relatively insensitive to electronic perturbation due to its intrinsic weaker electron-phonon coupling, whereas the G- mode is more sensitive to changes in the local electronic environment. A previous study established that covalent modifications, such as oxidation, shift G-band frequencies through disruption of the sp² carbon network²⁵. We observed that non-covalent 5HT and DA adsorption of DA and 5-HT onto nIRHT through the π-π stacking differentially modulates the G-band features through interfacial strain and redox-driven ability. Structurally, the compact catechol group of DA enables closer contact with the highly curved CNT surface, potentially inducing stronger electronic perturbation and more effective electronic interactions compared to the larger planar indole structure of 5-HT. In addition, DA undergoes the catechol-to-quinone redox process, facilitating efficient interfacial charge redistribution and stronger modulation of electron–phonon coupling with the circumference of CNT. In contrast, 5-HT follows a less efficient redox reaction due to its more stabilized π electron system of the indole moiety, resulting in weaker electronic perturbation and smaller G^- shift. The E6#9 ssDNA corona phase further contributes to the analyte-specific Raman modulation. The ssDNA wrapping creates molecular binding pockets in close proximity to the SWCNT surface, where analyte adsorption perturbs not only the fluorescence response but also the vibrational modes of the underlying sp² carbon lattice. As simulated for other ssDNA-SWCNT nanosensors^26,27, π–π stacking between the aromatic rings of neurotransmitters and SWCNT sp²-carbons facilitates direct surface contact, while the ssDNA tertiary structure stabilizes the binding geometry in a non-covalent manner.

Raman spectral changes across seven concentrations of 0.01, 0.1, 1, 5, 10, 50, and 100 µM were measured, encompassing physiological neurotransmitter levels (Supplementary Figure 1). DA exhibited pronounced concentration-dependent G⁻ band suppression, with maximum attenuation at 100 µM that progressively diminished at lower concentrations (Figure 2A). 5HT induced qualitatively similar but quantitatively weaker concentration-dependent changes, maintaining distinguishable spectral features even at low concentrations (Figure 2B).

For quantification of analyte-specific spectral changes, we calculated differential Raman (ΔRaman) spectra by subtracting baseline measurements from analyte-exposed samples.

ΔRaman = {Raman (after) - Raman (before)}/Raman (before)

At 100 µM, DA and 5HT generated distinct ΔRaman peak patterns (Figure 2C,D), confirming that each neurotransmitter induces unique spectroscopic shifts despite similar fluorescence responses. These differential signatures arise from varying molecular orientations and binding geometries on the SWCNT surface.

ΔRaman spectra at 1570 cm^-1 yielded a concentration-dependent curve fitted to the Hill equation (Figure 2E). The calculated dissociation constants (K_d) were 2.06 µM for 5HT and 0.89 µM for DA. Both neurochemicals showed sigmoidal dose-response curves with signal saturation above 100 µM. The distinct spectral patterns will enable reliable classification between 5-HT and DA, which would be difficult in fluorescence-based detection, where both analytes produce indistinguishable responses. The temporal response of nano sensors was measured against 5HT and DA (Supplementary Figure 2). The response time is below 1 s for both analytes, which shows the rapid recognition of neurochemicals on the SWCNT corona phase.

While the 1 s scale temporal resolution has a limitation for capturing the msec-scale synaptic response, our sensor ensures a sufficient resolution for 5HT sensing in the biochemical applications. For instance, 5HT can act as a neuromodulator through volume transmission, diffusing across the extracellular space²⁸. Furthermore, the temporal resolution of our sensor is highly sufficient for real-time monitoring of 5-HT in previous research. The conventional method for 5HT sensing in vivo, such as microdialysis, has poor resolution around ~1 min²⁹. To compare this methodology, 1 s resolution is suitable for real-time monitoring. Overall, a 1 s response time provides a sufficient and practical resolution for tracking extracellular 5-HT dynamics.

To evaluate the selectivity of nIRHT's Raman response to other neurochemicals, we examined spectral changes induced by other major neurotransmitters: acetylcholine (ACh), γ-aminobutyric acid (GABA), and glutamate (Glu). While previous fluorescence studies showed minimal responses to these analytes, their effects on Raman spectra had not been characterized. Relative ΔRaman was calculated relative to baseline measurements (Figure 2F). ACh, GABA, and Glu produced negligible Raman spectral changes, with ΔRaman values remaining near zero across the entire spectral range. This absence of spectral modulation contrasts sharply with the pronounced G⁻ band suppression observed for 5HT and DA. This orthogonal validation confirms that the Raman spectral changes are specific to catecholamine and indoleamine neurotransmitters rather than general ionic effects. The G-/G+ ratios of non-target analytes were consistent with those of the control group (Supplementary Figure 3). However, DA showed a highly suppressed ratio of ~0.06, and 5HT indicated an intermediate ratio of ~0.14. The clear quantitative separation between DA and 5HT, and the non-target agent, assures the analyte-specific spectral responses and supports the selectivity of Raman-based classification.

These results demonstrate that Raman spectroscopy provides orthogonal chemical information complementary to fluorescence measurements, transforming the nIRHT sensor from a sensitive but non-selective detector into a platform capable of molecular discrimination. The concentration-dependent spectral evolution and analyte-specific ΔRaman signatures establish the foundation for machine learning-based classification algorithms to achieve robust neurotransmitter identification in complex biological environments.

To evaluate the quantitative correlation between the G- band peak position and target analyte molarity, we extracted these positions across the tested concentration ranges (Supplementary Figure 4). Our analysis revealed a distinct, concentration-dependent shift to lower wavenumbers. Notably, 5-HT exhibited a significantly smaller downshift (~6 cm⁻¹) compared to DA (~10 cm⁻¹). This clear difference in the magnitude of the concentration-dependent shift provides a reliable analytical metric, further reinforcing the specificity of the sensor for distinguishing 5-HT from DA.

The differential Raman spectra were theoretically distinct for 5HT and DA, while it is challenging to discriminate visually at physiological concentrations. For more robust classification, we implemented machine learning algorithms, including a convolutional neural network (CNN), support vector machine (SVM), and random forest (RF), for determining the best-performing model to analyze the spectral datasets (Figure 3A). Even though CNN is not suitable for a miniature data size, we tested this model due to its well- generalized performance. Notably, even prior to exhaustive optimization, the CNN yielded highly reliable results, achieving a classification accuracy of approximately 90% without exhibiting significant overfitting (Supplementary Figure 5). This robust performance indicates that the spectral features of 5-HT acquired in our study are highly distinct and informative, enabling the neural network to effectively extract discriminative patterns despite the limited sample size.

The training dataset consists of 208 experimental Raman spectra collected across seven concentrations (0.01-100 µM): 76 5HT response, 77 DA response, and 55 control measurements treated with PBS buffer only. Due to the limited dataset size, we employed 5-fold cross-validation to maximize training efficiency and minimize overfitting. The dataset was split into training, validation, and test sets in a 6:2:2 ratio. For each of the validation folds, the data was randomly shuffled to remove unintended clustering. Performance metrics were calculated as the mean across all validation folds to ensure statistical robustness.

Initial evaluation compared classification accuracy (ACC) across the three models using both normalized Raman spectra and ΔRaman as explained in the previous section (Figure 3B). For normalized Raman spectra, RF achieved the highest ACC at 87.9%, while SVM showed the lowest performance at 71.6%. ΔRaman processing significantly enhanced model performance across all algorithms. The RF model maintained superior performance with 95.8% accuracy, while CNN and SVM showed significant improvements of 9% and 14% rather than the normalized Raman model, respectively. This enhancement demonstrates that baseline subtraction effectively removes systematic noise and helps to focus on analyte-specific spectral features for classification. Based on these results, we selected the ΔRaman-based RF model as the optimal classifier model.

We also evaluated normalized Raman and ΔRaman RF models using standard performance metrics, including ACC, area under the curve (AUC), recall, and F1 score (Figure 3B). The normalized Raman RF model achieved 94.6% AUC, 88.1% ACC, 86.8% F1 score, and 86.3% recall. However, the ΔRaman-based model showed superior, more balanced performance across all metrics, with 95.5% AUC, 95% ACC, 94.1% F1 score, and 95.2% recall. The receiver operating characteristic (ROC) curve for the ΔRaman RF model showed excellent discrimination capability (Figure 3D).

To establish the practical detection range, we evaluated model performance as a function of analyte concentration (Figure 3E). We evaluated classification performance by sequentially excluding the training dataset at each concentration to determine the limit of detection (LOD) indirectly. In comparison to 94.1% ACC without any exclusion, ACC increased to 97.10% after excluding 0.01 µM data. It implied Raman spectral change by 0.01 µM 5HT is somewhat indistinguishable from baseline measurements. While the model trained with 0.1 µM and above, model ACC was saturated between 97%-100%, establishing 0.1 µM as the practical LOD for reliable classification. This detection limit aligns with physiological 5HT concentration in synaptic environments³⁰, demonstrating the practical applicability of this approach for biological applications.

These results establish that machine learning-assisted Raman analysis will improve the nIRHT sensor from a sensitive but less-selective detector into a platform enabling robust neurotransmitter discrimination. The combination of ΔRaman processing and RF classification provides a robust analytical framework that overcomes the fundamental limitations of fluorescence-based approaches.

Figure 1: Schema and nIRHT spectra. (A) Scheme for the classification of serotonin (5-hydroxytrypamine, 5HT) and dopamine (DA) by using machine learning-assisted Raman spectral analysis on near-infrared 5HT-responsive ssDNA-SWCNT (nIRHT) nanosensor. (B) Fluorescence spectra of nIRHT solution before (black) and after incubation with 100 µM 5HT (blue) and DA (red). (C) Normalized Raman G band spectra of nIRHT solution before (black) and after incubation with 100 µM 5HT (red) and DA (blue). Please click here to view a larger version of this figure.

Figure 2: Raman spectra. (A) Raman spectra after incubation with 5HT at different concentrations of 100, 50, 10, 5, 1, 0.1 and 0.01 µM. (B) Raman spectra after incubation with DA at different concentrations of 100, 50, 10, 5, 1, 0.1 and 0.01 µM. (C) Differential Raman (ΔRaman) spectrum of 5HT at seven concentrations, each 100, 50, 10, 5, 1, 0.1 and 0.01 µM. The lighter the color, the lower the concentration. (D) ΔRaman spectrum of DA at seven concentrations, each 100, 50, 10, 5, 1, 0.1 and 0.01 µM. (E) Concentration-dependent ΔRaman at 1570 cm^-1 for 5HT and DA. Data is fitted with the Hill equation. (F) ΔRaman intensity value at 1570 cm^-1 for different neurotransmitters at 10 µM, which was treated with ACh, GABA, Glu, DA, 5HT, and 1x PBS only (Ctrl). Please click here to view a larger version of this figure.

Figure 3: Machine learning analysis of nanosensor Raman spectral data. (A) Scheme of machine learning analysis of nanosensor Raman spectral data. After acquisition of spectrum, Raman spectra were trained by three models of CNN, SVM, and RF, in 5-fold cross validation, and evaluated by ACC to select the best model of them. After selection of the best model, the model was assessed by general performance metrics. (B) Accuracy (ACC) performance for each data type and model. (C) General performance metrics at ΔRaman-based RF model and pristine Raman-based RF model. (D) Receiver Operating Characteristic (ROC) curve of ΔRaman-based RF model. (E) ACC of ΔRaman-based RF model depending on different training dataset. Full data uses all concentration data, and > 0.1 µM implies to include the truncated dataset which possesses the dataset using more than > 0.1 µM concentration of 5HT and DA. Please click here to view a larger version of this figure.

Supplementary Figure 1: Raman spectra at different concentrations. (A) Raman spectra after incubation with 5HT at different concentrations of 100, 50, 10, 5, 1, 0.1 and 0.01 µM. (B) Raman spectra after incubation with DA at different concentrations of 100, 50, 10, 5, 1, 0.1 and 0.01µM. (C) Differential Raman spectrum of 5HT at seven concentrations, each 100, 50, 10, 5, 1, 0.1 and 0.01 µM. (D) Differential Raman spectrum of DA at seven concentrations, each 100, 50, 10, 5, 1, 0.1 and 0.01µM. The lighter the color, the lower the concentration.Please click here to download this file.

Supplementary Figure 2: Time-dependent change of differential Raman signal after 5HT (red) and DA (blue) addition.Please click here to download this file.

Supplementary Figure 3: G^-/G⁺ ratios for different neurotransmitters at 10 µM, which was treated with ACh, GABA, Glu, DA, 5HT, and 1X PBS only (Ctrl).Please click here to download this file.

Supplementary Figure 4: Raman peak shift of the G^- peak with 5HT (red) and DA (blue) at different concentrations of 100, 50, 10, 5, 1, 0.1, and 0.01 µM. n=3, error bars show mean ± standard error. Please click here to download this file.

Supplementary Figure 5: Training (blue) and test curve (orange) about accuracy of the CNN model in 50 epochs.Please click here to download this file.

Supplementary Figure 6: Raman spectra with sequential binding and washing cycles. (A) Raman spectra of the sensor at baseline (dashed gray line), after the addition of 5HT (red), and after the subsequent washing step (solid grey). The peak intensity increases upon 5-HT binding and recovers to the baseline level after washing. (B) Raman spectra of the sensor at baseline (dashed gray line), after the addition of DA (blue), and after the subsequent washing step (solid grey). The peak intensity increases upon binding DA and recovers to the baseline level after washing. (C) Quantitative comparison of the G^- peak intensity across the sequential binding and washing cycles with 5HT. (D) Quantitative comparison of the G^- peak intensity across the sequential binding and washing cycles with DA.Please click here to download this file.

Supplementary Figure 7: Classification of DA and 5HT using Linear Discriminant Analysis (LDA). Histogram shows the distribution of the first linear discriminant (LD1) scores derived from the Raman spectrum of 5HT (red) and DA (blue).Please click here to download this file.

Discussion

Raman spectroscopic analysis provided a significant enhancement in the specificity of the nIRHT sensor, a capability not achievable with its conventional fluorescence response. We observed that DA induced a more pronounced suppression of the G⁻ band (1570 cm^-1) compared to 5HT. These differential signatures likely arise from varying molecular orientations and binding geometries on the SWCNT surface. Specifically, DA's catechol moiety likely forms stronger π-π interactions with the SWCNT surface, inducing greater electronic perturbation manifested as enhanced G⁻ suppression. In contrast, 5HT's indole ring system appears to produce weaker electronic coupling, preserving more of the intrinsic Raman signature. This differential interaction provides a clear spectroscopic basis for discriminating between these two structurally similar neurotransmitters.

To evaluate the practical feasibility of our nanosensors for continuous monitoring, we performed wash-out experiments using nanosensors immobilized on an APTES-coated glass substrate. The Raman spectra exhibited distinct peak changes upon the addition of 5-HT and DA and successfully recovered to their initial baselines after a subsequent washing step (Supplementary Figure 6). This clearly demonstrates the reversible nature of the sensor's response. Interestingly, the specific trend of these spectral shifts in the substrate-immobilized setup is opposite to the behavior typically observed in a solution state. While the underlying mechanism for this discrepancy warrants further investigation, the confirmed reversibility strongly supports the utility of this platform for monitoring transient neurotransmitter release.

Importantly, the Raman measurements in this study were conducted using a nanosensor concentration of 10 mg/L, which is directly comparable to the concentrations typically employed in fluorescence-based neurotransmitter detection²⁹. The ability to perform both Raman and photoluminescence (PL) measurements effectively at this consistent concentration highlights the potential of the system as a practical multimodal platform. This compatibility enables the simultaneous application of dual optical modalities to reliably identify and differentiate neurotransmitters such as serotonin and dopamine.

To verify that the classification of DA and 5-HT was driven by genuinely distinct spectral features rather than concentration-dependent variations, we performed Linear Discriminant Analysis (LDA) on the Raman spectra (Supplementary Figure 7). The LDA results demonstrated that the spectral data possess distinctly separable features unique to each analyte. Furthermore, this clear discrimination was maintained entirely independent of their concentrations, confirming that our classification approach relies on intrinsic molecular signatures.

The use of Raman spectroscopy significantly improved the sensor's specificity over fluorescence measurement. Our machine learning model, trained on the entire concentration dataset, classified serotonin with an accuracy exceeding 95%. This compares favorably to the limited specificity reported for fluorescence-based serotonin classification¹⁶, a result we attribute to the structural specificity of Raman spectroscopy. Furthermore, the model's performance increased by over 8% after applying ΔRaman processing, compared to the model based on normalized Raman spectra. This enhancement is attributed to the removal of experimental errors and systematic noise via the ΔRaman processing. However, a limitation of the ΔRaman-based RF model is its low sensitivity, which is evident from the limit of detection (LOD) of 0.1 µM. To overcome this sensitivity limitation, we propose to combine our Raman analysis with the fluorescence analysis to improve classification performance at these low concentrations.

The primary technical significance of this work is the establishment of a robust analytical framework that overcomes the fundamental limitations of fluorescence-based approaches for nIRHT sensors. The distinct spectral patterns we identified enable reliable classification between 5-HT and DA, a task that is intractable using conventional fluorescence detection, where both analytes produce indistinguishable responses. These results demonstrate that Raman spectroscopy provides orthogonal chemical information complementary to fluorescence measurements. This capability effectively transforms the nIRHT sensor from a sensitive but non-selective detector into a platform capable of molecular discrimination. Furthermore, the concentration-dependent spectral evolution and the analyte-specific ΔRaman signatures establish the foundation for machine learning-based classification algorithms. This combined ΔRaman processing and RF classification provides a robust analytical framework to achieve robust neurotransmitter identification, even in complex biological environments. Crucially, the established detection limit aligns with physiological 5HT concentrations in synaptic environments³⁰, demonstrating the practical applicability of this approach for biological applications. This machine learning-assisted Raman analysis ultimately improves the nIRHT sensor into a platform enabling robust neurotransmitter discrimination.

We successfully demonstrated that machine learning-assisted Raman spectroscopy enables the nIRHT nanosensor to discern 5HT from DA with 95.8% accuracy. This approach addresses the fundamental selectivity limitation that has constrained SWCNT-based neurotransmitter sensing. While the nIRHT nanosensor exhibits decent sensitivity through near-infrared fluorescence, it cannot fully distinguish between 5HT and catecholamines. By incorporating Raman spectroscopy as an orthogonal detection modality, we can enhance the molecular specificity necessary for practical applications. The negligible Raman responses to Ach, GABA, and Glu further strengthen the sensor specificity.

Future development should focus on integrating fluorescence and Raman detection into a single platform for simultaneous sensitivity and selectivity, developing deep learning architectures for enhanced classification of neurotransmitter mixtures. This multimodal sensing paradigm, combining SWCNT nanosensor, fluorescence imaging, and Raman spectroscopy, provides a robust framework for addressing selectivity challenges in molecular sensing and opens new possibilities for real-time monitoring of neurotransmission dynamics.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
HiPCo raw	Nanointegris
iDus1.7 InGaAs	ANDOR	DU490A-1,9
Ramantouch	Nano Photon
Vibra Cell 130	SONICS